Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 6 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 01:04:27.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)

info Neo4j load anomaly test26 resolved impact 9 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) opened 84d ago · last observed 2d ago · observed 113× · #107 · 7edd4533dc7becc22419

What happened template

docproc's Neo4j load is 343 q/min on test26, above a normal of 13 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 93.0, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

Queries per minute far outside the hour-of-week normal.

The facts

measureneo_qpm
unitq/min
value343.40
expected_median12.60
expected_mad3.56
z92.97
rel26.25
buckets_hit3
n65
weeks6
slotMon 05:00 UTC
bucket2026-11-16 05:40:00 UTC (2d ago)
discriminatorneo_qpm
window2026-11-16 05:30:00 UTC → 2026-11-16 05:50:00 UTC

What is failing in docproc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-16 05:30:00 UTC → 2026-11-16 05:50:00 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

159× docsvc PATCH failed for 7e570bbe-1444-4de5-a198-6ee79f87cafd: 403 {"message":"email_not_verified: verify your email addr… since 2026-11-02 03:59:13 UTC
8× Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in l… since 2026-10-28 15:09:17 UTC
8× needs_attention failed for 35f8e0bf-4a32-40b4-9de0-065f34780232: HTTP 500: {"message":"could not update document metadat… since 2026-11-16 05:07:04 UTC
2× needs_attention failed for 3ca55aca-898d-4f8a-8f8e-db0fcca6f097: HTTP 500: {"message":"could not update document metadat… since 2026-11-16 05:06:50 UTC
1× CBR promote failed for 47b5b3f9-4cbd-4db2-a550-bb7c0ea36756: HTTP 500: {"message":"could not update document stateTag: E… since 2026-11-16 05:38:59 UTC

All docproc errors, grouped → · service page · its logs

History

  • 2026-11-16 01:36:14 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 182.4, 'expected_median': 10.8, 'expected_mad': 1.33434, 'z': 128.6029048068708, 'rel': 15.888888888888888, 'buckets_hit': 3, 'n': 64, 'weeks': 6, 'slot': 'Mon 01:00 UTC', 'bucket': 1794792300, 'window': [1794791400, 1794792600], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794769800&to=1794792600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794769800&to=1794792600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794792974, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.8, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794792974, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 108, 'narrative': "docproc's Neo4j load is 30 q/min on test26, above a normal of 10 q/min ± 0 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 67.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7069641, 'stale_s': 13}
  • 2026-11-16 02:02:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 182.4, 'expected_median': 10.8, 'expected_mad': 1.33434, 'z': 128.6029048068708, 'rel': 15.888888888888888, 'buckets_hit': 3, 'n': 64, 'weeks': 6, 'slot': 'Mon 01:00 UTC', 'bucket': 1794792300, 'window': [1794791700, 1794792900], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794770100&to=1794792900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794770100&to=1794792900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794793430, 'age_seconds': 12, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.8, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794793430, 'resolved_at': 1794794547, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 109, 'narrative': "docproc's Neo4j load is 182 q/min on test26, above a normal of 11 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.6, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7071215, 'stale_s': 1131}
  • 2026-11-16 02:18:03 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 224 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 224 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 224.2, 'expected_median': 12.6, 'expected_mad': 2.66868, 'z': 79.29013594735973, 'rel': 16.793650793650794, 'buckets_hit': 3, 'n': 53, 'weeks': 5, 'slot': 'Mon 02:00 UTC', 'bucket': 1794794400, 'window': [1794793800, 1794795000], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794772200&to=1794795000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794772200&to=1794795000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794795483, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.7, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794795483, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 110, 'narrative': "docproc's Neo4j load is 182 q/min on test26, above a normal of 11 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.6, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794795483, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7072150, 'stale_s': 13}
  • 2026-11-16 02:51:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 183 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 183.2, 'expected_median': 12.6, 'expected_mad': 2.66868, 'z': 63.926735314837295, 'rel': 13.53968253968254, 'buckets_hit': 3, 'n': 53, 'weeks': 5, 'slot': 'Mon 02:00 UTC', 'bucket': 1794795600, 'window': [1794794700, 1794795900], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794773100&to=1794795900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794773100&to=1794795900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794796458, 'age_seconds': 14, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.8, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794796458, 'resolved_at': 1794797512, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 112, 'narrative': "docproc's Neo4j load is 183 q/min on test26, above a normal of 13 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 63.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794795483, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7074181, 'stale_s': 1069}
  • 2026-11-16 06:01:45 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 343.4, 'expected_median': 12.6, 'expected_mad': 3.55824, 'z': 92.96730968119068, 'rel': 26.25396825396825, 'buckets_hit': 3, 'n': 65, 'weeks': 6, 'slot': 'Mon 05:00 UTC', 'bucket': 1794807600, 'window': [1794807000, 1794808200], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794785400&to=1794808200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794785400&to=1794808200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808903, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.6, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794808905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 113, 'narrative': "docproc's Neo4j load is 183 q/min on test26, above a normal of 13 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 63.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7085578, 'stale_s': 19}
  • 2026-11-16 06:22:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 343.4, 'expected_median': 12.6, 'expected_mad': 3.55824, 'z': 92.96730968119068, 'rel': 26.25396825396825, 'buckets_hit': 3, 'n': 65, 'weeks': 6, 'slot': 'Mon 05:00 UTC', 'bucket': 1794807600, 'window': [1794807000, 1794808200], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794785400&to=1794808200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794785400&to=1794808200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808903, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.6, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794808905, 'resolved_at': 1794810147, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 113, 'narrative': "docproc's Neo4j load is 343 q/min on test26, above a normal of 13 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 93.0, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7086822, 'stale_s': 1263}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794808903
age_seconds20
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorseasonal