docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)
info
Neo4j load anomaly
test26
resolved
impact 9
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 84d ago · last observed 2d ago · observed 113× · #107 · 7edd4533dc7becc22419
What happened template
docproc's Neo4j load is 343 q/min on test26, above a normal of 13 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 93.0, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Queries per minute far outside the hour-of-week normal.
The facts
| measure | neo_qpm |
| unit | q/min |
| value | 343.40 |
| expected_median | 12.60 |
| expected_mad | 3.56 |
| z | 92.97 |
| rel | 26.25 |
| buckets_hit | 3 |
| n | 65 |
| weeks | 6 |
| slot | Mon 05:00 UTC |
| bucket | 2026-11-16 05:40:00 UTC (2d ago) |
| discriminator | neo_qpm |
| window | 2026-11-16 05:30:00 UTC → 2026-11-16 05:50:00 UTC |
What is failing in docproc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-16 05:30:00 UTC → 2026-11-16 05:50:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 159× | docsvc PATCH failed for 7e570bbe-1444-4de5-a198-6ee79f87cafd: 403 {"message":"email_not_verified: verify your email addr… since 2026-11-02 03:59:13 UTC |
| 8× | Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in l… since 2026-10-28 15:09:17 UTC |
| 8× | needs_attention failed for 35f8e0bf-4a32-40b4-9de0-065f34780232: HTTP 500: {"message":"could not update document metadat… since 2026-11-16 05:07:04 UTC |
| 2× | needs_attention failed for 3ca55aca-898d-4f8a-8f8e-db0fcca6f097: HTTP 500: {"message":"could not update document metadat… since 2026-11-16 05:06:50 UTC |
| 1× | CBR promote failed for 47b5b3f9-4cbd-4db2-a550-bb7c0ea36756: HTTP 500: {"message":"could not update document stateTag: E… since 2026-11-16 05:38:59 UTC |
History
- 2026-11-16 01:36:14 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 182.4, 'expected_median': 10.8, 'expected_mad': 1.33434, 'z': 128.6029048068708, 'rel': 15.888888888888888, 'buckets_hit': 3, 'n': 64, 'weeks': 6, 'slot': 'Mon 01:00 UTC', 'bucket': 1794792300, 'window': [1794791400, 1794792600], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794769800&to=1794792600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794769800&to=1794792600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794792974, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.8, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794792974, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 108, 'narrative': "docproc's Neo4j load is 30 q/min on test26, above a normal of 10 q/min ± 0 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 67.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7069641, 'stale_s': 13}
- 2026-11-16 02:02:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 182 q/min vs a normal 11 q/min ± 1 q/min for Mon 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 182.4, 'expected_median': 10.8, 'expected_mad': 1.33434, 'z': 128.6029048068708, 'rel': 15.888888888888888, 'buckets_hit': 3, 'n': 64, 'weeks': 6, 'slot': 'Mon 01:00 UTC', 'bucket': 1794792300, 'window': [1794791700, 1794792900], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794770100&to=1794792900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794770100&to=1794792900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794793430, 'age_seconds': 12, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.8, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794793430, 'resolved_at': 1794794547, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 109, 'narrative': "docproc's Neo4j load is 182 q/min on test26, above a normal of 11 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.6, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7071215, 'stale_s': 1131}
- 2026-11-16 02:18:03 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 224 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 224 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 224.2, 'expected_median': 12.6, 'expected_mad': 2.66868, 'z': 79.29013594735973, 'rel': 16.793650793650794, 'buckets_hit': 3, 'n': 53, 'weeks': 5, 'slot': 'Mon 02:00 UTC', 'bucket': 1794794400, 'window': [1794793800, 1794795000], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794772200&to=1794795000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794772200&to=1794795000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794795483, 'age_seconds': 13, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.7, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794795483, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 110, 'narrative': "docproc's Neo4j load is 182 q/min on test26, above a normal of 11 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.6, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794795483, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7072150, 'stale_s': 13}
- 2026-11-16 02:51:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 183 q/min vs a normal 13 q/min ± 3 q/min for Mon 02:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 183.2, 'expected_median': 12.6, 'expected_mad': 2.66868, 'z': 63.926735314837295, 'rel': 13.53968253968254, 'buckets_hit': 3, 'n': 53, 'weeks': 5, 'slot': 'Mon 02:00 UTC', 'bucket': 1794795600, 'window': [1794794700, 1794795900], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794773100&to=1794795900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794773100&to=1794795900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794796458, 'age_seconds': 14, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.8, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794796458, 'resolved_at': 1794797512, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 112, 'narrative': "docproc's Neo4j load is 183 q/min on test26, above a normal of 13 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 63.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794795483, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7074181, 'stale_s': 1069}
- 2026-11-16 06:01:45 UTCreopened x-ray · severity=info was=info title=docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 343.4, 'expected_median': 12.6, 'expected_mad': 3.55824, 'z': 92.96730968119068, 'rel': 26.25396825396825, 'buckets_hit': 3, 'n': 65, 'weeks': 6, 'slot': 'Mon 05:00 UTC', 'bucket': 1794807600, 'window': [1794807000, 1794808200], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794785400&to=1794808200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794785400&to=1794808200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808903, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 3.6, 'status': 'open', 'first_seen': 1787723346, 'last_seen': 1794808905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 113, 'narrative': "docproc's Neo4j load is 183 q/min on test26, above a normal of 13 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 63.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7085578, 'stale_s': 19}
- 2026-11-16 06:22:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 107, 'fingerprint': '7edd4533dc7becc22419', 'kind': 'anomaly.neo4j', 'severity': 'info', 'cluster': 'test26', 'subject': 'docproc', 'title': 'docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'neo_qpm', 'unit': 'q/min', 'value': 343.4, 'expected_median': 12.6, 'expected_mad': 3.55824, 'z': 92.96730968119068, 'rel': 26.25396825396825, 'buckets_hit': 3, 'n': 65, 'weeks': 6, 'slot': 'Mon 05:00 UTC', 'bucket': 1794807600, 'window': [1794807000, 1794808200], 'recent_deploys': [], 'discriminator': 'neo_qpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=test26&from=1794785400&to=1794808200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794785400&to=1794808200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808903, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.6, 'status': 'resolved', 'first_seen': 1787723346, 'last_seen': 1794808905, 'resolved_at': 1794810147, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 113, 'narrative': "docproc's Neo4j load is 343 q/min on test26, above a normal of 13 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 93.0, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'neo_qpm', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'Neo4j load anomaly', 'group': 'Anomalies', 'age_s': 7086822, 'stale_s': 1263}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794808903 |
| age_seconds | 20 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |