docproc request rate 5.6 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 18:00 UTC (3 of the last 4 checks)
info
Traffic anomaly
production
resolved
impact 18
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 70d ago · last observed 1d ago · observed 436× · #1311 · 9bf9442311ba102f6466
What happened template
docproc's request rate is 5.6 req/min on production, above a normal of 0.4 req/min ± 0.6 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 8.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 5.60 |
| expected_median | 0.40 |
| expected_mad | 0.59 |
| z | 8.77 |
| rel | 13.00 |
| buckets_hit | 3 |
| n | 48 |
| weeks | 4 |
| slot | Mon 18:00 UTC |
| bucket | 2026-11-16 18:55:00 UTC (1d ago) |
| discriminator | api_rpm |
| window | 2026-11-16 18:45:00 UTC → 2026-11-16 19:05:00 UTC |
What is failing in docproc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-16 18:45:00 UTC → 2026-11-16 19:05:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 200× | [dd5021b7-ca83-4be1-8e43-cb6f396e09e2] PubSub certify failed: AIRun creation failed for document dd5021b7-ca83-4be1-8e43… since 2026-08-26 14:55:56 UTC |
| 146× | [0dc5292f-37b8-457d-87e8-7df83bc10b1c] ══════════════════════════════════════════════════════ since 2026-08-31 09:11:43 UTC |
| 92× | [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] PubSub version upload failed: AIRun creation failed for document b6ce3808-ab0a-44… since 2026-08-25 22:20:29 UTC |
| 92× | docsvc PATCH aiEnabled failed before creating AI run since 2026-11-07 14:31:44 UTC |
| 92× | [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] Version upload handling failed: AIRun creation failed for document b6ce3808-ab0a-… since 2026-08-25 22:20:29 UTC |
History
- 2026-11-16 09:50:12 UTCreopened x-ray · severity=info was=info title=docproc request rate 23.6 req/min vs a normal 0.1 req/min ± 0.1 req/min for Mon 09:00 UTC (4 of the last 4 checks) level=low was_level=low finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 23.6 req/min vs a normal 0.1 req/min ± 0.1 req/min for Mon 09:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 23.6, 'expected_median': 0.1, 'expected_mad': 0.14826, 'z': 158.50532847699986, 'rel': 235.0, 'buckets_hit': 4, 'n': 48, 'weeks': 4, 'slot': 'Mon 09:00 UTC', 'bucket': 1794821400, 'window': [1794820500, 1794821700], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794798900&to=1794821700'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794798900&to=1794821700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794822123, 'age_seconds': 505, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.8, 'status': 'open', 'first_seen': 1788934840, 'last_seen': 1794822612, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 425, 'narrative': "docproc's request rate is 34.6 req/min on production, above a normal of 4.7 req/min ± 5.8 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.2, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794822612, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5887789, 'stale_s': 17}
- 2026-11-16 10:19:38 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 23.6 req/min vs a normal 0.1 req/min ± 0.1 req/min for Mon 09:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 23.6, 'expected_median': 0.1, 'expected_mad': 0.14826, 'z': 158.50532847699986, 'rel': 235.0, 'buckets_hit': 4, 'n': 48, 'weeks': 4, 'slot': 'Mon 09:00 UTC', 'bucket': 1794821400, 'window': [1794820500, 1794821700], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794798900&to=1794821700'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794798900&to=1794821700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794822123, 'age_seconds': 505, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 17.8, 'status': 'resolved', 'first_seen': 1788934840, 'last_seen': 1794822612, 'resolved_at': 1794824378, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 425, 'narrative': "docproc's request rate is 23.6 req/min on production, above a normal of 0.1 req/min ± 0.1 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 158.5, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794822612, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5889558, 'stale_s': 1786}
- 2026-11-16 10:49:21 UTCreopened x-ray · severity=info was=info title=docproc request rate 26.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 10:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 26.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 10:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 26.0, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.2, 'rel': inf, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 10:00 UTC', 'bucket': 1794824700, 'window': [1794823800, 1794825000], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794802200&to=1794825000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794802200&to=1794825000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794825640, 'age_seconds': 537, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.4, 'status': 'open', 'first_seen': 1788934840, 'last_seen': 1794826161, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 426, 'narrative': "docproc's request rate is 23.6 req/min on production, above a normal of 0.1 req/min ± 0.1 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 158.5, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794826161, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5891338, 'stale_s': 17}
- 2026-11-16 12:29:31 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 28.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 11:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 28.2, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.64, 'rel': inf, 'buckets_hit': 3, 'n': 53, 'weeks': 5, 'slot': 'Mon 11:00 UTC', 'bucket': 1794829200, 'window': [1794828300, 1794829500], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794806700&to=1794829500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794806700&to=1794829500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794830130, 'age_seconds': 698, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 19.9, 'status': 'resolved', 'first_seen': 1788934840, 'last_seen': 1794830812, 'resolved_at': 1794832171, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 429, 'narrative': "docproc's request rate is 28.2 req/min on production, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794826161, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5897347, 'stale_s': 1375}
- 2026-11-16 18:07:03 UTCreopened x-ray · severity=info was=info title=docproc request rate 55.6 req/min vs a normal 1.7 req/min ± 2.5 req/min for Mon 17:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 55.6 req/min vs a normal 1.7 req/min ± 2.5 req/min for Mon 17:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 55.6, 'expected_median': 1.7, 'expected_mad': 2.52042, 'z': 21.38532466811087, 'rel': 31.705882352941178, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 17:00 UTC', 'bucket': 1794851700, 'window': [1794850800, 1794852000], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794829200&to=1794852000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794829200&to=1794852000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794852385, 'age_seconds': 51, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.7, 'status': 'open', 'first_seen': 1788934840, 'last_seen': 1794852423, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 430, 'narrative': "docproc's request rate is 28.2 req/min on production, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794852423, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5917597, 'stale_s': 14}
- 2026-11-16 18:24:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 55.6 req/min vs a normal 1.7 req/min ± 2.5 req/min for Mon 17:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 55.6, 'expected_median': 1.7, 'expected_mad': 2.52042, 'z': 21.38532466811087, 'rel': 31.705882352941178, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 17:00 UTC', 'bucket': 1794851700, 'window': [1794850800, 1794852000], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794829200&to=1794852000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794829200&to=1794852000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794852385, 'age_seconds': 51, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 17.7, 'status': 'resolved', 'first_seen': 1788934840, 'last_seen': 1794852423, 'resolved_at': 1794853467, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 430, 'narrative': "docproc's request rate is 55.6 req/min on production, above a normal of 1.7 req/min ± 2.5 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 21.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794852423, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5918643, 'stale_s': 1060}
- 2026-11-16 18:32:09 UTCreopened x-ray · severity=info was=info title=docproc request rate 35.0 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 18:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 35.0 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 18:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 35.0, 'expected_median': 0.4, 'expected_mad': 0.59304, 'z': 58.34345069472548, 'rel': 86.5, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 18:00 UTC', 'bucket': 1794853200, 'window': [1794852300, 1794853500], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794830700&to=1794853500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794830700&to=1794853500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794853896, 'age_seconds': 46, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.0, 'status': 'open', 'first_seen': 1788934840, 'last_seen': 1794853929, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 431, 'narrative': "docproc's request rate is 55.6 req/min on production, above a normal of 1.7 req/min ± 2.5 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 21.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5919103, 'stale_s': 14}
- 2026-11-16 19:33:28 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1311, 'fingerprint': '9bf9442311ba102f6466', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc request rate 5.6 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 18:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 5.6, 'expected_median': 0.4, 'expected_mad': 0.59304, 'z': 8.768379873195736, 'rel': 12.999999999999998, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 18:00 UTC', 'bucket': 1794855300, 'window': [1794854700, 1794855900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docproc?cluster=production&from=1794833100&to=1794855900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=production&from=1794833100&to=1794855900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794856301, 'age_seconds': 193, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 18.1, 'status': 'resolved', 'first_seen': 1788934840, 'last_seen': 1794856481, 'resolved_at': 1794857608, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 436, 'narrative': "docproc's request rate is 5.6 req/min on production, above a normal of 0.4 req/min ± 0.6 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 8.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': 'low', 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 5922783, 'stale_s': 1142}
Notifications
- 2026-11-16 19:33:49 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-16 18:25:37 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-16 12:30:09 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-16 10:20:04 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-16 09:50:53 UTCdigest-itemin-app,prod-notifications · sent
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794856301 |
| age_seconds | 193 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |