docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)
info
Traffic anomaly
test26
resolved
impact 17
Owner: Prasad Limaye (backup: Harish Kumar)
opened 20d ago · last observed 2d ago · observed 3× · #9031 · 8df05a00ea5283edc14b
What happened template
docsvc's request rate is 28.8 req/min on test26, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 28.80 |
| expected_median | 0.00 |
| expected_mad | 5.00 |
| z | 5.76 |
| rel | inf |
| buckets_hit | 3 |
| n | 54 |
| weeks | 5 |
| slot | Mon 06:00 UTC |
| bucket | 2026-11-16 06:30:00 UTC (2d ago) |
| discriminator | api_rpm |
| window | 2026-11-16 06:15:00 UTC → 2026-11-16 06:35:00 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 06:15:00 UTC → 2026-11-16 06:35:00 UTC) there were 1; every line links to the grouped errors behind it.
PATCH /api/documents/folders/subArtifacts/:id |
1 bad of 1 calls · 100% of the service's 5xx |
The error shapes behind them
| 156× | template backfill: could not move document — document has no primary version, cannot move to Templates since 2026-11-16 13:27:23 UTC |
| 2× | Request completed — 500 · tmfLevel \ since 2026-10-06 17:42:05 UTC |
| 1× | tmfLevel "Country" cannot be removed from this folder/subArtifact as it contains documents and placeholders since 2026-11-16 06:16:16 UTC |
History
- 2026-11-16 06:42:08 UTCreopened x-ray · severity=info was=info title=docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 9031, 'fingerprint': '8df05a00ea5283edc14b', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 28.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.76, 'rel': inf, 'buckets_hit': 3, 'n': 54, 'weeks': 5, 'slot': 'Mon 06:00 UTC', 'bucket': 1794810600, 'window': [1794809700, 1794810900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docsvc?cluster=test26&from=1794788100&to=1794810900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794788100&to=1794810900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811328, 'age_seconds': 18, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.4, 'status': 'open', 'first_seen': 1793235682, 'last_seen': 1794811328, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': "docsvc's request rate is 55.6 req/min on test26, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 11.1, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1575664, 'stale_s': 18}
- 2026-11-16 07:08:43 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 9031, 'fingerprint': '8df05a00ea5283edc14b', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 28.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.76, 'rel': inf, 'buckets_hit': 3, 'n': 54, 'weeks': 5, 'slot': 'Mon 06:00 UTC', 'bucket': 1794810600, 'window': [1794809700, 1794810900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/docsvc?cluster=test26&from=1794788100&to=1794810900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794788100&to=1794810900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811328, 'age_seconds': 18, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 17.4, 'status': 'resolved', 'first_seen': 1793235682, 'last_seen': 1794811328, 'resolved_at': 1794812923, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': "docsvc's request rate is 28.8 req/min on test26, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1577258, 'stale_s': 1612}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794811328 |
| age_seconds | 18 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |