← findings
execsvc request rate 143.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 07:00 UTC (4 of the last 4 checks)
info
Traffic anomaly
test27
resolved
impact 12
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 83d ago · last observed 1d ago · observed 14× · #214 · 146a3d9967e16e043831
What happened template
execsvc's request rate is 143.8 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 28.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 143.80 |
| expected_median | 0.00 |
| expected_mad | 5.00 |
| z | 28.76 |
| rel | inf |
| buckets_hit | 4 |
| n | 55 |
| weeks | 5 |
| slot | Tue 07:00 UTC |
| bucket | 2026-11-17 07:20:00 UTC (1d ago) |
| discriminator | api_rpm |
| window | 2026-11-17 07:05:00 UTC → 2026-11-17 07:25:00 UTC |
History
- 2026-11-17 06:55:37 UTCreopened x-ray · severity=info was=info title=execsvc request rate 48.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 06:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 214, 'fingerprint': '146a3d9967e16e043831', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test27', 'subject': 'execsvc', 'title': 'execsvc request rate 48.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 06:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 48.0, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 9.6, 'rel': inf, 'buckets_hit': 3, 'n': 55, 'weeks': 5, 'slot': 'Tue 06:00 UTC', 'bucket': 1794897300, 'window': [1794896400, 1794897600], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=test27&from=1794874800&to=1794897600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794874800&to=1794897600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794898379, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.6, 'status': 'open', 'first_seen': 1787813369, 'last_seen': 1794898537, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 13, 'narrative': "execsvc's request rate is 201.0 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 40.2, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794898537, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7085183, 'stale_s': 15}
- 2026-11-17 07:24:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 214, 'fingerprint': '146a3d9967e16e043831', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test27', 'subject': 'execsvc', 'title': 'execsvc request rate 48.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 06:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 48.0, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 9.6, 'rel': inf, 'buckets_hit': 3, 'n': 55, 'weeks': 5, 'slot': 'Tue 06:00 UTC', 'bucket': 1794897300, 'window': [1794896400, 1794897600], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=test27&from=1794874800&to=1794897600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794874800&to=1794897600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794898379, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.6, 'status': 'resolved', 'first_seen': 1787813369, 'last_seen': 1794898537, 'resolved_at': 1794900249, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 13, 'narrative': "execsvc's request rate is 48.0 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 9.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794898537, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7086898, 'stale_s': 1730}
- 2026-11-17 07:42:20 UTCreopened x-ray · severity=info was=info title=execsvc request rate 143.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 07:00 UTC (4 of the last 4 checks) level=low was_level=low finding={'id': 214, 'fingerprint': '146a3d9967e16e043831', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test27', 'subject': 'execsvc', 'title': 'execsvc request rate 143.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 07:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 143.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 28.76, 'rel': inf, 'buckets_hit': 4, 'n': 55, 'weeks': 5, 'slot': 'Tue 07:00 UTC', 'bucket': 1794900000, 'window': [1794899100, 1794900300], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=test27&from=1794877500&to=1794900300'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794877500&to=1794900300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794901057, 'age_seconds': 301, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.5, 'status': 'open', 'first_seen': 1787813369, 'last_seen': 1794901340, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 14, 'narrative': "execsvc's request rate is 48.0 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 9.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794901340, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7087990, 'stale_s': 19}
- 2026-11-17 08:04:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 214, 'fingerprint': '146a3d9967e16e043831', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test27', 'subject': 'execsvc', 'title': 'execsvc request rate 143.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 07:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 143.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 28.76, 'rel': inf, 'buckets_hit': 4, 'n': 55, 'weeks': 5, 'slot': 'Tue 07:00 UTC', 'bucket': 1794900000, 'window': [1794899100, 1794900300], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=test27&from=1794877500&to=1794900300'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794877500&to=1794900300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794901057, 'age_seconds': 301, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.5, 'status': 'resolved', 'first_seen': 1787813369, 'last_seen': 1794901340, 'resolved_at': 1794902649, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 14, 'narrative': "execsvc's request rate is 143.8 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 28.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794901340, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7089298, 'stale_s': 1327}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794901057 |
| age_seconds | 301 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |