← findings
onboardingsvc request rate 8.0 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 13:00 UTC (3 of the last 4 checks)
info
Traffic anomaly
test26
resolved
impact 12
Owner: Prasad Limaye (backup: Sambasiva Bandarupalli)
opened 55d ago · last observed 2d ago · observed 114× · #2463 · cd9abbf51898b6108b65
What happened template
onboardingsvc's request rate is 8.0 req/min on test26, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 26.3, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 8.00 |
| expected_median | 0.20 |
| expected_mad | 0.30 |
| z | 26.31 |
| rel | 39.00 |
| buckets_hit | 3 |
| n | 57 |
| weeks | 5 |
| slot | Mon 13:00 UTC |
| bucket | 2026-11-16 13:35:00 UTC (2d ago) |
| discriminator | api_rpm |
| window | 2026-11-16 13:20:00 UTC → 2026-11-16 13:40:00 UTC |
History
- 2026-11-16 06:42:08 UTCreopened x-ray · severity=info was=info title=onboardingsvc request rate 6.8 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 06:00 UTC (4 of the last 4 checks) level=low was_level=low finding={'id': 2463, 'fingerprint': 'cd9abbf51898b6108b65', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 6.8 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 06:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 6.8, 'expected_median': 0.2, 'expected_mad': 0.29652, 'z': 22.25819506272764, 'rel': 32.99999999999999, 'buckets_hit': 4, 'n': 54, 'weeks': 5, 'slot': 'Mon 06:00 UTC', 'bucket': 1794810600, 'window': [1794809700, 1794810900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=test26&from=1794788100&to=1794810900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794788100&to=1794810900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811328, 'age_seconds': 18, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.1, 'status': 'open', 'first_seen': 1790284026, 'last_seen': 1794811328, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 112, 'narrative': "onboardingsvc's request rate is 17.2 req/min on test26, above a normal of 0.4 req/min ± 0.6 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 28.3, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 4527320, 'stale_s': 18}
- 2026-11-16 07:08:43 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2463, 'fingerprint': 'cd9abbf51898b6108b65', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 6.8 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 06:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 6.8, 'expected_median': 0.2, 'expected_mad': 0.29652, 'z': 22.25819506272764, 'rel': 32.99999999999999, 'buckets_hit': 3, 'n': 54, 'weeks': 5, 'slot': 'Mon 06:00 UTC', 'bucket': 1794810600, 'window': [1794810000, 1794811200], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=test26&from=1794788400&to=1794811200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794788400&to=1794811200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794811767, 'age_seconds': 17, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.1, 'status': 'resolved', 'first_seen': 1790284026, 'last_seen': 1794811767, 'resolved_at': 1794812923, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 113, 'narrative': "onboardingsvc's request rate is 6.8 req/min on test26, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 22.3, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794811328, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 4528914, 'stale_s': 1173}
- 2026-11-16 13:48:59 UTCreopened x-ray · severity=info was=info title=onboardingsvc request rate 8.0 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 13:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 2463, 'fingerprint': 'cd9abbf51898b6108b65', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 8.0 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 13:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 8.0, 'expected_median': 0.2, 'expected_mad': 0.29652, 'z': 26.305139619587212, 'rel': 39.0, 'buckets_hit': 3, 'n': 57, 'weeks': 5, 'slot': 'Mon 13:00 UTC', 'bucket': 1794836100, 'window': [1794835200, 1794836400], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=test26&from=1794813600&to=1794836400'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794813600&to=1794836400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794836938, 'age_seconds': 19, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.4, 'status': 'open', 'first_seen': 1790284026, 'last_seen': 1794836939, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 114, 'narrative': "onboardingsvc's request rate is 6.8 req/min on test26, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 22.3, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794836939, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 4552931, 'stale_s': 18}
- 2026-11-16 14:10:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2463, 'fingerprint': 'cd9abbf51898b6108b65', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'test26', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 8.0 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 13:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 8.0, 'expected_median': 0.2, 'expected_mad': 0.29652, 'z': 26.305139619587212, 'rel': 39.0, 'buckets_hit': 3, 'n': 57, 'weeks': 5, 'slot': 'Mon 13:00 UTC', 'bucket': 1794836100, 'window': [1794835200, 1794836400], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=test26&from=1794813600&to=1794836400'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test26&from=1794813600&to=1794836400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794836938, 'age_seconds': 19, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.4, 'status': 'resolved', 'first_seen': 1790284026, 'last_seen': 1794836939, 'resolved_at': 1794838227, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 114, 'narrative': "onboardingsvc's request rate is 8.0 req/min on test26, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 26.3, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794836939, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 4554219, 'stale_s': 1306}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794836938 |
| age_seconds | 19 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |