← findings
onboardingsvc request rate 61.6 req/min vs a normal 6.3 req/min ± 9.3 req/min for Mon 12:00 UTC (3 of the last 4 checks)
info
Traffic anomaly
master
resolved
impact 16
Owner: Prasad Limaye (backup: Sambasiva Bandarupalli)
opened 83d ago · last observed 2d ago · observed 295× · #203 · aa8b51652446669f9f86
What happened template
onboardingsvc's request rate is 61.6 req/min on master, above a normal of 6.3 req/min ± 9.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 61.60 |
| expected_median | 6.30 |
| expected_mad | 9.34 |
| z | 5.92 |
| rel | 8.78 |
| buckets_hit | 3 |
| n | 48 |
| weeks | 4 |
| slot | Mon 12:00 UTC |
| bucket | 2026-11-16 12:35:00 UTC (2d ago) |
| discriminator | api_rpm |
| window | 2026-11-16 12:25:00 UTC → 2026-11-16 12:45:00 UTC |
History
- 2026-11-15 21:43:40 UTCreopened x-ray · severity=info was=info title=onboardingsvc request rate 8.4 req/min vs a normal 0.1 req/min ± 0.1 req/min for Sun 21:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 203, 'fingerprint': 'aa8b51652446669f9f86', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'master', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 8.4 req/min vs a normal 0.1 req/min ± 0.1 req/min for Sun 21:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 8.4, 'expected_median': 0.1, 'expected_mad': 0.14826, 'z': 55.982733036557406, 'rel': 83.0, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Sun 21:00 UTC', 'bucket': 1794777900, 'window': [1794777000, 1794778200], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=master&from=1794755400&to=1794778200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794755400&to=1794778200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794778810, 'age_seconds': 218, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 9.9, 'status': 'open', 'first_seen': 1787810049, 'last_seen': 1794779020, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 293, 'narrative': "onboardingsvc's request rate is 11.2 req/min on master, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 37.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794779020, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 6968979, 'stale_s': 8}
- 2026-11-15 22:10:50 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 203, 'fingerprint': 'aa8b51652446669f9f86', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'master', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 7.6 req/min vs a normal 0.1 req/min ± 0.1 req/min for Sun 21:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 7.6, 'expected_median': 0.1, 'expected_mad': 0.14826, 'z': 50.58680696074464, 'rel': 75.0, 'buckets_hit': 4, 'n': 48, 'weeks': 4, 'slot': 'Sun 21:00 UTC', 'bucket': 1794778500, 'window': [1794777600, 1794778800], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=master&from=1794756000&to=1794778800'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794756000&to=1794778800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794779447, 'age_seconds': 286, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.8, 'status': 'resolved', 'first_seen': 1787810049, 'last_seen': 1794779727, 'resolved_at': 1794780650, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 294, 'narrative': "onboardingsvc's request rate is 7.6 req/min on master, above a normal of 0.1 req/min ± 0.1 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 50.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794779020, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 6970608, 'stale_s': 930}
- 2026-11-16 12:58:26 UTCreopened x-ray · severity=info was=info title=onboardingsvc request rate 61.6 req/min vs a normal 6.3 req/min ± 9.3 req/min for Mon 12:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 203, 'fingerprint': 'aa8b51652446669f9f86', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'master', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 61.6 req/min vs a normal 6.3 req/min ± 9.3 req/min for Mon 12:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 61.6, 'expected_median': 6.3, 'expected_mad': 9.34038, 'z': 5.920529999850114, 'rel': 8.777777777777779, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 12:00 UTC', 'bucket': 1794832500, 'window': [1794831900, 1794833100], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=master&from=1794810300&to=1794833100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794810300&to=1794833100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794833691, 'age_seconds': 225, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 10.7, 'status': 'open', 'first_seen': 1787810049, 'last_seen': 1794833906, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 295, 'narrative': "onboardingsvc's request rate is 7.6 req/min on master, above a normal of 0.1 req/min ± 0.1 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 50.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794833906, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7023867, 'stale_s': 10}
- 2026-11-16 13:13:39 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 203, 'fingerprint': 'aa8b51652446669f9f86', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'master', 'subject': 'onboardingsvc', 'title': 'onboardingsvc request rate 61.6 req/min vs a normal 6.3 req/min ± 9.3 req/min for Mon 12:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 61.6, 'expected_median': 6.3, 'expected_mad': 9.34038, 'z': 5.920529999850114, 'rel': 8.777777777777779, 'buckets_hit': 3, 'n': 48, 'weeks': 4, 'slot': 'Mon 12:00 UTC', 'bucket': 1794832500, 'window': [1794831900, 1794833100], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/onboardingsvc?cluster=master&from=1794810300&to=1794833100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794810300&to=1794833100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794833691, 'age_seconds': 225, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.7, 'status': 'resolved', 'first_seen': 1787810049, 'last_seen': 1794833906, 'resolved_at': 1794834819, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 295, 'narrative': "onboardingsvc's request rate is 61.6 req/min on master, above a normal of 6.3 req/min ± 9.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794833906, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 7024782, 'stale_s': 925}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794833691 |
| age_seconds | 225 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |