← findings
workplansvc request rate 27.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)
info
Traffic anomaly
automation
resolved
impact 12
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 18d ago · last observed 1d ago · observed 22× · #12690 · d12f3c88a9830c64b436
What happened template
workplansvc's request rate is 27.2 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.4, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.
Requests per minute far outside what this hour of the week normally sees.
The facts
| measure | api_rpm |
| unit | req/min |
| value | 27.20 |
| expected_median | 0.00 |
| expected_mad | 5.00 |
| z | 5.44 |
| rel | inf |
| buckets_hit | 3 |
| n | 54 |
| weeks | 5 |
| slot | Tue 10:00 UTC |
| bucket | 2026-11-17 10:00:00 UTC (1d ago) |
| discriminator | api_rpm |
| window | 2026-11-17 09:45:00 UTC → 2026-11-17 10:05:00 UTC |
History
- 2026-11-16 10:19:38 UTCreopened x-ray · severity=info was=info title=workplansvc request rate 29.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 09:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 29.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 09:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 29.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.96, 'rel': inf, 'buckets_hit': 3, 'n': 56, 'weeks': 5, 'slot': 'Mon 09:00 UTC', 'bucket': 1794822600, 'window': [1794821700, 1794822900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794800100&to=1794822900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794800100&to=1794822900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794823497, 'age_seconds': 898, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.8, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794824378, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 17, 'narrative': "workplansvc's request rate is 34.0 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794824378, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1379337, 'stale_s': 17}
- 2026-11-16 10:49:21 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 29.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 09:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 29.8, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.96, 'rel': inf, 'buckets_hit': 3, 'n': 56, 'weeks': 5, 'slot': 'Mon 09:00 UTC', 'bucket': 1794822600, 'window': [1794821700, 1794822900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794800100&to=1794822900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794800100&to=1794822900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794823497, 'age_seconds': 898, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.8, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794824378, 'resolved_at': 1794826161, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 17, 'narrative': "workplansvc's request rate is 29.8 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794824378, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1381117, 'stale_s': 1797}
- 2026-11-16 12:06:52 UTCreopened x-ray · severity=info was=info title=workplansvc request rate 33.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 11:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 33.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 11:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 33.4, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 6.68, 'rel': inf, 'buckets_hit': 3, 'n': 58, 'weeks': 6, 'slot': 'Mon 11:00 UTC', 'bucket': 1794828900, 'window': [1794828000, 1794829200], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794806400&to=1794829200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794806400&to=1794829200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794829780, 'age_seconds': 1046, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.9, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794830812, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': "workplansvc's request rate is 29.8 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1385768, 'stale_s': 14}
- 2026-11-16 12:29:31 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 33.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 11:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 33.4, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 6.68, 'rel': inf, 'buckets_hit': 3, 'n': 58, 'weeks': 6, 'slot': 'Mon 11:00 UTC', 'bucket': 1794828900, 'window': [1794828000, 1794829200], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794806400&to=1794829200'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794806400&to=1794829200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794829780, 'age_seconds': 1046, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.9, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794830812, 'resolved_at': 1794832171, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': "workplansvc's request rate is 33.4 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.7, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1387127, 'stale_s': 1373}
- 2026-11-17 06:00:02 UTCreopened x-ray · severity=info was=info title=workplansvc request rate 41.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 05:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 41.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 41.2, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 8.24, 'rel': inf, 'buckets_hit': 3, 'n': 55, 'weeks': 5, 'slot': 'Tue 05:00 UTC', 'bucket': 1794894000, 'window': [1794893100, 1794894300], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794871500&to=1794894300'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794871500&to=1794894300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794894820, 'age_seconds': 394, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.8, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794895202, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 19, 'narrative': "workplansvc's request rate is 33.4 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.7, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794895202, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1450156, 'stale_s': 12}
- 2026-11-17 06:55:37 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 35.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 06:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 35.2, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 7.040000000000001, 'rel': inf, 'buckets_hit': 4, 'n': 55, 'weeks': 5, 'slot': 'Tue 06:00 UTC', 'bucket': 1794895800, 'window': [1794894900, 1794896100], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794873300&to=1794896100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794873300&to=1794896100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794896645, 'age_seconds': 322, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.0, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794896955, 'resolved_at': 1794898537, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 21, 'narrative': "workplansvc's request rate is 35.2 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 7.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794895202, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1453490, 'stale_s': 1593}
- 2026-11-17 10:21:45 UTCreopened x-ray · severity=info was=info title=workplansvc request rate 27.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 27.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 27.2, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.4399999999999995, 'rel': inf, 'buckets_hit': 3, 'n': 54, 'weeks': 5, 'slot': 'Tue 10:00 UTC', 'bucket': 1794909600, 'window': [1794908700, 1794909900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794887100&to=1794909900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794887100&to=1794909900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794910460, 'age_seconds': 462, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.5, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794910905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 22, 'narrative': "workplansvc's request rate is 35.2 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 7.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794910905, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1465864, 'stale_s': 17}
- 2026-11-17 10:40:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12690, 'fingerprint': 'd12f3c88a9830c64b436', 'kind': 'anomaly.traffic', 'severity': 'info', 'cluster': 'automation', 'subject': 'workplansvc', 'title': 'workplansvc request rate 27.2 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_rpm', 'unit': 'req/min', 'value': 27.2, 'expected_median': 0.0, 'expected_mad': 5.0, 'z': 5.4399999999999995, 'rel': inf, 'buckets_hit': 3, 'n': 54, 'weeks': 5, 'slot': 'Tue 10:00 UTC', 'bucket': 1794909600, 'window': [1794908700, 1794909900], 'recent_deploys': [], 'discriminator': 'api_rpm'}, 'evidence': [{'label': 'Service', 'href': '/service/workplansvc?cluster=automation&from=1794887100&to=1794909900'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794887100&to=1794909900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794910460, 'age_seconds': 462, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.5, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794910905, 'resolved_at': 1794912041, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 22, 'narrative': "workplansvc's request rate is 27.2 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.4, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'api_rpm', 'level': 'low', 'told_level': None, 'episode_at': 1794910905, 'chat_thread': None, 'kind_label': 'Traffic anomaly', 'group': 'Anomalies', 'age_s': 1466999, 'stale_s': 1152}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794910460 |
| age_seconds | 462 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |