execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)
info
Latency anomaly
master
resolved
impact 14
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 70d ago · last observed 2d ago · observed 26× · #1326 · fd5ca62348e139dfffe8
What happened template
execsvc's p95 is 4.2 s on master, above a normal of 380 ms ± 282 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 13.7, 4 weeks of history). automated changed execsvc 52 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.
p95 far above its hour-of-week normal — and at least 1 s; under a second is noise.
The facts
| measure | api_p95_ms |
| unit | ms |
| value | 4,250 |
| expected_median | 380.00 |
| expected_mad | 281.69 |
| z | 13.74 |
| rel | 10.18 |
| buckets_hit | 3 |
| n | 33 |
| weeks | 4 |
| slot | Mon 16:00 UTC |
| bucket | 2026-11-16 16:10:00 UTC (2d ago) |
| floor_ms | 1,000 |
| discriminator | api_p95_ms |
| window | 2026-11-16 16:00:00 UTC → 2026-11-16 16:20:00 UTC |
Adjacent deploys — adjacency, not proof
| automated | execsvc · 52 min before |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 16:00:00 UTC → 2026-11-16 16:20:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 787× | Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC |
| 787× | Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC |
| 787× | could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC |
| 30× | No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC |
| 29× | no result from branch evaluation since 2026-11-13 15:11:12 UTC |
History
- 2026-11-16 14:58:48 UTCreopened x-ray · severity=info was=info title=execsvc p95 4.68 s vs a normal 457 ms ± 388 ms for Mon 14:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 1326, 'fingerprint': 'fd5ca62348e139dfffe8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc p95 4.68 s vs a normal 457 ms ± 388 ms for Mon 14:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 4684.999999999998, 'expected_median': 456.875, 'expected_mad': 388.2559, 'z': 10.890047002505302, 'rel': 9.25444596443228, 'buckets_hit': 3, 'n': 26, 'weeks': 4, 'slot': 'Mon 14:00 UTC', 'bucket': 1794840000, 'window': [1794839400, 1794840600], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794817800&to=1794840600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794817800&to=1794840600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794841006, 'age_seconds': 131, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.5, 'status': 'open', 'first_seen': 1788950474, 'last_seen': 1794841128, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 24, 'narrative': "execsvc's p95 is 1.7 s on master, above a normal of 485 ms ± 252 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 4.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794841128, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 5890663, 'stale_s': 9}
- 2026-11-16 15:36:38 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1326, 'fingerprint': 'fd5ca62348e139dfffe8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc p95 4.68 s vs a normal 457 ms ± 388 ms for Mon 14:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 4684.999999999998, 'expected_median': 456.875, 'expected_mad': 388.2559, 'z': 10.890047002505302, 'rel': 9.25444596443228, 'buckets_hit': 3, 'n': 26, 'weeks': 4, 'slot': 'Mon 14:00 UTC', 'bucket': 1794840000, 'window': [1794839400, 1794840600], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794817800&to=1794840600'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794817800&to=1794840600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794841006, 'age_seconds': 131, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.5, 'status': 'resolved', 'first_seen': 1788950474, 'last_seen': 1794841128, 'resolved_at': 1794843398, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 24, 'narrative': "execsvc's p95 is 4.7 s on master, above a normal of 457 ms ± 388 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 10.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794841128, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 5892932, 'stale_s': 2278}
- 2026-11-16 16:22:34 UTCreopened x-ray · severity=info was=info title=execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 1326, 'fingerprint': 'fd5ca62348e139dfffe8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 4250.0, 'expected_median': 380.0, 'expected_mad': 281.694, 'z': 13.738311785128543, 'rel': 10.18421052631579, 'buckets_hit': 3, 'n': 33, 'weeks': 4, 'slot': 'Mon 16:00 UTC', 'bucket': 1794845400, 'window': [1794844500, 1794845700], 'floor_ms': 1000, 'recent_deploys': [{'who': 'automated', 'resource': 'execsvc', 'image': None, 'revision': '29620eb7090eebd0ca7163840ec0fdd716835b19', 'ts': 1794843586, 'minutes_before': 42}], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794822900&to=1794845700'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794822900&to=1794845700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846070, 'age_seconds': 92, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.5, 'status': 'open', 'first_seen': 1788950474, 'last_seen': 1794846154, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 25, 'narrative': "execsvc's p95 is 4.7 s on master, above a normal of 457 ms ± 388 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 10.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794846154, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 5895688, 'stale_s': 8}
- 2026-11-16 16:55:04 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1326, 'fingerprint': 'fd5ca62348e139dfffe8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 4250.0, 'expected_median': 380.0, 'expected_mad': 281.694, 'z': 13.738311785128543, 'rel': 10.18421052631579, 'buckets_hit': 3, 'n': 33, 'weeks': 4, 'slot': 'Mon 16:00 UTC', 'bucket': 1794845400, 'window': [1794844800, 1794846000], 'floor_ms': 1000, 'recent_deploys': [{'who': 'automated', 'resource': 'execsvc', 'image': None, 'revision': '29620eb7090eebd0ca7163840ec0fdd716835b19', 'ts': 1794843586, 'minutes_before': 52}], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794823200&to=1794846000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794823200&to=1794846000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846578, 'age_seconds': 153, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.5, 'status': 'resolved', 'first_seen': 1788950474, 'last_seen': 1794846721, 'resolved_at': 1794848104, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 26, 'narrative': "execsvc's p95 is 4.2 s on master, above a normal of 380 ms ± 282 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 13.7, 4 weeks of history). automated changed execsvc 52 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794846154, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 5897642, 'stale_s': 1395}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794846578 |
| age_seconds | 153 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |