execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)
info
Latency anomaly
automation
resolved
impact 14
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 8d ago · last observed 1d ago · observed 9× · #16997 · e9268f5abb3c0bb1c6b7
What happened template
execsvc's p95 is 1.5 s on automation, above a normal of 472 ms ± 178 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 5.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.
p95 far above its hour-of-week normal — and at least 1 s; under a second is noise.
The facts
| measure | api_p95_ms |
| unit | ms |
| value | 1,525 |
| expected_median | 471.83 |
| expected_mad | 178.06 |
| z | 5.91 |
| rel | 2.23 |
| buckets_hit | 4 |
| n | 29 |
| weeks | 5 |
| slot | Tue 05:00 UTC |
| bucket | 2026-11-17 05:20:00 UTC (1d ago) |
| floor_ms | 1,000 |
| discriminator | api_p95_ms |
| window | 2026-11-17 05:05:00 UTC → 2026-11-17 05:25:00 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 05:05:00 UTC → 2026-11-17 05:25:00 UTC) there were 1; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/activity/v2/complete |
1 bad of 9 calls · 100% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 20× | could not list DutyPlans for Study d9b557d0-7319-45b6-b4fb-57e2ec1298b1: passed Study was not found since 2026-10-08 12:44:23 UTC |
| 4× | failed to create Actor for Study error: cannot modify actors for site 78adecaa-0c77-4b00-9595-1a27ec13cd01: it is owned … since 2026-08-26 01:01:46 UTC |
| 4× | failed to create Actor for Study — cannot modify actors for site d8c7a8b4-daaf-44c9-b09b-a76b9cac399d: it is owned by a … since 2026-10-30 00:45:57 UTC |
| 4× | failed to get Adopted ProtocolVersion for site and plan definition since 2026-11-17 05:57:21 UTC |
History
- 2026-11-17 05:26:24 UTCreopened x-ray · severity=info was=info title=execsvc p95 1.94 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16997, 'fingerprint': 'e9268f5abb3c0bb1c6b7', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc p95 1.94 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1940.9090909090903, 'expected_median': 471.82925, 'expected_mad': 178.05803, 'z': 8.250567755405866, 'rel': 3.1135836553352516, 'buckets_hit': 3, 'n': 29, 'weeks': 5, 'slot': 'Tue 05:00 UTC', 'bucket': 1794892200, 'window': [1794891300, 1794892500], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=automation&from=1794869700&to=1794892500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794869700&to=1794892500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794893090, 'age_seconds': 104, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 9.4, 'status': 'open', 'first_seen': 1794321212, 'last_seen': 1794893184, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': "execsvc's p95 is 1.7 s on automation, above a normal of 263 ms ± 348 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 4.0, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794893184, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 571983, 'stale_s': 11}
- 2026-11-17 06:00:02 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16997, 'fingerprint': 'e9268f5abb3c0bb1c6b7', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1524.9999999999986, 'expected_median': 471.82925, 'expected_mad': 178.05803, 'z': 5.9147613280906155, 'rel': 2.232101443477696, 'buckets_hit': 4, 'n': 29, 'weeks': 5, 'slot': 'Tue 05:00 UTC', 'bucket': 1794892800, 'window': [1794891900, 1794893100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=automation&from=1794870300&to=1794893100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794870300&to=1794893100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794893590, 'age_seconds': 102, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 14.3, 'status': 'resolved', 'first_seen': 1794321212, 'last_seen': 1794893682, 'resolved_at': 1794895202, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "execsvc's p95 is 1.5 s on automation, above a normal of 472 ms ± 178 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 5.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794893184, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 574002, 'stale_s': 1532}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794893590 |
| age_seconds | 102 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |