execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)
critical
Error-rate anomaly
master
resolved
impact 62
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 1d ago · last observed 1d ago · observed 1× · #17504 · 8593c9721f3d6c05f491
What happened ai
execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.
Error percentage far above its hour-of-week normal.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls, putting execsvc 5xx at 89.19% against a normal 0.00%, since 2026-11-17 04:05 UTC.
All 700 error lines are one failure: "could not match participant question to activity graph" (execution/activity_details_v2.go:2007), sample questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with empty refId, variableName and title.
Checked: the 5 execsvc deploys in the prior 24 h were config-only autosyncs (last d8d21151cd28 at 01:33 UTC, 151 min before onset) and brought no commits, so nothing links them to the failure.
The empty refId/variableName/title points at study or activity-graph data rather than code; a smaller episode on 2026-11-09 (finding 16947) hit the same endpoint and was resolved.
Next step: open the trace, find the study whose question 5571182f-... is missing from its activity graph, and check what changed in that study's design around 04:05 UTC.
low confidence
claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| measure | err5_pct |
| unit | % |
| value | 89.19 |
| expected_median | 0.00 |
| expected_mad | 1.00 |
| z | 89.19 |
| rel | inf |
| buckets_hit | 3 |
| n | 24 |
| weeks | 4 |
| slot | Tue 04:00 UTC |
| bucket | 2026-11-17 04:15:00 UTC (1d ago) |
| failing_endpoints | {'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787} |
| discriminator | err5_pct |
| window | 2026-11-17 04:05:00 UTC → 2026-11-17 04:25:00 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 04:05:00 UTC → 2026-11-17 04:25:00 UTC) there were 787; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/activity/status |
787 bad of 787 calls · 100% of the service's 5xx |
The error shapes behind them
| 787× | Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC |
| 787× | Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC |
| 787× | could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC |
| 30× | No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC |
| 29× | no result from branch evaluation since 2026-11-13 15:11:12 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-11-17 04:00:00 UTCerrors execsvc 5xx peaked at 787 in one five-minute bucket (median 0 in this window)
- 2026-11-17 04:16:30 UTCfinding · critical execsvc: a new error shape, 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to)
- 2026-11-17 04:45:57 UTCthis finding execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)
History
- 2026-11-17 04:45:57 UTCopened x-ray · severity=critical title=execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx) finding={'id': 17504, 'fingerprint': '8593c9721f3d6c05f491', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 89.1891891891892, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 89.1891891891892, 'rel': inf, 'buckets_hit': 3, 'n': 24, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 650, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.9, 'status': 'open', 'first_seen': 1794890757, 'last_seen': 1794890757, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 10, 'stale_s': 10}
- 2026-11-17 04:46:53 UTCnotified x-ray · alert_id=3 channel=in-app status=delivered send_id=52005
- 2026-11-17 04:46:53 UTCnotified x-ray · alert_id=3 channel=dev-notifications status=sent send_id=52005
- 2026-11-17 05:34:42 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17504, 'fingerprint': '8593c9721f3d6c05f491', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 89.1891891891892, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 89.1891891891892, 'rel': inf, 'buckets_hit': 3, 'n': 24, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 650, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.9, 'status': 'resolved', 'first_seen': 1794890757, 'last_seen': 1794890757, 'resolved_at': 1794893682, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': 'execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.', 'narrative_kind': 'ai', 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 2934, 'stale_s': 2934}
Notifications
- 2026-11-17 04:46:53 UTCopenedin-app · delivered
- 2026-11-17 04:46:53 UTCopeneddev-notifications · sent
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794890117 |
| age_seconds | 650 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |