execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min
critical
Endpoint failing
master
resolved
impact 73
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 33d ago · last observed 1d ago · observed 11× · #4930 · bc9026a1ed6317cb02e3
What happened template
execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 473 of 473 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.
One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; execsvc POST /api/trialexec/v2/study/:id/activity/status failed 655 of 655 calls (100 %, all 5xx) in 15 min. Onset was 04:00 UTC; errors start at 04:05 UTC.
Each failure is a 500 from activity_details_v2.go:458: 'could not match participant question to activity graph', with an empty refId, variableName and title (sample questionId 5571182f-6c21-491e-b8c6-51fce3166e1e). Sampled failures are all org smotest1.
The 5 execsvc deploys in the prior 24 h were config-only autosyncs with no image change. The last was at 01:33 UTC, 146 min before onset. No commit links to the failure.
The same error has recurred since 2026-10-16 across 5 earlier episodes. A new shape from execution/activity_details_v2.go:2007 appeared at 04:05 UTC.
Next step: check the smotest1 study data for question 5571182f-6c21-491e-b8c6-51fce3166e1e and why it has no match in the activity graph, starting at execution/activity_details_v2.go:2007.
low confidence
claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | POST /api/trialexec/v2/study/:id/activity/status |
| service | execsvc |
| calls | 473 |
| failed | 473 |
| c4xx | 0 |
| fail_pct | 100.00 |
| window_min | 15 |
| discriminator | POST /api/trialexec/v2/study/:id/activity/status |
| window | 2026-11-17 04:10:00 UTC → 2026-11-17 04:25:00 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 04:10:00 UTC → 2026-11-17 04:25:00 UTC) there were 473; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/activity/status |
473 bad of 473 calls · 100% of the service's 5xx |
The error shapes behind them
| 787× | Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC |
| 787× | Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC |
| 787× | could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC |
| 30× | No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC |
| 29× | no result from branch evaluation since 2026-11-13 15:11:12 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-10-16 12:23:39 UTCthis finding execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min
- 2026-10-28 12:30:08 UTCfinding · info execsvc runs 26 foreground Neo4j queries per request (3,407 queries / 130 calls, last hour)
- 2026-11-14 15:19:13 UTCdeploy automated deployed execsvc
- 2026-11-15 00:59:05 UTCdeploy automated deployed execsvc
- 2026-11-15 01:29:15 UTCdeploy automated deployed execsvc
- 2026-11-15 08:13:26 UTCfinding · warning execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)
- 2026-11-16 00:00:00 UTClatency execsvc p95 peaked at 1.89 s
- 2026-11-16 01:19:29 UTCdeploy automated deployed execsvc
- 2026-11-16 01:27:59 UTCdeploy automated deployed execsvc
- 2026-11-16 05:19:28 UTCdeploy automated deployed execsvc
- 2026-11-16 07:34:54 UTCfinding · warning execsvc: a new error shape, 44× in the last hour, never seen in 90 days (pubsub not initialized, dropping onboarding event)
- 2026-11-16 08:05:51 UTCdeploy automated deployed execsvc
- 2026-11-16 15:39:46 UTCdeploy automated deployed execsvc
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T)
- 2026-11-16 18:21:13 UTCdeploy automated deployed execsvc
- 2026-11-17 00:00:00 UTCerrors execsvc 5xx peaked at 787 in one five-minute bucket (median 0 in this window)
- 2026-11-17 01:33:37 UTCdeploy automated deployed execsvc
History
- 2026-11-17 04:27:25 UTCreopened x-ray · severity=critical was=critical title=execsvc POST /api/trialexec/v2/study/:id/activity/status: 655 of 655 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4930, 'fingerprint': 'bc9026a1ed6317cb02e3', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/activity/status: 655 of 655 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'service': 'execsvc', 'calls': 655, 'failed': 655, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794888000, 1794888900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794888000&to=1794889645'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794888000&to=1794889645'}], 'provenance': {'source': 'rollup', 'collected_at': 1794889413, 'age_seconds': 240, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.2, 'status': 'open', 'first_seen': 1792153419, 'last_seen': 1794889645, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 145 of 208 calls (70 %) in the last 15 min. automated changed execsvc 54 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status', 'level': 'high', 'told_level': None, 'episode_at': 1794889645, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2736234, 'stale_s': 8}
- 2026-11-17 06:15:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4930, 'fingerprint': 'bc9026a1ed6317cb02e3', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'service': 'execsvc', 'calls': 473, 'failed': 473, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794888600, 1794889500], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794888600&to=1794890757'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794888600&to=1794890757'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 649, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.9, 'status': 'resolved', 'first_seen': 1792153419, 'last_seen': 1794890757, 'resolved_at': 1794896105, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 473 of 473 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status', 'level': 'high', 'told_level': None, 'episode_at': 1794889645, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2742694, 'stale_s': 5356}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794890117 |
| age_seconds | 649 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_failing |