Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 6 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:40:02.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min

critical Endpoint failing master resolved impact 73 Owner: Madhava Mukkamala (backup: Shashank Agrawal) opened 33d ago · last observed 1d ago · observed 11× · #4930 · bc9026a1ed6317cb02e3

What happened template

execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 473 of 473 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; execsvc POST /api/trialexec/v2/study/:id/activity/status failed 655 of 655 calls (100 %, all 5xx) in 15 min. Onset was 04:00 UTC; errors start at 04:05 UTC.
Each failure is a 500 from activity_details_v2.go:458: 'could not match participant question to activity graph', with an empty refId, variableName and title (sample questionId 5571182f-6c21-491e-b8c6-51fce3166e1e). Sampled failures are all org smotest1.
The 5 execsvc deploys in the prior 24 h were config-only autosyncs with no image change. The last was at 01:33 UTC, 146 min before onset. No commit links to the failure.
The same error has recurred since 2026-10-16 across 5 earlier episodes. A new shape from execution/activity_details_v2.go:2007 appeared at 04:05 UTC.
Next step: check the smotest1 study data for question 5571182f-6c21-491e-b8c6-51fce3166e1e and why it has no match in the activity graph, starting at execution/activity_details_v2.go:2007.

low confidence

claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

endpointPOST /api/trialexec/v2/study/:id/activity/status
serviceexecsvc
calls473
failed473
c4xx0
fail_pct100.00
window_min15
discriminatorPOST /api/trialexec/v2/study/:id/activity/status
window2026-11-17 04:10:00 UTC → 2026-11-17 04:25:00 UTC

What is failing in execsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 04:10:00 UTC → 2026-11-17 04:25:00 UTC) there were 473; every line links to the grouped errors behind it.

POST /api/trialexec/v2/study/:id/activity/status 473 bad of 473 calls · 100% of the service's 5xx

The error shapes behind them

787× Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC
787× Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC
787× could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC
30× No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC
29× no result from branch evaluation since 2026-11-13 15:11:12 UTC

All execsvc errors, grouped → · service page · its logs

The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.

History

  • 2026-11-17 04:27:25 UTCreopened x-ray · severity=critical was=critical title=execsvc POST /api/trialexec/v2/study/:id/activity/status: 655 of 655 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4930, 'fingerprint': 'bc9026a1ed6317cb02e3', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/activity/status: 655 of 655 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'service': 'execsvc', 'calls': 655, 'failed': 655, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794888000, 1794888900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794888000&to=1794889645'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794888000&to=1794889645'}], 'provenance': {'source': 'rollup', 'collected_at': 1794889413, 'age_seconds': 240, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.2, 'status': 'open', 'first_seen': 1792153419, 'last_seen': 1794889645, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 145 of 208 calls (70 %) in the last 15 min. automated changed execsvc 54 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status', 'level': 'high', 'told_level': None, 'episode_at': 1794889645, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2736234, 'stale_s': 8}
  • 2026-11-17 06:15:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4930, 'fingerprint': 'bc9026a1ed6317cb02e3', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'service': 'execsvc', 'calls': 473, 'failed': 473, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794888600, 1794889500], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794888600&to=1794890757'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794888600&to=1794890757'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 649, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.9, 'status': 'resolved', 'first_seen': 1792153419, 'last_seen': 1794890757, 'resolved_at': 1794896105, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 473 of 473 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/activity/status', 'level': 'high', 'told_level': None, 'episode_at': 1794889645, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2742694, 'stale_s': 5356}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794890117
age_seconds649
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_failing