execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective
warning
Error budget burning
master
resolved
impact 39
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 84d ago · last observed 1d ago · observed 233× · #163 · 019ea64d73a946c7a704
What happened template
execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.
Bad events are consuming the error budget faster than the objective allows.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; execsvc failed 314 of 704 calls (all 5xx, 55.4% vs a 95% objective), burning 18.0× (1 h) / 8.9× (6 h), budget at 0%, p95 2632 ms.
The finding opened at 2026-11-17 04:16 UTC; this is a recurring finding with 5 earlier episodes this month.
Checked: no error lines stored for 03:46–04:16 UTC, no other findings at the same time, and the 5 deploys in the prior 24 h were config-only autosyncs (last at 01:33 UTC, config d8d21151cd28, 162 min before onset) with no code change.
An earlier first look (2026-11-06) tied a similar burst to Neo4j DeadlockDetected during changelog inserts, but nothing in this window confirms that.
Next step: open the execsvc service page for the window to see which endpoint returns the 500s, then pull that pod's logs directly since none were stored.
low confidence
claude-opus-5-5 · 1d ago · $0.03 — every link above is evidence it read; nothing else was claimed.
The facts
| objective | 95.00 |
| objective_source | derived |
| achieved | 58.69 |
| calls | 1,905 |
| bad | 787.00 |
| c5xx | 787.00 |
| budget | 95.25 |
| spent | 8.26 |
| remaining_pct | 0.00 |
| burn_1h | 0.00 |
| burn_6h | 8.26 |
| verdict | burning |
| p95_ms | 704.41 |
| discriminator | |
| window | 2026-11-17 03:59:37 UTC → 2026-11-17 09:59:37 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 03:59:37 UTC → 2026-11-17 09:59:37 UTC) there were 787; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/activity/status |
787 bad of 951 calls · 100% of the service's 5xx |
The error shapes behind them
| 787× | Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC |
| 787× | Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC |
| 787× | could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC |
| 30× | No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC |
| 29× | no result from branch evaluation since 2026-11-13 15:11:12 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-08-26 17:40:08 UTCthis finding execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective
- 2026-08-26 18:55:17 UTCfinding · info execsvc: p99 latency (ms) 7.08 s — 12.9× the fleet's usual 550 ms (also p95 latency (ms) 4.01 s)
- 2026-09-09 10:41:14 UTCfinding · info execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)
- 2026-10-16 12:23:39 UTCfinding · critical execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min
- 2026-10-28 12:30:08 UTCfinding · info execsvc runs 26 foreground Neo4j queries per request (3,407 queries / 130 calls, last hour)
- 2026-11-14 15:19:13 UTCdeploy automated deployed execsvc
- 2026-11-15 00:59:05 UTCdeploy automated deployed execsvc
- 2026-11-15 01:29:15 UTCdeploy automated deployed execsvc
- 2026-11-15 08:13:26 UTCfinding · warning execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)
- 2026-11-16 00:00:00 UTClatency execsvc p95 peaked at 1.89 s
- 2026-11-16 01:19:29 UTCdeploy automated deployed execsvc
- 2026-11-16 01:27:59 UTCdeploy automated deployed execsvc
- 2026-11-16 05:19:28 UTCdeploy automated deployed execsvc
- 2026-11-16 07:34:54 UTCfinding · warning execsvc: a new error shape, 44× in the last hour, never seen in 90 days (pubsub not initialized, dropping onboarding event)
- 2026-11-16 08:05:51 UTCdeploy automated deployed execsvc
- 2026-11-16 15:39:46 UTCdeploy automated deployed execsvc
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)
- 2026-11-16 18:21:13 UTCdeploy automated deployed execsvc
- 2026-11-17 00:00:00 UTCerrors execsvc 5xx peaked at 787 in one five-minute bucket (median 0 in this window)
- 2026-11-17 01:33:37 UTCdeploy automated deployed execsvc
- 2026-11-17 09:07:54 UTCdeploy automated deployed execsvc
- 2026-11-17 09:25:52 UTCdeploy automated deployed execsvc
History
- 2026-11-17 04:16:30 UTCreopened x-ray · severity=critical was=warning title=execsvc is burning its error budget at 18.0× (1 h) / 8.9× (6 h) against a 95.00% objective level=medium was_level=low finding={'id': 163, 'fingerprint': '019ea64d73a946c7a704', 'kind': 'budget.burning', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc is burning its error budget at 18.0× (1 h) / 8.9× (6 h) against a 95.00% objective', 'facts': {'objective': 95.0, 'objective_source': 'derived', 'achieved': 55.39772727272727, 'calls': 704.0, 'bad': 314.0, 'c5xx': 314.0, 'budget': 35.2, 'spent': 8.920454545454545, 'remaining_pct': 0.0, 'burn_1h': 17.99426934097421, 'burn_6h': 8.920454545454545, 'verdict': 'burning fast', 'p95_ms': 2632.432432432429, 'window': [1794867390, 1794888990], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794867390&to=1794888990'}, {'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794867390&to=1794888990'}], 'provenance': {'source': 'rollup', 'collected_at': 1794888882, 'age_seconds': 110, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 60.3, 'status': 'open', 'first_seen': 1787766008, 'last_seen': 1794888990, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 211, 'narrative': 'execsvc is spending its error budget 6.5× faster than the objective allows over the last hour (3.6× over six hours). The objective is 98.0%, derived — a baseline, not a promise. 10 bad events against 140 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794888990, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7122984, 'stale_s': 2}
- 2026-11-17 10:21:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 163, 'fingerprint': '019ea64d73a946c7a704', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective', 'facts': {'objective': 95.0, 'objective_source': 'derived', 'achieved': 58.68766404199475, 'calls': 1905.0, 'bad': 787.0, 'c5xx': 787.0, 'budget': 95.25, 'spent': 8.26246719160105, 'remaining_pct': 0.0, 'burn_1h': 0.0, 'burn_6h': 8.262467191601049, 'verdict': 'burning', 'p95_ms': 704.4117647058823, 'window': [1794887977, 1794909577], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794887977&to=1794909577'}, {'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794887977&to=1794909577'}], 'provenance': {'source': 'rollup', 'collected_at': 1794908969, 'age_seconds': 610, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 39.2, 'status': 'resolved', 'first_seen': 1787766008, 'last_seen': 1794909577, 'resolved_at': 1794910905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 233, 'narrative': 'execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794888990, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7144898, 'stale_s': 1329}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794908969 |
| age_seconds | 610 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | budgets |