Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:10:44.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective

warning Error budget burning master resolved impact 39 Owner: Madhava Mukkamala (backup: Shashank Agrawal) opened 83d ago · last observed 1d ago · observed 233× · #163 · 019ea64d73a946c7a704

What happened template

execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.

Bad events are consuming the error budget faster than the objective allows.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; execsvc failed 314 of 704 calls (all 5xx, 55.4% vs a 95% objective), burning 18.0× (1 h) / 8.9× (6 h), budget at 0%, p95 2632 ms.
The finding opened at 2026-11-17 04:16 UTC; this is a recurring finding with 5 earlier episodes this month.
Checked: no error lines stored for 03:46–04:16 UTC, no other findings at the same time, and the 5 deploys in the prior 24 h were config-only autosyncs (last at 01:33 UTC, config d8d21151cd28, 162 min before onset) with no code change.
An earlier first look (2026-11-06) tied a similar burst to Neo4j DeadlockDetected during changelog inserts, but nothing in this window confirms that.
Next step: open the execsvc service page for the window to see which endpoint returns the 500s, then pull that pod's logs directly since none were stored.

low confidence

claude-opus-5-5 · 1d ago · $0.03 — every link above is evidence it read; nothing else was claimed.

The facts

objective95.00
objective_sourcederived
achieved58.69
calls1,905
bad787.00
c5xx787.00
budget95.25
spent8.26
remaining_pct0.00
burn_1h0.00
burn_6h8.26
verdictburning
p95_ms704.41
discriminator
window2026-11-17 03:59:37 UTC → 2026-11-17 09:59:37 UTC

What is failing in execsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 03:59:37 UTC → 2026-11-17 09:59:37 UTC) there were 787; every line links to the grouped errors behind it.

POST /api/trialexec/v2/study/:id/activity/status 787 bad of 951 calls · 100% of the service's 5xx

The error shapes behind them

787× Request completed — 500 · failed to fetch activity status: could not match participant question to activity graph: quest… since 2026-10-16 12:11:42 UTC
787× Failed to fetch activity status — could not match participant question to activity graph: questionId="b3228925-d16e-41cf… since 2026-11-04 11:18:42 UTC
787× could not match participant questionnaire/question/table/doc-upload to activity graph since 2026-11-17 04:05:17 UTC
30× No cached node found for activation chain ref node since 2026-11-12 13:43:21 UTC
29× no result from branch evaluation since 2026-11-13 15:11:12 UTC

All execsvc errors, grouped → · service page · its logs

The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.

History

  • 2026-11-17 04:16:30 UTCreopened x-ray · severity=critical was=warning title=execsvc is burning its error budget at 18.0× (1 h) / 8.9× (6 h) against a 95.00% objective level=medium was_level=low finding={'id': 163, 'fingerprint': '019ea64d73a946c7a704', 'kind': 'budget.burning', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc is burning its error budget at 18.0× (1 h) / 8.9× (6 h) against a 95.00% objective', 'facts': {'objective': 95.0, 'objective_source': 'derived', 'achieved': 55.39772727272727, 'calls': 704.0, 'bad': 314.0, 'c5xx': 314.0, 'budget': 35.2, 'spent': 8.920454545454545, 'remaining_pct': 0.0, 'burn_1h': 17.99426934097421, 'burn_6h': 8.920454545454545, 'verdict': 'burning fast', 'p95_ms': 2632.432432432429, 'window': [1794867390, 1794888990], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794867390&to=1794888990'}, {'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794867390&to=1794888990'}], 'provenance': {'source': 'rollup', 'collected_at': 1794888882, 'age_seconds': 110, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 60.3, 'status': 'open', 'first_seen': 1787766008, 'last_seen': 1794888990, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 211, 'narrative': 'execsvc is spending its error budget 6.5× faster than the objective allows over the last hour (3.6× over six hours). The objective is 98.0%, derived — a baseline, not a promise. 10 bad events against 140 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794888990, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7122984, 'stale_s': 2}
  • 2026-11-17 10:21:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 163, 'fingerprint': '019ea64d73a946c7a704', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective', 'facts': {'objective': 95.0, 'objective_source': 'derived', 'achieved': 58.68766404199475, 'calls': 1905.0, 'bad': 787.0, 'c5xx': 787.0, 'budget': 95.25, 'spent': 8.26246719160105, 'remaining_pct': 0.0, 'burn_1h': 0.0, 'burn_6h': 8.262467191601049, 'verdict': 'burning', 'p95_ms': 704.4117647058823, 'window': [1794887977, 1794909577], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794887977&to=1794909577'}, {'label': 'execsvc service page', 'href': '/service/execsvc?cluster=master&from=1794887977&to=1794909577'}], 'provenance': {'source': 'rollup', 'collected_at': 1794908969, 'age_seconds': 610, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 39.2, 'status': 'resolved', 'first_seen': 1787766008, 'last_seen': 1794909577, 'resolved_at': 1794910905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 233, 'narrative': 'execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794888990, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7144898, 'stale_s': 1329}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794908969
age_seconds610
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorbudgets