Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 12:06:19.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective

warning Error budget burning test26 resolved impact 28 Owner: Prasad Limaye (backup: Harish Kumar) opened 84d ago · last observed 2d ago · observed 174× · #95 · 0eb6163866e506c0a717

What happened template

docsvc is spending its error budget 6.5× faster than the objective allows over the last hour (1.2× over six hours). The objective is 97.0%, derived — a baseline, not a promise. 26 bad events against 731 calls in the window.

Bad events are consuming the error budget faster than the objective allows.

The facts

objective97.00
objective_sourcederived
achieved96.44
calls731.00
bad26.00
c5xx26.00
budget21.93
spent1.19
remaining_pct0.00
burn_1h6.54
burn_6h1.19
verdictburning
p95_ms1,091
discriminator
window2026-11-15 21:12:35 UTC → 2026-11-16 03:12:35 UTC

What is failing in docsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-15 21:12:35 UTC → 2026-11-16 03:12:35 UTC) there were 26; every line links to the grouped errors behind it.

PUT /api/documents/promote 12 bad of 15 calls · 46% of the service's 5xx
PATCH /api/documents/v2 11 bad of 90 calls · 42% of the service's 5xx
PUT /api/documents/folders/manager/:id 2 bad of 2 calls · 8% of the service's 5xx
PATCH /api/documents/folders/subArtifacts/:id 1 bad of 2 calls · 4% of the service's 5xx

The error shapes behind them

156× template backfill: could not move document — document has no primary version, cannot move to Templates since 2026-11-16 13:27:23 UTC
123× Request completed — 500 · could not update document metadata for doc id 8a0be22c-dffd-478a-90fd-5bf70b6ddcbb: could not … since 2026-10-22 16:52:52 UTC
123× could not update document metadata for doc id dd558be4-87a4-4c01-a343-dc91933b5d63: could not update the document: Eithe… since 2026-11-16 02:02:27 UTC
41× Request completed — 500 · 0/20 documents promoted to TMF and 20 failed since 2026-11-16 05:43:36 UTC
41× Request completed — 500 · could not update document metadata for doc id 3288d94d-ab30-4035-ad71-adfa2d1161ad: could not … since 2026-10-28 04:10:23 UTC

All docsvc errors, grouped → · service page · its logs

The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.

History

  • 2026-11-16 03:12:35 UTCreopened x-ray · severity=warning was=warning title=docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective level=low was_level=low finding={'id': 95, 'fingerprint': '0eb6163866e506c0a717', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective', 'facts': {'objective': 97.0, 'objective_source': 'derived', 'achieved': 96.44322845417237, 'calls': 731.0, 'bad': 26.0, 'c5xx': 26.0, 'budget': 21.93, 'spent': 1.1855905152758779, 'remaining_pct': 0.0, 'burn_1h': 6.5359477124183005, 'burn_6h': 1.1855905152758779, 'verdict': 'burning', 'p95_ms': 1090.6249999999957, 'window': [1794777155, 1794798755], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=test26&from=1794777155&to=1794798755'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794777155&to=1794798755'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798755, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 23.3, 'status': 'open', 'first_seen': 1787713848, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc is spending its error budget — faster than the objective allows over the last hour (7.5× over six hours). The objective is 98.5%, derived — a baseline, not a promise. 6 bad events against 53 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7084911, 'stale_s': 4}
  • 2026-11-16 04:22:10 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 95, 'fingerprint': '0eb6163866e506c0a717', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective', 'facts': {'objective': 97.0, 'objective_source': 'derived', 'achieved': 96.44322845417237, 'calls': 731.0, 'bad': 26.0, 'c5xx': 26.0, 'budget': 21.93, 'spent': 1.1855905152758779, 'remaining_pct': 0.0, 'burn_1h': 6.5359477124183005, 'burn_6h': 1.1855905152758779, 'verdict': 'burning', 'p95_ms': 1090.6249999999957, 'window': [1794777155, 1794798755], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=test26&from=1794777155&to=1794798755'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794777155&to=1794798755'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798755, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.3, 'status': 'resolved', 'first_seen': 1787713848, 'last_seen': 1794798755, 'resolved_at': 1794802930, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc is spending its error budget 6.5× faster than the objective allows over the last hour (1.2× over six hours). The objective is 97.0%, derived — a baseline, not a promise. 26 bad events against 731 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7089085, 'stale_s': 4178}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794798755
age_seconds5
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorbudgets