docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective
warning
Error budget burning
test26
resolved
impact 28
Owner: Prasad Limaye (backup: Harish Kumar)
opened 84d ago · last observed 2d ago · observed 174× · #95 · 0eb6163866e506c0a717
What happened template
docsvc is spending its error budget 6.5× faster than the objective allows over the last hour (1.2× over six hours). The objective is 97.0%, derived — a baseline, not a promise. 26 bad events against 731 calls in the window.
Bad events are consuming the error budget faster than the objective allows.
The facts
| objective | 97.00 |
| objective_source | derived |
| achieved | 96.44 |
| calls | 731.00 |
| bad | 26.00 |
| c5xx | 26.00 |
| budget | 21.93 |
| spent | 1.19 |
| remaining_pct | 0.00 |
| burn_1h | 6.54 |
| burn_6h | 1.19 |
| verdict | burning |
| p95_ms | 1,091 |
| discriminator | |
| window | 2026-11-15 21:12:35 UTC → 2026-11-16 03:12:35 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-15 21:12:35 UTC → 2026-11-16 03:12:35 UTC) there were 26; every line links to the grouped errors behind it.
PUT /api/documents/promote |
12 bad of 15 calls · 46% of the service's 5xx |
PATCH /api/documents/v2 |
11 bad of 90 calls · 42% of the service's 5xx |
PUT /api/documents/folders/manager/:id |
2 bad of 2 calls · 8% of the service's 5xx |
PATCH /api/documents/folders/subArtifacts/:id |
1 bad of 2 calls · 4% of the service's 5xx |
The error shapes behind them
| 156× | template backfill: could not move document — document has no primary version, cannot move to Templates since 2026-11-16 13:27:23 UTC |
| 123× | Request completed — 500 · could not update document metadata for doc id 8a0be22c-dffd-478a-90fd-5bf70b6ddcbb: could not … since 2026-10-22 16:52:52 UTC |
| 123× | could not update document metadata for doc id dd558be4-87a4-4c01-a343-dc91933b5d63: could not update the document: Eithe… since 2026-11-16 02:02:27 UTC |
| 41× | Request completed — 500 · 0/20 documents promoted to TMF and 20 failed since 2026-11-16 05:43:36 UTC |
| 41× | Request completed — 500 · could not update document metadata for doc id 3288d94d-ab30-4035-ad71-adfa2d1161ad: could not … since 2026-10-28 04:10:23 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-08-26 03:10:48 UTCthis finding docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective
- 2026-08-26 11:37:27 UTCfinding · info docsvc Neo4j load 180 q/min vs a normal 1 q/min ± 1 q/min for Mon 06:00 UTC (3 of the last 4 checks)
- 2026-08-26 14:40:05 UTCfinding · info docsvc: p99 latency (ms) 3.64 s — 9.9× the fleet's usual 368 ms
- 2026-09-03 21:27:59 UTCfinding · warning docsvc shape 00000000: p95 1.29 s now vs 55 ms earlier (23.2×, 941 runs)
- 2026-10-29 01:01:22 UTCfinding · info docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)
- 2026-11-02 05:09:15 UTCfinding · warning docsvc GET /api/documents/:id: 4xx — 123 of 130 calls (95 %; other 4xx×123) in the last 15 min — it usually has none
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 00:00:00 UTClatency docsvc p95 peaked at 2.55 s
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 26 in one five-minute bucket (median 0 in this window)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc: a new error shape, 11× in the last hour, never seen in 90 days (Request completed — ? · document cannot be promoted to TMF: document m)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc PUT /api/documents/promote: 16 of 16 calls failed (100 %) in the last 15 min
History
- 2026-11-16 03:12:35 UTCreopened x-ray · severity=warning was=warning title=docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective level=low was_level=low finding={'id': 95, 'fingerprint': '0eb6163866e506c0a717', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective', 'facts': {'objective': 97.0, 'objective_source': 'derived', 'achieved': 96.44322845417237, 'calls': 731.0, 'bad': 26.0, 'c5xx': 26.0, 'budget': 21.93, 'spent': 1.1855905152758779, 'remaining_pct': 0.0, 'burn_1h': 6.5359477124183005, 'burn_6h': 1.1855905152758779, 'verdict': 'burning', 'p95_ms': 1090.6249999999957, 'window': [1794777155, 1794798755], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=test26&from=1794777155&to=1794798755'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794777155&to=1794798755'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798755, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 23.3, 'status': 'open', 'first_seen': 1787713848, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc is spending its error budget — faster than the objective allows over the last hour (7.5× over six hours). The objective is 98.5%, derived — a baseline, not a promise. 6 bad events against 53 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7084911, 'stale_s': 4}
- 2026-11-16 04:22:10 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 95, 'fingerprint': '0eb6163866e506c0a717', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'test26', 'subject': 'docsvc', 'title': 'docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective', 'facts': {'objective': 97.0, 'objective_source': 'derived', 'achieved': 96.44322845417237, 'calls': 731.0, 'bad': 26.0, 'c5xx': 26.0, 'budget': 21.93, 'spent': 1.1855905152758779, 'remaining_pct': 0.0, 'burn_1h': 6.5359477124183005, 'burn_6h': 1.1855905152758779, 'verdict': 'burning', 'p95_ms': 1090.6249999999957, 'window': [1794777155, 1794798755], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=test26&from=1794777155&to=1794798755'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=test26&from=1794777155&to=1794798755'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798755, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.3, 'status': 'resolved', 'first_seen': 1787713848, 'last_seen': 1794798755, 'resolved_at': 1794802930, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc is spending its error budget 6.5× faster than the objective allows over the last hour (1.2× over six hours). The objective is 97.0%, derived — a baseline, not a promise. 26 bad events against 731 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 7089085, 'stale_s': 4178}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794798755 |
| age_seconds | 5 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | budgets |