docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective
warning
Error budget burning
master
resolved
impact 38
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 10d ago · last observed 2d ago · observed 63× · #16828 · fa41081bd3d92369dc52
What happened template
docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.
Bad events are consuming the error budget faster than the objective allows.
First look — investigated before anyone looked
docproc returned 5xx on 3 of 35 calls (91.4% vs 99.5% objective), burning budget at 17.1× over 1 h and 6 h; p95 708 ms.
Started 21:16 UTC on 2026-11-15, when several new error shapes appeared together. The last error was the 500 at 21:18.
The chain in the logs: ServiceTokenManager refresh got 429 Too Many Requests from https://login.tilda.pizza/oauth/token with no usable cached token (tilda_auth/service_token_manager.py:185). This left 'No docsvc authorization', so docsvc PATCH aiEnabled failed (queries/ai_run_queries.py:294) and AIRun creation aborted the pipeline (orchestration/document_orchestrator.py:4514, handlers/pubsub_handl
The two autosync deploys before onset were config-only, 15–20 h earlier, and nothing links them to this.
Next: find out why the OAuth token endpoint rate-limited docproc (quota or refresh storm), and check that the token manager keeps and reuses a cached token through 429 backoff.
medium confidence dependency login.tilda.pizza OAuth token endpoint (429)
claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| objective | 99.50 |
| objective_source | derived |
| achieved | 96.98 |
| calls | 199.00 |
| bad | 6.00 |
| c5xx | 6.00 |
| budget | 0.99 |
| spent | 6.03 |
| remaining_pct | 0.00 |
| burn_1h | 0.00 |
| burn_6h | 6.03 |
| verdict | burning |
| p95_ms | 485.73 |
| discriminator | |
| window | 2026-11-15 17:26:00 UTC → 2026-11-15 23:26:00 UTC |
What is failing in docproc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-15 17:26:00 UTC → 2026-11-15 23:26:00 UTC) there were 6; every line links to the grouped errors behind it.
POST /api/v1/docproc/:id/split |
6 bad of 6 calls · 100% of the service's 5xx |
The error shapes behind them
| 38× | Failed to create document processing AIRun — aborting pipeline (refuse to run untracked) — No docsvc authorization (call… since 2026-11-15 21:16:35 UTC |
| 38× | docsvc PATCH aiEnabled failed before creating AI run since 2026-11-15 21:16:35 UTC |
| 29× | ServiceTokenManager refresh failed (429 Client Error: Too Many Requests for url: https://login.tilda.pizza/oauth/token) … since 2026-11-15 21:16:35 UTC |
| 23× | [90814349-b97c-4cf0-b423-9decc2a42448] PubSub certify failed: AIRun creation failed for document 90814349-b97c-4cf0-b423… since 2026-11-01 21:47:55 UTC |
| 10× | Recovery failed — AIRun creation failed for document 708e1425-a271-4065-9830-6f906d9e1004 (document processing): No docs… since 2026-11-15 21:17:08 UTC |
The story — No docproc deploy or hand change is recorded in the hour before this opened. Everything below is docproc only.
- 2026-11-07 10:44:56 UTCcypher docproc shape 5F8F3F4B p95 went 5 ms → 48 ms across the window (117 runs)
- 2026-11-07 11:44:56 UTCthis finding docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective
- 2026-11-14 19:35:47 UTCdeploy automated deployed docproc
- 2026-11-14 19:37:58 UTCdeploy automated deployed docproc
- 2026-11-14 20:27:41 UTCdeploy automated deployed docproc
- 2026-11-15 00:00:00 UTCerrors docproc 5xx peaked at 6 in one five-minute bucket (median 0 in this window)
- 2026-11-15 00:59:14 UTCdeploy automated deployed docproc
- 2026-11-15 06:25:49 UTCdeploy automated deployed docproc
- 2026-11-15 21:24:06 UTCfinding · warning docproc: a new error shape, 19× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)
- 2026-11-15 21:24:06 UTCfinding · warning docproc: a new error shape, 19× in the last hour, never seen in 90 days (docsvc PATCH aiEnabled failed before creating AI run)
- 2026-11-15 21:24:06 UTCfinding · warning docproc: a new error shape, 10× in the last hour, never seen in 90 days (Recovery failed — AIRun creation failed for document ? (document proce)
- 2026-11-15 21:43:40 UTCfinding · warning docproc: a new error shape, 16× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )
- 2026-11-15 22:03:48 UTCfinding · warning docproc: a new error shape, 5× in the last hour, never seen in 90 days (docsvc call not possible)
- 2026-11-15 22:39:56 UTCfinding · warning docproc: a new error shape, 5× in the last hour, never seen in 90 days ([?] Pubsub:Attribute Update failed: AIRun creation failed for document)
History
- 2026-11-15 21:33:35 UTCreopened x-ray · severity=critical was=warning title=docproc is burning its error budget at 17.1× (1 h) / 17.1× (6 h) against a 99.50% objective level=medium was_level=low finding={'id': 16828, 'fingerprint': 'fa41081bd3d92369dc52', 'kind': 'budget.burning', 'severity': 'critical', 'cluster': 'master', 'subject': 'docproc', 'title': 'docproc is burning its error budget at 17.1× (1 h) / 17.1× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 91.42857142857143, 'calls': 35.0, 'bad': 3.0, 'c5xx': 3.0, 'budget': 0.175, 'spent': 17.142857142857142, 'remaining_pct': 0.0, 'burn_1h': 17.142857142857142, 'burn_6h': 17.142857142857142, 'verdict': 'burning fast', 'p95_ms': 708.3333333333334, 'window': [1794756815, 1794778415], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794756815&to=1794778415'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=master&from=1794756815&to=1794778415'}], 'provenance': {'source': 'rollup', 'collected_at': 1794778340, 'age_seconds': 77, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 56.2, 'status': 'open', 'first_seen': 1794051896, 'last_seen': 1794778415, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 49, 'narrative': 'docproc is spending its error budget — faster than the objective allows over the last hour (12.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 3 bad events against 47 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794778415, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 726521, 'stale_s': 2}
- 2026-11-15 23:46:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16828, 'fingerprint': 'fa41081bd3d92369dc52', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'master', 'subject': 'docproc', 'title': 'docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 96.98492462311557, 'calls': 199.0, 'bad': 6.0, 'c5xx': 6.0, 'budget': 0.995, 'spent': 6.030150753768845, 'remaining_pct': 0.0, 'burn_1h': 0.0, 'burn_6h': 6.030150753768844, 'verdict': 'burning', 'p95_ms': 485.7317073170731, 'window': [1794763560, 1794785160], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794763560&to=1794785160'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=master&from=1794763560&to=1794785160'}], 'provenance': {'source': 'rollup', 'collected_at': 1794785070, 'age_seconds': 92, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 38.1, 'status': 'resolved', 'first_seen': 1794051896, 'last_seen': 1794785160, 'resolved_at': 1794786365, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 63, 'narrative': 'docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794778415, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 734470, 'stale_s': 1206}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794785070 |
| age_seconds | 92 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | budgets |