docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
critical
4xx spike (403/401/404…)
production
resolved
impact 80
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 31d ago · last observed 2d ago · observed 8× · #5619 · 735a25cf6522d8646ccd
What happened template
4xx spike (403/401/404…): docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none. Facts: endpoint GET /api/v1/docproc/ai-runs/:id, service docproc, calls 53, c4xx 28, share_pct 52.83018867924528, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; GET /api/v1/docproc/ai-runs/:id returned 403 on 28 of 63 calls (44%) in 15 min, against a usual 0.
Onset 07:35 UTC on 2026-11-16. The first failing request logged is at 07:46, and all 8 sampled 403s come from org perceiveBio.
The only deploy, at 05:10 UTC by [email protected], was config only (6db9855a7d34), 144 min earlier, and nothing ties it to the failure.
At 07:45 docproc logged a ServiceTokenManager refresh failure (invalid_grant, no usable cached token) and a needs_attention HTTP 401. Around 07:26 virtual-assistant opened a 401 spike.
Next step: check whether docproc's service token, or the auth service behind it, is failing. Open a 403 trace to see whether the permission check depends on that token.
low confidence
claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /api/v1/docproc/ai-runs/:id |
| service | docproc |
| calls | 53 |
| c4xx | 28 |
| share_pct | 52.83 |
| c401 | 0 |
| c403 | 28 |
| other_4xx | 0 |
| rejecting | no |
| usual_per_window | 0.00 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/v1/docproc/ai-runs/:id |
| window | 2026-11-16 07:45:00 UTC → 2026-11-16 08:00:00 UTC |
What is failing in docproc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-16 07:45:00 UTC → 2026-11-16 08:00:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 200× | [dd5021b7-ca83-4be1-8e43-cb6f396e09e2] PubSub certify failed: AIRun creation failed for document dd5021b7-ca83-4be1-8e43… since 2026-08-26 14:55:56 UTC |
| 146× | [0dc5292f-37b8-457d-87e8-7df83bc10b1c] ══════════════════════════════════════════════════════ since 2026-08-31 09:11:43 UTC |
| 92× | [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] PubSub version upload failed: AIRun creation failed for document b6ce3808-ab0a-44… since 2026-08-25 22:20:29 UTC |
| 92× | docsvc PATCH aiEnabled failed before creating AI run since 2026-11-07 14:31:44 UTC |
| 92× | [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] Version upload handling failed: AIRun creation failed for document b6ce3808-ab0a-… since 2026-08-25 22:20:29 UTC |
The story — No docproc deploy or hand change is recorded in the hour before this opened. Everything below is docproc only.
- 2026-10-18 18:16:26 UTCthis finding docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
History
- 2026-11-16 08:02:30 UTCreopened x-ray · severity=critical was=warning title=docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none level=high was_level=medium finding={'id': 5619, 'fingerprint': '735a25cf6522d8646ccd', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/v1/docproc/ai-runs/:id', 'service': 'docproc', 'calls': 63, 'c4xx': 28, 'share_pct': 44.44444444444444, 'c401': 0, 'c403': 28, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794814500, 1794815400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/docproc/ai-runs/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794814500&to=1794815400&service=docproc&status=4xx'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=production&from=1794814500&to=1794815400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794815942, 'age_seconds': 216, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.7, 'status': 'open', 'first_seen': 1792347386, 'last_seen': 1794816150, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': None, 'narrative_kind': None, 'discriminator': 'GET /api/v1/docproc/ai-runs/:id', 'level': 'high', 'told_level': 'medium', 'episode_at': 1794816150, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2468771, 'stale_s': 7}
- 2026-11-16 08:39:06 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5619, 'fingerprint': '735a25cf6522d8646ccd', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/v1/docproc/ai-runs/:id', 'service': 'docproc', 'calls': 53, 'c4xx': 28, 'share_pct': 52.83018867924528, 'c401': 0, 'c403': 28, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794815100, 1794816000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/docproc/ai-runs/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794815100&to=1794816000&service=docproc&status=4xx'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=production&from=1794815100&to=1794816000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794816532, 'age_seconds': 417, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 80.4, 'status': 'resolved', 'first_seen': 1792347386, 'last_seen': 1794816941, 'resolved_at': 1794818346, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': '4xx spike (403/401/404…): docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none. Facts: endpoint GET /api/v1/docproc/ai-runs/:id, service docproc, calls 53, c4xx 28, share_pct 52.83018867924528, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/docproc/ai-runs/:id', 'level': 'high', 'told_level': 'high', 'episode_at': 1794816150, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2470967, 'stale_s': 1412}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794816532 |
| age_seconds | 417 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |