Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:14:08.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none

critical 4xx spike (403/401/404…) production resolved impact 80 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) opened 31d ago · last observed 2d ago · observed 8× · #5619 · 735a25cf6522d8646ccd

What happened template

4xx spike (403/401/404…): docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none. Facts: endpoint GET /api/v1/docproc/ai-runs/:id, service docproc, calls 53, c4xx 28, share_pct 52.83018867924528, c401 0.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; GET /api/v1/docproc/ai-runs/:id returned 403 on 28 of 63 calls (44%) in 15 min, against a usual 0.
Onset 07:35 UTC on 2026-11-16. The first failing request logged is at 07:46, and all 8 sampled 403s come from org perceiveBio.
The only deploy, at 05:10 UTC by [email protected], was config only (6db9855a7d34), 144 min earlier, and nothing ties it to the failure.
At 07:45 docproc logged a ServiceTokenManager refresh failure (invalid_grant, no usable cached token) and a needs_attention HTTP 401. Around 07:26 virtual-assistant opened a 401 spike.
Next step: check whether docproc's service token, or the auth service behind it, is failing. Open a 403 trace to see whether the permission check depends on that token.

low confidence

claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.

The facts

endpointGET /api/v1/docproc/ai-runs/:id
servicedocproc
calls53
c4xx28
share_pct52.83
c4010
c40328
other_4xx0
rejectingno
usual_per_window0.00
baseline_days7
window_min15
discriminatorGET /api/v1/docproc/ai-runs/:id
window2026-11-16 07:45:00 UTC → 2026-11-16 08:00:00 UTC

What is failing in docproc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docproc. In this finding's window (2026-11-16 07:45:00 UTC → 2026-11-16 08:00:00 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

200× [dd5021b7-ca83-4be1-8e43-cb6f396e09e2] PubSub certify failed: AIRun creation failed for document dd5021b7-ca83-4be1-8e43… since 2026-08-26 14:55:56 UTC
146× [0dc5292f-37b8-457d-87e8-7df83bc10b1c] ══════════════════════════════════════════════════════ since 2026-08-31 09:11:43 UTC
92× [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] PubSub version upload failed: AIRun creation failed for document b6ce3808-ab0a-44… since 2026-08-25 22:20:29 UTC
92× docsvc PATCH aiEnabled failed before creating AI run since 2026-11-07 14:31:44 UTC
92× [b6ce3808-ab0a-4471-8c8a-2ebc16b85624] Version upload handling failed: AIRun creation failed for document b6ce3808-ab0a-… since 2026-08-25 22:20:29 UTC

All docproc errors, grouped → · service page · its logs

The story — No docproc deploy or hand change is recorded in the hour before this opened. Everything below is docproc only.

  • 2026-10-18 18:16:26 UTCthis finding docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
  • 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc

History

  • 2026-11-16 08:02:30 UTCreopened x-ray · severity=critical was=warning title=docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none level=high was_level=medium finding={'id': 5619, 'fingerprint': '735a25cf6522d8646ccd', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/v1/docproc/ai-runs/:id', 'service': 'docproc', 'calls': 63, 'c4xx': 28, 'share_pct': 44.44444444444444, 'c401': 0, 'c403': 28, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794814500, 1794815400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/docproc/ai-runs/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794814500&to=1794815400&service=docproc&status=4xx'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=production&from=1794814500&to=1794815400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794815942, 'age_seconds': 216, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 72.7, 'status': 'open', 'first_seen': 1792347386, 'last_seen': 1794816150, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': None, 'narrative_kind': None, 'discriminator': 'GET /api/v1/docproc/ai-runs/:id', 'level': 'high', 'told_level': 'medium', 'episode_at': 1794816150, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2468771, 'stale_s': 7}
  • 2026-11-16 08:39:06 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5619, 'fingerprint': '735a25cf6522d8646ccd', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'docproc', 'title': 'docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/v1/docproc/ai-runs/:id', 'service': 'docproc', 'calls': 53, 'c4xx': 28, 'share_pct': 52.83018867924528, 'c401': 0, 'c403': 28, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794815100, 1794816000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/docproc/ai-runs/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794815100&to=1794816000&service=docproc&status=4xx'}, {'label': 'docproc service page', 'href': '/service/docproc?cluster=production&from=1794815100&to=1794816000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794816532, 'age_seconds': 417, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 80.4, 'status': 'resolved', 'first_seen': 1792347386, 'last_seen': 1794816941, 'resolved_at': 1794818346, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': '4xx spike (403/401/404…): docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none. Facts: endpoint GET /api/v1/docproc/ai-runs/:id, service docproc, calls 53, c4xx 28, share_pct 52.83018867924528, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/docproc/ai-runs/:id', 'level': 'high', 'told_level': 'high', 'episode_at': 1794816150, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2470967, 'stale_s': 1412}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794816532
age_seconds417
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike