Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 04:32:42.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docsvc GET /api/documents/:id: 4xx — 23 of 59 calls (39 %; other 4xx×23) in the last 15 min — 9× its usual 2.6

warning 4xx spike (403/401/404…) automation resolved impact 33 Owner: Prasad Limaye (backup: Harish Kumar) opened 20d ago · last observed 1d ago · observed 37× · #9969 · 615462aa7ea03f44a2fc

What happened template

4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 23 of 59 calls (39 %; other 4xx×23) in the last 15 min — 9× its usual 2.6. Facts: endpoint GET /api/documents/:id, service docsvc, calls 59, c4xx 23, share_pct 38.983050847457626, c401 0.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

The facts

endpointGET /api/documents/:id
servicedocsvc
calls59
c4xx23
share_pct38.98
c4010
c4030
other_4xx23
rejectingno
usual_per_window2.64
baseline_days7
window_min15
discriminatorGET /api/documents/:id
window2026-11-17 04:45:00 UTC → 2026-11-17 05:00:00 UTC

What is failing in docsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-17 04:45:00 UTC → 2026-11-17 05:00:00 UTC) there were 2; every line links to the grouped errors behind it.

POST /api/documents/folders/manager 2 bad of 2 calls · 100% of the service's 5xx

The error shapes behind them

38× Request completed — 500 · folder manager already has folders since 2026-09-25 08:36:31 UTC
29× failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transac… since 2026-10-31 21:25:03 UTC
29× failed to process db result — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=1… since 2026-10-31 07:21:15 UTC
27× failed to process db result — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=2… since 2026-10-31 21:27:46 UTC
17× failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transac… since 2026-10-31 22:01:39 UTC

All docsvc errors, grouped → · service page · its logs

The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.

History

  • 2026-11-15 19:47:21 UTCreopened x-ray · severity=warning was=warning title=docsvc GET /api/documents/:id: 4xx — 98 of 797 calls (12 %; other 4xx×98) in the last 15 min — 63× its usual 1.6 level=medium was_level=medium finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 98 of 797 calls (12 %; other 4xx×98) in the last 15 min — 63× its usual 1.6', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 797, 'c4xx': 98, 'share_pct': 12.296110414052698, 'c401': 0, 'c403': 0, 'other_4xx': 98, 'rejecting': False, 'usual_per_window': 1.57, 'baseline_days': 7, 'window': [1794770700, 1794771600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794770700&to=1794771600&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794770700&to=1794771600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771969, 'age_seconds': 77, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 38.6, 'status': 'open', 'first_seen': 1793321553, 'last_seen': 1794772041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 33, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 20 of 90 calls (22 %; other 4xx×20) in the last 15 min — 11× its usual 1.9. Facts: endpoint GET /api/documents/:id, service docsvc, calls 90, c4xx 20, share_pct 22.22222222222222, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794772041, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1450493, 'stale_s': 5}
  • 2026-11-15 20:07:34 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 98 of 797 calls (12 %; other 4xx×98) in the last 15 min — 63× its usual 1.6', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 797, 'c4xx': 98, 'share_pct': 12.296110414052698, 'c401': 0, 'c403': 0, 'other_4xx': 98, 'rejecting': False, 'usual_per_window': 1.57, 'baseline_days': 7, 'window': [1794770700, 1794771600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794770700&to=1794771600&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794770700&to=1794771600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771969, 'age_seconds': 77, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 43.6, 'status': 'resolved', 'first_seen': 1793321553, 'last_seen': 1794772041, 'resolved_at': 1794773254, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 33, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 98 of 797 calls (12 %; other 4xx×98) in the last 15 min — 63× its usual 1.6. Facts: endpoint GET /api/documents/:id, service docsvc, calls 797, c4xx 98, share_pct 12.296110414052698, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794772041, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1451706, 'stale_s': 1218}
  • 2026-11-16 08:53:28 UTCreopened x-ray · severity=critical was=warning title=docsvc GET /api/documents/:id: permission denied — 93 of 573 calls (16 %; 403×93) in the last 15 min — 55× its usual 1.7 level=high was_level=medium finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: permission denied — 93 of 573 calls (16 %; 403×93) in the last 15 min — 55× its usual 1.7', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 573, 'c4xx': 93, 'share_pct': 16.230366492146597, 'c401': 0, 'c403': 93, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 1.68, 'baseline_days': 7, 'window': [1794817800, 1794818700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794817800&to=1794818700&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794817800&to=1794818700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794819089, 'age_seconds': 125, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 52.0, 'status': 'open', 'first_seen': 1793321553, 'last_seen': 1794819208, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 34, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 98 of 797 calls (12 %; other 4xx×98) in the last 15 min — 63× its usual 1.6. Facts: endpoint GET /api/documents/:id, service docsvc, calls 797, c4xx 98, share_pct 12.296110414052698, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'high', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1497661, 'stale_s': 6}
  • 2026-11-16 09:50:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: permission denied — 93 of 573 calls (16 %; 403×93) in the last 15 min — 55× its usual 1.7', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 573, 'c4xx': 93, 'share_pct': 16.230366492146597, 'c401': 0, 'c403': 93, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 1.68, 'baseline_days': 7, 'window': [1794817800, 1794818700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794817800&to=1794818700&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794817800&to=1794818700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794819089, 'age_seconds': 125, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 57.0, 'status': 'resolved', 'first_seen': 1793321553, 'last_seen': 1794819208, 'resolved_at': 1794822612, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 34, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: permission denied — 93 of 573 calls (16 %; 403×93) in the last 15 min — 55× its usual 1.7. Facts: endpoint GET /api/documents/:id, service docsvc, calls 573, c4xx 93, share_pct 16.230366492146597, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'high', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1501067, 'stale_s': 3412}
  • 2026-11-17 00:58:16 UTCreopened x-ray · severity=warning was=critical title=docsvc GET /api/documents/:id: 4xx — 118 of 844 calls (14 %; 403×1, other 4xx×117) in the last 15 min — 48× its usual 2.5 level=medium was_level=high finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 118 of 844 calls (14 %; 403×1, other 4xx×117) in the last 15 min — 48× its usual 2.5', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 844, 'c4xx': 118, 'share_pct': 13.981042654028435, 'c401': 0, 'c403': 1, 'other_4xx': 117, 'rejecting': False, 'usual_per_window': 2.48, 'baseline_days': 7, 'window': [1794875400, 1794876300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794875400&to=1794876300&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794875400&to=1794876300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794876899, 'age_seconds': 206, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'open', 'first_seen': 1793321553, 'last_seen': 1794877096, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 35, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: permission denied — 93 of 573 calls (16 %; 403×93) in the last 15 min — 55× its usual 1.7. Facts: endpoint GET /api/documents/:id, service docsvc, calls 573, c4xx 93, share_pct 16.230366492146597, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794877096, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1555552, 'stale_s': 9}
  • 2026-11-17 01:15:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 118 of 844 calls (14 %; 403×1, other 4xx×117) in the last 15 min — 48× its usual 2.5', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 844, 'c4xx': 118, 'share_pct': 13.981042654028435, 'c401': 0, 'c403': 1, 'other_4xx': 117, 'rejecting': False, 'usual_per_window': 2.48, 'baseline_days': 7, 'window': [1794875400, 1794876300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794875400&to=1794876300&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794875400&to=1794876300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794876899, 'age_seconds': 206, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1793321553, 'last_seen': 1794877096, 'resolved_at': 1794878127, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 35, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 118 of 844 calls (14 %; 403×1, other 4xx×117) in the last 15 min — 48× its usual 2.5. Facts: endpoint GET /api/documents/:id, service docsvc, calls 844, c4xx 118, share_pct 13.981042654028435, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794877096, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1556582, 'stale_s': 1039}
  • 2026-11-17 05:00:47 UTCreopened x-ray · severity=warning was=warning title=docsvc GET /api/documents/:id: 4xx — 42 of 138 calls (30 %; other 4xx×42) in the last 15 min — 16× its usual 2.6 level=medium was_level=medium finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 42 of 138 calls (30 %; other 4xx×42) in the last 15 min — 16× its usual 2.6', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 138, 'c4xx': 42, 'share_pct': 30.434782608695652, 'c401': 0, 'c403': 0, 'other_4xx': 42, 'rejecting': False, 'usual_per_window': 2.64, 'baseline_days': 7, 'window': [1794889800, 1794890700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794889800&to=1794890700&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794889800&to=1794890700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794891276, 'age_seconds': 379, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.4, 'status': 'open', 'first_seen': 1793321553, 'last_seen': 1794891647, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 36, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 118 of 844 calls (14 %; 403×1, other 4xx×117) in the last 15 min — 48× its usual 2.5. Facts: endpoint GET /api/documents/:id, service docsvc, calls 844, c4xx 118, share_pct 13.981042654028435, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794891647, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1570101, 'stale_s': 7}
  • 2026-11-17 05:26:24 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 9969, 'fingerprint': '615462aa7ea03f44a2fc', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'docsvc', 'title': 'docsvc GET /api/documents/:id: 4xx — 23 of 59 calls (39 %; other 4xx×23) in the last 15 min — 9× its usual 2.6', 'facts': {'endpoint': 'GET /api/documents/:id', 'service': 'docsvc', 'calls': 59, 'c4xx': 23, 'share_pct': 38.983050847457626, 'c401': 0, 'c403': 0, 'other_4xx': 23, 'rejecting': False, 'usual_per_window': 2.64, 'baseline_days': 7, 'window': [1794890700, 1794891600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/documents/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794890700&to=1794891600&service=docsvc&status=4xx'}, {'label': 'docsvc service page', 'href': '/service/docsvc?cluster=automation&from=1794890700&to=1794891600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794892132, 'age_seconds': 68, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 33.0, 'status': 'resolved', 'first_seen': 1793321553, 'last_seen': 1794892192, 'resolved_at': 1794893184, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 37, 'narrative': '4xx spike (403/401/404…): docsvc GET /api/documents/:id: 4xx — 23 of 59 calls (39 %; other 4xx×23) in the last 15 min — 9× its usual 2.6. Facts: endpoint GET /api/documents/:id, service docsvc, calls 59, c4xx 23, share_pct 38.983050847457626, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/documents/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794891647, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1571638, 'stale_s': 999}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794892132
age_seconds68
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike