Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:15:19.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min

critical Endpoint failing master resolved impact 63 Owner: Prasad Limaye (backup: Harish Kumar) opened 29d ago · last observed 2d ago · observed 10× · #5073 · 2d44aa8b4ea0694932cf

What happened template

docsvc's POST /api/documents/folders on master failed 8 of 8 calls (100 %) in the last 15 min. automated changed docsvc 23 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.

First look — investigated before anyone looked

docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min.
Cause not clear from logs and recent changes; checked: error lines (none stored), failing requests (8), deploys in the 24 h before (3), commits in the deploy's range (0), the code at the failing place (1), this finding's history, other findings at the same time (1).

low confidence

claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed. the model named handlers/helpers.go:61, which the evidence does not hold — its answer was withheld.

The facts

endpointPOST /api/documents/folders
servicedocsvc
calls8
failed8
c4xx0
fail_pct100.00
window_min15
discriminatorPOST /api/documents/folders
window2026-11-16 07:00:00 UTC → 2026-11-16 07:15:00 UTC

Adjacent deploys — adjacency, not proof

automateddocsvc · 23 min before

What is failing in docsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 07:00:00 UTC → 2026-11-16 07:15:00 UTC) there were 8; every line links to the grouped errors behind it.

POST /api/documents/folders 8 bad of 8 calls · 100% of the service's 5xx

The error shapes behind them

26× Request completed — 500 · duplicate subfolder name 'Chat uploads' found in the parent folder(asas22) since 2026-11-12 11:38:04 UTC
26× duplicate subfolder name 'Chat uploads' found in the parent folder(asas22) since 2026-11-12 11:38:04 UTC
23× Request completed — 500 · could not update document metadata for doc id 0211da3b-b92d-4f58-a742-62624434f054: could not … since 2026-09-30 16:47:17 UTC
11× Request completed — 500 · failed to share Document: failed to retrieve folder for DocType(), err: could not find the def… since 2026-09-24 04:19:21 UTC
7× Request completed — 500 · could not update document metadata for doc id 5e8b6a30-080e-4d35-9878-e57e75108c3d: could not … since 2026-10-08 19:24:13 UTC

All docsvc errors, grouped → · service page · its logs

The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.

History

  • 2026-11-16 07:17:55 UTCreopened x-ray · severity=critical was=critical title=docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5073, 'fingerprint': '2d44aa8b4ea0694932cf', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/documents/folders', 'service': 'docsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794811800, 1794812700], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': 'd54f13665de9543057cc6c434f059c55524b4e40', 'ts': 1794812601, 'minutes_before': 14}], 'discriminator': 'POST /api/documents/folders'}, 'evidence': [{'label': 'docsvc service page', 'href': '/service/docsvc?cluster=master&from=1794811800&to=1794813475'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794811800&to=1794813475'}], 'provenance': {'source': 'rollup', 'collected_at': 1794813292, 'age_seconds': 190, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 59.2, 'status': 'open', 'first_seen': 1792497176, 'last_seen': 1794813475, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "docsvc's POST /api/documents/folders on master failed 5 of 5 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/documents/folders', 'level': 'high', 'told_level': None, 'episode_at': 1794813475, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2316305, 'stale_s': 6}
  • 2026-11-16 09:50:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5073, 'fingerprint': '2d44aa8b4ea0694932cf', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/documents/folders', 'service': 'docsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794812400, 1794813300], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': 'd54f13665de9543057cc6c434f059c55524b4e40', 'ts': 1794812601, 'minutes_before': 23}], 'discriminator': 'POST /api/documents/folders'}, 'evidence': [{'label': 'docsvc service page', 'href': '/service/docsvc?cluster=master&from=1794812400&to=1794814013'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794812400&to=1794814013'}], 'provenance': {'source': 'rollup', 'collected_at': 1794813864, 'age_seconds': 155, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 63.2, 'status': 'resolved', 'first_seen': 1792497176, 'last_seen': 1794814013, 'resolved_at': 1794822612, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': "docsvc's POST /api/documents/folders on master failed 8 of 8 calls (100 %) in the last 15 min. automated changed docsvc 23 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/documents/folders', 'level': 'high', 'told_level': None, 'episode_at': 1794813475, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2325444, 'stale_s': 8607}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794813864
age_seconds155
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_failing