docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min
critical
Endpoint failing
master
resolved
impact 63
Owner: Prasad Limaye (backup: Harish Kumar)
opened 29d ago · last observed 2d ago · observed 10× · #5073 · 2d44aa8b4ea0694932cf
What happened template
docsvc's POST /api/documents/folders on master failed 8 of 8 calls (100 %) in the last 15 min. automated changed docsvc 23 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.
One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.
First look — investigated before anyone looked
docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min.
Cause not clear from logs and recent changes; checked: error lines (none stored), failing requests (8), deploys in the 24 h before (3), commits in the deploy's range (0), the code at the failing place (1), this finding's history, other findings at the same time (1).
low confidence
claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed. the model named handlers/helpers.go:61, which the evidence does not hold — its answer was withheld.
The facts
| endpoint | POST /api/documents/folders |
| service | docsvc |
| calls | 8 |
| failed | 8 |
| c4xx | 0 |
| fail_pct | 100.00 |
| window_min | 15 |
| discriminator | POST /api/documents/folders |
| window | 2026-11-16 07:00:00 UTC → 2026-11-16 07:15:00 UTC |
Adjacent deploys — adjacency, not proof
| automated | docsvc · 23 min before |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 07:00:00 UTC → 2026-11-16 07:15:00 UTC) there were 8; every line links to the grouped errors behind it.
POST /api/documents/folders |
8 bad of 8 calls · 100% of the service's 5xx |
The error shapes behind them
| 26× | Request completed — 500 · duplicate subfolder name 'Chat uploads' found in the parent folder(asas22) since 2026-11-12 11:38:04 UTC |
| 26× | duplicate subfolder name 'Chat uploads' found in the parent folder(asas22) since 2026-11-12 11:38:04 UTC |
| 23× | Request completed — 500 · could not update document metadata for doc id 0211da3b-b92d-4f58-a742-62624434f054: could not … since 2026-09-30 16:47:17 UTC |
| 11× | Request completed — 500 · failed to share Document: failed to retrieve folder for DocType(), err: could not find the def… since 2026-09-24 04:19:21 UTC |
| 7× | Request completed — 500 · could not update document metadata for doc id 5e8b6a30-080e-4d35-9878-e57e75108c3d: could not … since 2026-10-08 19:24:13 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-10-20 11:52:56 UTCthis finding docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min
- 2026-10-23 14:04:30 UTCfinding · info docsvc p95 1.69 s vs a normal 494 ms ± 213 ms for Mon 17:00 UTC (3 of the last 4 checks)
- 2026-11-14 15:17:18 UTCdeploy automated deployed docsvc
- 2026-11-15 00:59:17 UTCdeploy automated deployed docsvc
- 2026-11-15 01:27:34 UTCdeploy automated deployed docsvc
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 8 in one five-minute bucket (median 0 in this window)
- 2026-11-16 00:00:00 UTClatency docsvc p95 peaked at 1.25 s
- 2026-11-16 01:17:48 UTCdeploy automated deployed docsvc
- 2026-11-16 01:27:36 UTCdeploy automated deployed docsvc
- 2026-11-16 05:19:11 UTCdeploy automated deployed docsvc
- 2026-11-16 07:03:21 UTCdeploy automated deployed docsvc
- 2026-11-16 07:33:37 UTCdeploy automated deployed docsvc
History
- 2026-11-16 07:17:55 UTCreopened x-ray · severity=critical was=critical title=docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5073, 'fingerprint': '2d44aa8b4ea0694932cf', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/documents/folders', 'service': 'docsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794811800, 1794812700], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': 'd54f13665de9543057cc6c434f059c55524b4e40', 'ts': 1794812601, 'minutes_before': 14}], 'discriminator': 'POST /api/documents/folders'}, 'evidence': [{'label': 'docsvc service page', 'href': '/service/docsvc?cluster=master&from=1794811800&to=1794813475'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794811800&to=1794813475'}], 'provenance': {'source': 'rollup', 'collected_at': 1794813292, 'age_seconds': 190, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 59.2, 'status': 'open', 'first_seen': 1792497176, 'last_seen': 1794813475, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "docsvc's POST /api/documents/folders on master failed 5 of 5 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/documents/folders', 'level': 'high', 'told_level': None, 'episode_at': 1794813475, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2316305, 'stale_s': 6}
- 2026-11-16 09:50:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5073, 'fingerprint': '2d44aa8b4ea0694932cf', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/documents/folders', 'service': 'docsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794812400, 1794813300], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'docsvc', 'image': None, 'revision': 'd54f13665de9543057cc6c434f059c55524b4e40', 'ts': 1794812601, 'minutes_before': 23}], 'discriminator': 'POST /api/documents/folders'}, 'evidence': [{'label': 'docsvc service page', 'href': '/service/docsvc?cluster=master&from=1794812400&to=1794814013'}, {'label': 'Error budgets', 'href': '/budgets?cluster=master&from=1794812400&to=1794814013'}], 'provenance': {'source': 'rollup', 'collected_at': 1794813864, 'age_seconds': 155, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 63.2, 'status': 'resolved', 'first_seen': 1792497176, 'last_seen': 1794814013, 'resolved_at': 1794822612, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': "docsvc's POST /api/documents/folders on master failed 8 of 8 calls (100 %) in the last 15 min. automated changed docsvc 23 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/documents/folders', 'level': 'high', 'told_level': None, 'episode_at': 1794813475, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2325444, 'stale_s': 8607}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794813864 |
| age_seconds | 155 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_failing |