studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8
critical
4xx spike (403/401/404…)
master
resolved
impact 65
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 15d ago · last observed 1d ago · observed 25× · #15927 · dfd90ba310938cb8c4c8
What happened template
4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 36, share_pct 73.46938775510205, c401 36.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; studiosvc GET /api/studio/events returned 401 on 21 of 49 calls (43 %) in 15 min, 28× its usual 0.8 per window.
Onset 15:25 UTC on 2026-11-16; the first 401 in the request log is at 15:28 UTC. This finding has recurred many times since 2026-11-03, and earlier episodes cleared on their own in about 20–30 min.
Checked: 0 stored error lines, 4 config-only autosync deploys (the last at 08:05 UTC, 7 h before) with no image or commit change, and the SSE handler. The handler returns 401 when authlib.GetRequestContext fails and logs a warning, but no such lines are stored.
Next step: open one of the 401 traces to see whether the auth context or Organization header is missing, and compare with the open sibling finding on /api/studio/triggers (100 % 401).
low confidence
claude-opus-5-5 · 1d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /api/studio/events |
| service | studiosvc |
| calls | 49 |
| c4xx | 36 |
| share_pct | 73.47 |
| c401 | 36 |
| c403 | 0 |
| other_4xx | 0 |
| rejecting | no |
| usual_per_window | 0.81 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/studio/events |
| window | 2026-11-17 06:25:00 UTC → 2026-11-17 06:40:00 UTC |
What is failing in studiosvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-17 06:25:00 UTC → 2026-11-17 06:40:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 249× | http error response returned to client since 2026-10-06 15:35:21 UTC |
| 3× | Request completed — 500 · response set with same title and variablename already exists since 2026-11-11 03:34:02 UTC |
| 3× | response set creation failed — response set with same title and variablename already exists since 2026-11-17 06:56:38 UTC |
The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.
- 2026-11-03 04:38:01 UTCthis finding studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8
- 2026-11-03 06:38:01 UTCfinding · info studiosvc p95 2.0 min vs a normal 369 ms ± 242 ms for Mon 18:00 UTC (3 of the last 4 checks)
- 2026-11-11 07:53:18 UTCfinding · warning studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1
- 2026-11-14 00:00:00 UTClatency studiosvc p95 peaked at ≥ 2.0 min
- 2026-11-14 15:15:50 UTCdeploy automated deployed studiosvc
- 2026-11-15 00:00:00 UTCerrors studiosvc 5xx peaked at 59 in one five-minute bucket (median 0 in this window)
- 2026-11-15 01:01:27 UTCdeploy automated deployed studiosvc
- 2026-11-15 01:29:10 UTCdeploy automated deployed studiosvc
- 2026-11-16 01:19:27 UTCdeploy automated deployed studiosvc
- 2026-11-16 01:27:19 UTCdeploy automated deployed studiosvc
- 2026-11-16 05:19:14 UTCdeploy automated deployed studiosvc
- 2026-11-16 08:05:47 UTCdeploy automated deployed studiosvc
- 2026-11-16 10:19:38 UTCfinding · warning studiosvc shape 8D3802C7: p95 1.73 s now vs 746 ms earlier (2.3×, 232 runs)
- 2026-11-16 15:39:08 UTCdeploy automated deployed studiosvc
- 2026-11-16 18:19:54 UTCdeploy automated deployed studiosvc
- 2026-11-17 07:05:31 UTCdeploy automated deployed studiosvc
History
- 2026-11-16 15:51:04 UTCreopened x-ray · severity=critical was=critical title=studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8 level=high was_level=high finding={'id': 15927, 'fingerprint': 'dfd90ba310938cb8c4c8', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 49, 'c4xx': 21, 'share_pct': 42.857142857142854, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.75, 'baseline_days': 7, 'window': [1794842700, 1794843600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794842700&to=1794843600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794842700&to=1794843600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794844162, 'age_seconds': 108, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 64.2, 'status': 'open', 'first_seen': 1793680681, 'last_seen': 1794844264, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 24, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 39 of 91 calls (43 %; 401×39) in the last 15 min — 70× its usual 0.6. Facts: endpoint GET /api/studio/events, service studiosvc, calls 91, c4xx 39, share_pct 42.857142857142854, c401 39.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'high', 'told_level': None, 'episode_at': 1794844264, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1163589, 'stale_s': 6}
- 2026-11-16 16:07:15 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 15927, 'fingerprint': 'dfd90ba310938cb8c4c8', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 49, 'c4xx': 21, 'share_pct': 42.857142857142854, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.75, 'baseline_days': 7, 'window': [1794842700, 1794843600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794842700&to=1794843600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794842700&to=1794843600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794844162, 'age_seconds': 108, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 69.2, 'status': 'resolved', 'first_seen': 1793680681, 'last_seen': 1794844264, 'resolved_at': 1794845235, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 24, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 21, share_pct 42.857142857142854, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'high', 'told_level': None, 'episode_at': 1794844264, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1164560, 'stale_s': 977}
- 2026-11-17 06:55:37 UTCreopened x-ray · severity=critical was=critical title=studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8 level=high was_level=high finding={'id': 15927, 'fingerprint': 'dfd90ba310938cb8c4c8', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 49, 'c4xx': 36, 'share_pct': 73.46938775510205, 'c401': 36, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.81, 'baseline_days': 7, 'window': [1794896700, 1794897600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794896700&to=1794897600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794896700&to=1794897600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794898162, 'age_seconds': 383, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 60.1, 'status': 'open', 'first_seen': 1793680681, 'last_seen': 1794898537, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 25, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 21 of 49 calls (43 %; 401×21) in the last 15 min — 28× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 21, share_pct 42.857142857142854, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'high', 'told_level': None, 'episode_at': 1794898537, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1217863, 'stale_s': 7}
- 2026-11-17 07:24:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 15927, 'fingerprint': 'dfd90ba310938cb8c4c8', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/events', 'service': 'studiosvc', 'calls': 49, 'c4xx': 36, 'share_pct': 73.46938775510205, 'c401': 36, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 0.81, 'baseline_days': 7, 'window': [1794896700, 1794897600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/events'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794896700&to=1794897600&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794896700&to=1794897600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794898162, 'age_seconds': 383, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 65.1, 'status': 'resolved', 'first_seen': 1793680681, 'last_seen': 1794898537, 'resolved_at': 1794900249, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 25, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 36, share_pct 73.46938775510205, c401 36.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/events', 'level': 'high', 'told_level': None, 'episode_at': 1794898537, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1219577, 'stale_s': 1721}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794898162 |
| age_seconds | 383 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |