studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3
warning
4xx spike (403/401/404…)
test27
resolved
impact 38
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 16d ago · last observed 1d ago · observed 128× · #14944 · 112fd09ce25e5d44bd54
What happened template
4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 84, c4xx 84, share_pct 100.0, c401 84.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
The facts
| endpoint | GET /api/studio/triggers |
| service | studiosvc |
| calls | 84 |
| c4xx | 84 |
| share_pct | 100.00 |
| c401 | 84 |
| c403 | 0 |
| other_4xx | 0 |
| rejecting | yes |
| usual_per_window | 16.26 |
| baseline_days | 7 |
| window_min | 15 |
| severity_capped | critical → warning: test27 is a test-tier cluster |
| discriminator | GET /api/studio/triggers |
| window | 2026-11-17 05:10:00 UTC → 2026-11-17 05:25:00 UTC |
What is failing in studiosvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-17 05:10:00 UTC → 2026-11-17 05:25:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 17× | http error response returned to client since 2026-10-06 01:20:44 UTC |
| 6× | Request completed — 500 · could not create activation rule chain:activation rule \ since 2026-10-02 11:58:33 UTC |
The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.
- 2026-11-02 06:54:56 UTCthis finding studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3
- 2026-11-05 16:33:42 UTCfinding · warning studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)
- 2026-11-05 22:27:19 UTCfinding · info studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)
- 2026-11-14 00:00:00 UTClatency studiosvc p95 peaked at ≥ 2.0 min
- 2026-11-14 12:00:44 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:25:45 UTCfinding · warning studiosvc: a new error shape, ≥13006× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 14:26:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 00:00:00 UTCerrors studiosvc 5xx peaked at 7 in one five-minute bucket (median 0 in this window)
- 2026-11-15 16:03:03 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 16:28:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 17:53:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 18:48:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:09:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:30:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:53:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 05:47:14 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/triggers: permission denied — 96 of 96 calls (100 %; 401×96) in the last 15 min — it usually has none level=medium was_level=medium finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 96 of 96 calls (100 %; 401×96) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 96, 'c4xx': 96, 'share_pct': 100.0, 'c401': 96, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794806400, 1794807300], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794806400&to=1794807300&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794806400&to=1794807300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794807873, 'age_seconds': 167, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1793602496, 'last_seen': 1794808034, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The GET /api/studio/triggers endpoint in studiosvc is experiencing a 4xx spike: all 43 calls (100%) failed with 401 errors in the last 15 minutes, versus zero usual calls. This represents permission denied errors. Severity is capped at warning because test27 is a test-tier cluster.', 'narrative_kind': 'ai', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794808034, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1205544, 'stale_s': 6}
- 2026-11-16 06:22:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 36 of 36 calls (100 %; 401×36) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 36, 'c4xx': 36, 'share_pct': 100.0, 'c401': 36, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794807000, 1794807900], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794807000&to=1794807900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794807000&to=1794807900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808668, 'age_seconds': 245, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1793602496, 'last_seen': 1794808905, 'resolved_at': 1794810147, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 36 of 36 calls (100 %; 401×36) in the last 15 min — it usually has none. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 36, c4xx 36, share_pct 100.0, c401 36.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794808034, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1207661, 'stale_s': 1252}
- 2026-11-16 06:33:37 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/triggers: permission denied — 31 of 31 calls (100 %; 401×31) in the last 15 min — 148× its usual 0.2 level=medium was_level=medium finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 31 of 31 calls (100 %; 401×31) in the last 15 min — 148× its usual 0.2', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 31, 'c4xx': 31, 'share_pct': 100.0, 'c401': 31, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 0.21, 'baseline_days': 7, 'window': [1794809100, 1794810000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794809100&to=1794810000&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794809100&to=1794810000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794810679, 'age_seconds': 144, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1793602496, 'last_seen': 1794810817, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 36 of 36 calls (100 %; 401×36) in the last 15 min — it usually has none. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 36, c4xx 36, share_pct 100.0, c401 36.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794810817, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1208326, 'stale_s': 5}
- 2026-11-16 19:23:44 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 41 of 41 calls (100 %; 401×41) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 41, 'c4xx': 41, 'share_pct': 100.0, 'c401': 41, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 8.72, 'baseline_days': 7, 'window': [1794854400, 1794855300], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794854400&to=1794855300&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794854400&to=1794855300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794855841, 'age_seconds': 11, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1793602496, 'last_seen': 1794855844, 'resolved_at': 1794857024, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 67, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 41 of 41 calls (100 %; 401×41) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 41, c4xx 41, share_pct 100.0, c401 41.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794810817, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1254535, 'stale_s': 1187}
- 2026-11-16 19:33:28 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/triggers: permission denied — 47 of 47 calls (100 %; 401×47) in the last 15 min — 5× its usual 9.3 level=medium was_level=medium finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 47 of 47 calls (100 %; 401×47) in the last 15 min — 5× its usual 9.3', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 47, 'c4xx': 47, 'share_pct': 100.0, 'c401': 47, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 9.27, 'baseline_days': 7, 'window': [1794856200, 1794857100], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794856200&to=1794857100&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794856200&to=1794857100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794857606, 'age_seconds': 11, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1793602496, 'last_seen': 1794857608, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 68, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 41 of 41 calls (100 %; 401×41) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 41, c4xx 41, share_pct 100.0, c401 41.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794857608, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1255121, 'stale_s': 9}
- 2026-11-17 06:00:02 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 14944, 'fingerprint': '112fd09ce25e5d44bd54', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3', 'facts': {'endpoint': 'GET /api/studio/triggers', 'service': 'studiosvc', 'calls': 84, 'c4xx': 84, 'share_pct': 100.0, 'c401': 84, 'c403': 0, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 16.26, 'baseline_days': 7, 'window': [1794892200, 1794893100], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /api/studio/triggers'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794892200&to=1794893100&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test27&from=1794892200&to=1794893100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794893679, 'age_seconds': 11, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1793602496, 'last_seen': 1794893682, 'resolved_at': 1794895202, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 128, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 84, c4xx 84, share_pct 100.0, c401 84.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/triggers', 'level': 'medium', 'told_level': None, 'episode_at': 1794857608, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1292714, 'stale_s': 1528}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794893679 |
| age_seconds | 11 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |