Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 09:21:07.
› →|
updated 22h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1

warning 4xx spike (403/401/404…) master resolved impact 47 Owner: Suresh Boddapati (backup: Madhava Mukkamala) opened 7d ago · last observed 22h ago · observed 16× · #17065 · ff40fc298073211b4751

What happened template

4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 210, c4xx 37, share_pct 17.61904761904762, c401 0.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

The facts

endpointGET /api/studio/protocols/:id/visits/:id
servicestudiosvc
calls210
c4xx37
share_pct17.62
c4010
c4030
other_4xx37
rejectingno
usual_per_window1.10
baseline_days7
window_min15
discriminatorGET /api/studio/protocols/:id/visits/:id
window2026-11-17 09:50:00 UTC → 2026-11-17 10:05:00 UTC

Adjacent deploys — adjacency, not proof

automatedstudiosvc · 56 min before

The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.

History

  • 2026-11-15 08:04:18 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 250 calls (14 %; other 4xx×35) in the last 15 min — 48× its usual 0.7 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 250 calls (14 %; other 4xx×35) in the last 15 min — 48× its usual 0.7', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 250, 'c4xx': 35, 'share_pct': 14.0, 'c401': 0, 'c403': 0, 'other_4xx': 35, 'rejecting': False, 'usual_per_window': 0.73, 'baseline_days': 7, 'window': [1794728100, 1794729000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794728100&to=1794729000&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794728100&to=1794729000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794729581, 'age_seconds': 283, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 42.7, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794729858, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 284 calls (12 %; other 4xx×35) in the last 15 min — 54× its usual 0.6. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 284, c4xx 35, share_pct 12.32394366197183, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794729858, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 346265, 'stale_s': 5}
  • 2026-11-15 10:09:46 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 250 calls (14 %; other 4xx×35) in the last 15 min — 48× its usual 0.7', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 250, 'c4xx': 35, 'share_pct': 14.0, 'c401': 0, 'c403': 0, 'other_4xx': 35, 'rejecting': False, 'usual_per_window': 0.73, 'baseline_days': 7, 'window': [1794728100, 1794729000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794728100&to=1794729000&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794728100&to=1794729000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794729581, 'age_seconds': 283, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 47.7, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794729858, 'resolved_at': 1794737386, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 250 calls (14 %; other 4xx×35) in the last 15 min — 48× its usual 0.7. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 250, c4xx 35, share_pct 14.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794729858, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 353792, 'stale_s': 7532}
  • 2026-11-16 07:34:54 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 55 of 178 calls (31 %; 403×1, other 4xx×54) in the last 15 min — 66× its usual 0.8 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 55 of 178 calls (31 %; 403×1, other 4xx×54) in the last 15 min — 66× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 178, 'c4xx': 55, 'share_pct': 30.89887640449438, 'c401': 0, 'c403': 1, 'other_4xx': 54, 'rejecting': False, 'usual_per_window': 0.84, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794813000&to=1794813900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814423, 'age_seconds': 78, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 36.0, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794814494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 35 of 250 calls (14 %; other 4xx×35) in the last 15 min — 48× its usual 0.7. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 250, c4xx 35, share_pct 14.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 430903, 'stale_s': 7}
  • 2026-11-16 07:51:56 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 55 of 178 calls (31 %; 403×1, other 4xx×54) in the last 15 min — 66× its usual 0.8', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 178, 'c4xx': 55, 'share_pct': 30.89887640449438, 'c401': 0, 'c403': 1, 'other_4xx': 54, 'rejecting': False, 'usual_per_window': 0.84, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794813000&to=1794813900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814423, 'age_seconds': 78, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 41.0, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794814494, 'resolved_at': 1794815516, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 55 of 178 calls (31 %; 403×1, other 4xx×54) in the last 15 min — 66× its usual 0.8. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 178, c4xx 55, share_pct 30.89887640449438, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 431925, 'stale_s': 1029}
  • 2026-11-16 13:13:39 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 132 calls (15 %; other 4xx×20) in the last 15 min — 22× its usual 0.9 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 132 calls (15 %; other 4xx×20) in the last 15 min — 22× its usual 0.9', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 132, 'c4xx': 20, 'share_pct': 15.151515151515152, 'c401': 0, 'c403': 0, 'other_4xx': 20, 'rejecting': False, 'usual_per_window': 0.93, 'baseline_days': 7, 'window': [1794832800, 1794833700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794832800&to=1794833700&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794832800&to=1794833700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794834347, 'age_seconds': 482, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 36.6, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794834819, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 12, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 55 of 178 calls (31 %; 403×1, other 4xx×54) in the last 15 min — 66× its usual 0.8. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 178, c4xx 55, share_pct 30.89887640449438, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794834819, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 451231, 'stale_s': 10}
  • 2026-11-16 13:29:19 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 132 calls (15 %; other 4xx×20) in the last 15 min — 22× its usual 0.9', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 132, 'c4xx': 20, 'share_pct': 15.151515151515152, 'c401': 0, 'c403': 0, 'other_4xx': 20, 'rejecting': False, 'usual_per_window': 0.93, 'baseline_days': 7, 'window': [1794832800, 1794833700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794832800&to=1794833700&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794832800&to=1794833700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794834347, 'age_seconds': 482, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 41.6, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794834819, 'resolved_at': 1794835759, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 12, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 132 calls (15 %; other 4xx×20) in the last 15 min — 22× its usual 0.9. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 132, c4xx 20, share_pct 15.151515151515152, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794834819, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 452170, 'stale_s': 949}
  • 2026-11-16 15:08:36 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 21 of 107 calls (20 %; other 4xx×21) in the last 15 min — 22× its usual 1.0 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 21 of 107 calls (20 %; other 4xx×21) in the last 15 min — 22× its usual 1.0', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 107, 'c4xx': 21, 'share_pct': 19.626168224299064, 'c401': 0, 'c403': 0, 'other_4xx': 21, 'rejecting': False, 'usual_per_window': 0.97, 'baseline_days': 7, 'window': [1794840000, 1794840900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794840000&to=1794840900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794840000&to=1794840900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794841537, 'age_seconds': 188, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 35.5, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794841716, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 13, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 132 calls (15 %; other 4xx×20) in the last 15 min — 22× its usual 0.9. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 132, c4xx 20, share_pct 15.151515151515152, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794841716, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 458127, 'stale_s': 9}
  • 2026-11-16 15:28:22 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 21 of 107 calls (20 %; other 4xx×21) in the last 15 min — 22× its usual 1.0', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 107, 'c4xx': 21, 'share_pct': 19.626168224299064, 'c401': 0, 'c403': 0, 'other_4xx': 21, 'rejecting': False, 'usual_per_window': 0.97, 'baseline_days': 7, 'window': [1794840000, 1794840900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794840000&to=1794840900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794840000&to=1794840900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794841537, 'age_seconds': 188, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 40.5, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794841716, 'resolved_at': 1794842902, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 13, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 21 of 107 calls (20 %; other 4xx×21) in the last 15 min — 22× its usual 1.0. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 107, c4xx 21, share_pct 19.626168224299064, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794841716, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 459312, 'stale_s': 1194}
  • 2026-11-16 23:50:57 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 60 calls (33 %; other 4xx×20) in the last 15 min — 20× its usual 1.0 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 20 of 60 calls (33 %; other 4xx×20) in the last 15 min — 20× its usual 1.0', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 60, 'c4xx': 20, 'share_pct': 33.333333333333336, 'c401': 0, 'c403': 0, 'other_4xx': 20, 'rejecting': False, 'usual_per_window': 1.02, 'baseline_days': 7, 'window': [1794871800, 1794872700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794871800&to=1794872700&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794871800&to=1794872700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794873017, 'age_seconds': 47, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 43.2, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794873057, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 14, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 21 of 107 calls (20 %; other 4xx×21) in the last 15 min — 22× its usual 1.0. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 107, c4xx 21, share_pct 19.626168224299064, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794873057, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 489466, 'stale_s': 7}
  • 2026-11-17 01:15:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 36 of 81 calls (44 %; other 4xx×36) in the last 15 min — 34× its usual 1.0', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 81, 'c4xx': 36, 'share_pct': 44.44444444444444, 'c401': 0, 'c403': 0, 'other_4xx': 36, 'rejecting': False, 'usual_per_window': 1.05, 'baseline_days': 7, 'window': [1794875400, 1794876300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794875400&to=1794876300&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794875400&to=1794876300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794876896, 'age_seconds': 209, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.6, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794877096, 'resolved_at': 1794878127, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 15, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 36 of 81 calls (44 %; other 4xx×36) in the last 15 min — 34× its usual 1.0. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 81, c4xx 36, share_pct 44.44444444444444, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794873057, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 494537, 'stale_s': 1039}
  • 2026-11-17 10:21:45 UTCreopened x-ray · severity=warning was=warning title=studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1 level=medium was_level=medium finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 210, 'c4xx': 37, 'share_pct': 17.61904761904762, 'c401': 0, 'c403': 0, 'other_4xx': 37, 'rejecting': False, 'usual_per_window': 1.1, 'baseline_days': 7, 'window': [1794909000, 1794909900], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'studiosvc', 'image': None, 'revision': 'e4eb245d2dfc686c6d55e4d5abf48687fcd3f7b3', 'ts': 1794907543, 'minutes_before': 56}], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794909000&to=1794909900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794909000&to=1794909900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794910455, 'age_seconds': 462, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 42.1, 'status': 'open', 'first_seen': 1794383598, 'last_seen': 1794910905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 36 of 81 calls (44 %; other 4xx×36) in the last 15 min — 34× its usual 1.0. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 81, c4xx 36, share_pct 44.44444444444444, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794910905, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 527319, 'stale_s': 12}
  • 2026-11-17 10:40:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17065, 'fingerprint': 'ff40fc298073211b4751', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'master', 'subject': 'studiosvc', 'title': 'studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1', 'facts': {'endpoint': 'GET /api/studio/protocols/:id/visits/:id', 'service': 'studiosvc', 'calls': 210, 'c4xx': 37, 'share_pct': 17.61904761904762, 'c401': 0, 'c403': 0, 'other_4xx': 37, 'rejecting': False, 'usual_per_window': 1.1, 'baseline_days': 7, 'window': [1794909000, 1794909900], 'window_min': 15, 'recent_deploys': [{'who': 'automated', 'resource': 'studiosvc', 'image': None, 'revision': 'e4eb245d2dfc686c6d55e4d5abf48687fcd3f7b3', 'ts': 1794907543, 'minutes_before': 56}], 'discriminator': 'GET /api/studio/protocols/:id/visits/:id'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794909000&to=1794909900&service=studiosvc&status=4xx'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=master&from=1794909000&to=1794909900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794910455, 'age_seconds': 462, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 47.1, 'status': 'resolved', 'first_seen': 1794383598, 'last_seen': 1794910905, 'resolved_at': 1794912041, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': '4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 210, c4xx 37, share_pct 17.61904761904762, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/studio/protocols/:id/visits/:id', 'level': 'medium', 'told_level': None, 'episode_at': 1794910905, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 528454, 'stale_s': 1147}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794910455
age_seconds462
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike