Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 23:55:03.
› →|
updated 2d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6

critical 4xx spike (403/401/404…) production resolved impact 76 Owner: Murali Krishna Velamati (backup: Sambasiva Bandarupalli) opened 31d ago · last observed 3d ago · observed 62× · #5621 · cf8d58d2c0720beebe2e

What happened template

4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 39, c4xx 28, share_pct 71.7948717948718, c401 28.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; notifsvc GET /api/v1/notif/uievents returned 401 on 21 of 23 calls (91%) in 15 min, 8x its usual 2.6, since 2026-11-16 08:00 UTC.
Checked: 1 config-only deploy (rev 6db9855a7d34 by [email protected] at 05:10 UTC, 169 min before onset, no commits), 0 stored error lines, 0 4xx rows in the request log.
The same 401 spike has opened and self-resolved 5 times since 2026-11-10, including episodes with no deploys, so the config change is not linked.
The 401 comes from requestUser in wsHandler.go:49 rejecting the WebSocket auth (subprotocol/API key), which points at callers sending missing or expired credentials.
docproc opened a 403 spike 2 min later; check whether both share a caller or auth source.
Next step: open the 4xx calls list and identify which caller sends the rejected uievents connections.

low confidence

claude-opus-5-5 · 3d ago · $0.04 — every link above is evidence it read; nothing else was claimed.

The facts

endpointGET /api/v1/notif/uievents
servicenotifsvc
calls39
c4xx28
share_pct71.79
c40128
c4030
other_4xx0
rejectingno
usual_per_window2.57
baseline_days7
window_min15
discriminatorGET /api/v1/notif/uievents
window2026-11-16 16:05:00 UTC → 2026-11-16 16:20:00 UTC

The story — No notifsvc deploy or hand change is recorded in the hour before this opened. Everything below is notifsvc only.

  • 2026-10-19 06:46:26 UTCthis finding notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
  • 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc

History

  • 2026-11-16 08:28:23 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 23, 'c4xx': 21, 'share_pct': 91.30434782608695, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.58, 'baseline_days': 7, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794816000&to=1794816900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794816000&to=1794816900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817390, 'age_seconds': 320, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 71.8, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794817703, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 36 of 38 calls (95 %; 401×36) in the last 15 min — 8× its usual 4.4. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 38, c4xx 36, share_pct 94.73684210526316, c401 36.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794817703, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2425323, 'stale_s': 6}
  • 2026-11-16 08:44:24 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 23, 'c4xx': 21, 'share_pct': 91.30434782608695, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.58, 'baseline_days': 7, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794816000&to=1794816900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794816000&to=1794816900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817390, 'age_seconds': 320, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.8, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794817703, 'resolved_at': 1794818664, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 23, c4xx 21, share_pct 91.30434782608695, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794817703, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2426284, 'stale_s': 967}
  • 2026-11-16 13:29:19 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 33, 'c4xx': 26, 'share_pct': 78.78787878787878, 'c401': 26, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794834000, 1794834900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794834000&to=1794834900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794834000&to=1794834900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794835467, 'age_seconds': 302, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 71.1, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794835759, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 60, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 23, c4xx 21, share_pct 91.30434782608695, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794835759, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2443382, 'stale_s': 9}
  • 2026-11-16 13:48:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 33, 'c4xx': 26, 'share_pct': 78.78787878787878, 'c401': 26, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794834000, 1794834900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794834000&to=1794834900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794834000&to=1794834900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794835467, 'age_seconds': 302, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.1, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794835759, 'resolved_at': 1794836939, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 60, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 33, c4xx 26, share_pct 78.78787878787878, c401 26.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794835759, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2444561, 'stale_s': 1188}
  • 2026-11-16 16:22:34 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 44 calls (64 %; 401×28) in the last 15 min — 11× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 44 calls (64 %; 401×28) in the last 15 min — 11× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 44, 'c4xx': 28, 'share_pct': 63.63636363636363, 'c401': 28, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.6, 'baseline_days': 7, 'window': [1794844800, 1794845700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794844800&to=1794845700&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794844800&to=1794845700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846111, 'age_seconds': 50, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 70.9, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794846154, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 61, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 33, c4xx 26, share_pct 78.78787878787878, c401 26.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794846154, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2453774, 'stale_s': 6}
  • 2026-11-16 16:55:04 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 39, 'c4xx': 28, 'share_pct': 71.7948717948718, 'c401': 28, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.57, 'baseline_days': 7, 'window': [1794845100, 1794846000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794845100&to=1794846000&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794845100&to=1794846000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846624, 'age_seconds': 106, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 75.8, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794846721, 'resolved_at': 1794848104, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 62, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 39, c4xx 28, share_pct 71.7948717948718, c401 28.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794846154, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2455728, 'stale_s': 1393}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794846624
age_seconds106
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike