notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
critical
4xx spike (403/401/404…)
production
resolved
impact 76
Owner: Murali Krishna Velamati (backup: Sambasiva Bandarupalli)
opened 31d ago · last observed 3d ago · observed 62× · #5621 · cf8d58d2c0720beebe2e
What happened template
4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 39, c4xx 28, share_pct 71.7948717948718, c401 28.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; notifsvc GET /api/v1/notif/uievents returned 401 on 21 of 23 calls (91%) in 15 min, 8x its usual 2.6, since 2026-11-16 08:00 UTC.
Checked: 1 config-only deploy (rev 6db9855a7d34 by [email protected] at 05:10 UTC, 169 min before onset, no commits), 0 stored error lines, 0 4xx rows in the request log.
The same 401 spike has opened and self-resolved 5 times since 2026-11-10, including episodes with no deploys, so the config change is not linked.
The 401 comes from requestUser in wsHandler.go:49 rejecting the WebSocket auth (subprotocol/API key), which points at callers sending missing or expired credentials.
docproc opened a 403 spike 2 min later; check whether both share a caller or auth source.
Next step: open the 4xx calls list and identify which caller sends the rejected uievents connections.
low confidence
claude-opus-5-5 · 3d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /api/v1/notif/uievents |
| service | notifsvc |
| calls | 39 |
| c4xx | 28 |
| share_pct | 71.79 |
| c401 | 28 |
| c403 | 0 |
| other_4xx | 0 |
| rejecting | no |
| usual_per_window | 2.57 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/v1/notif/uievents |
| window | 2026-11-16 16:05:00 UTC → 2026-11-16 16:20:00 UTC |
The story — No notifsvc deploy or hand change is recorded in the hour before this opened. Everything below is notifsvc only.
- 2026-10-19 06:46:26 UTCthis finding notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
History
- 2026-11-16 08:28:23 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 23, 'c4xx': 21, 'share_pct': 91.30434782608695, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.58, 'baseline_days': 7, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794816000&to=1794816900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794816000&to=1794816900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817390, 'age_seconds': 320, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 71.8, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794817703, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 36 of 38 calls (95 %; 401×36) in the last 15 min — 8× its usual 4.4. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 38, c4xx 36, share_pct 94.73684210526316, c401 36.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794817703, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2425323, 'stale_s': 6}
- 2026-11-16 08:44:24 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 23, 'c4xx': 21, 'share_pct': 91.30434782608695, 'c401': 21, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.58, 'baseline_days': 7, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794816000&to=1794816900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794816000&to=1794816900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817390, 'age_seconds': 320, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.8, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794817703, 'resolved_at': 1794818664, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 23, c4xx 21, share_pct 91.30434782608695, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794817703, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2426284, 'stale_s': 967}
- 2026-11-16 13:29:19 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 33, 'c4xx': 26, 'share_pct': 78.78787878787878, 'c401': 26, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794834000, 1794834900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794834000&to=1794834900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794834000&to=1794834900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794835467, 'age_seconds': 302, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 71.1, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794835759, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 60, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 21 of 23 calls (91 %; 401×21) in the last 15 min — 8× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 23, c4xx 21, share_pct 91.30434782608695, c401 21.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794835759, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2443382, 'stale_s': 9}
- 2026-11-16 13:48:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 33, 'c4xx': 26, 'share_pct': 78.78787878787878, 'c401': 26, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.61, 'baseline_days': 7, 'window': [1794834000, 1794834900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794834000&to=1794834900&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794834000&to=1794834900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794835467, 'age_seconds': 302, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.1, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794835759, 'resolved_at': 1794836939, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 60, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 33, c4xx 26, share_pct 78.78787878787878, c401 26.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794835759, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2444561, 'stale_s': 1188}
- 2026-11-16 16:22:34 UTCreopened x-ray · severity=critical was=critical title=notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 44 calls (64 %; 401×28) in the last 15 min — 11× its usual 2.6 level=high was_level=high finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 44 calls (64 %; 401×28) in the last 15 min — 11× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 44, 'c4xx': 28, 'share_pct': 63.63636363636363, 'c401': 28, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.6, 'baseline_days': 7, 'window': [1794844800, 1794845700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794844800&to=1794845700&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794844800&to=1794845700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846111, 'age_seconds': 50, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 70.9, 'status': 'open', 'first_seen': 1792392386, 'last_seen': 1794846154, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 61, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 26 of 33 calls (79 %; 401×26) in the last 15 min — 10× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 33, c4xx 26, share_pct 78.78787878787878, c401 26.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794846154, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2453774, 'stale_s': 6}
- 2026-11-16 16:55:04 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5621, 'fingerprint': 'cf8d58d2c0720beebe2e', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'production', 'subject': 'notifsvc', 'title': 'notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6', 'facts': {'endpoint': 'GET /api/v1/notif/uievents', 'service': 'notifsvc', 'calls': 39, 'c4xx': 28, 'share_pct': 71.7948717948718, 'c401': 28, 'c403': 0, 'other_4xx': 0, 'rejecting': False, 'usual_per_window': 2.57, 'baseline_days': 7, 'window': [1794845100, 1794846000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/v1/notif/uievents'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=production&from=1794845100&to=1794846000&service=notifsvc&status=4xx'}, {'label': 'notifsvc service page', 'href': '/service/notifsvc?cluster=production&from=1794845100&to=1794846000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794846624, 'age_seconds': 106, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 75.8, 'status': 'resolved', 'first_seen': 1792392386, 'last_seen': 1794846721, 'resolved_at': 1794848104, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 62, 'narrative': '4xx spike (403/401/404…): notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6. Facts: endpoint GET /api/v1/notif/uievents, service notifsvc, calls 39, c4xx 28, share_pct 71.7948717948718, c401 28.', 'narrative_kind': 'template', 'discriminator': 'GET /api/v1/notif/uievents', 'level': 'high', 'told_level': 'high', 'episode_at': 1794846154, 'chat_thread': 'xray-alert-10-748fd5a928-cf8d58d2c0720beebe2e-1793702473', 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 2455728, 'stale_s': 1393}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794846624 |
| age_seconds | 106 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |