feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2
warning
4xx spike (403/401/404…)
test27
resolved
impact 38
opened 3d ago · last observed 2d ago · observed 3× · #17327 · d7e8dcd0ad71283b5c40
What happened template
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 62, c4xx 62, share_pct 100.0, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
The facts
| endpoint | GET /studies/:id/feed |
| service | feedsvc |
| calls | 62 |
| c4xx | 62 |
| share_pct | 100.00 |
| c401 | 0 |
| c403 | 61 |
| other_4xx | 1 |
| rejecting | yes |
| usual_per_window | 0.21 |
| baseline_days | 7 |
| window_min | 15 |
| severity_capped | critical → warning: test27 is a test-tier cluster |
| discriminator | GET /studies/:id/feed |
| window | 2026-11-16 13:55:00 UTC → 2026-11-16 14:10:00 UTC |
The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.
- 2026-11-15 19:38:48 UTCfinding · warning feedsvc GET /studies/:id/feed/week: 4xx — 54 of 54 calls (100 %; other 4xx×54) in the last 15 min — 267× its usual 0.2
- 2026-11-15 19:38:48 UTCthis finding feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2
- 2026-11-16 07:00:00 UTClatency feedsvc p95 peaked at 1.85 s
History
- 2026-11-15 19:38:48 UTCopened x-ray · severity=warning title=feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 70, 'c4xx': 70, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794770100, 1794771000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794770100&to=1794771000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794770100&to=1794771000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771526, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794771528, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794771528, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 6, 'stale_s': 6}
- 2026-11-15 21:43:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 70, 'c4xx': 70, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794770100, 1794771000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794770100&to=1794771000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794770100&to=1794771000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771526, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794771528, 'resolved_at': 1794779020, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 70, c4xx 70, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794771528, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 7498, 'stale_s': 7498}
- 2026-11-16 03:12:35 UTCreopened x-ray · severity=warning was=warning title=feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1 level=medium was_level=medium finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 71, 'c4xx': 71, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 2, 'rejecting': True, 'usual_per_window': 0.1, 'baseline_days': 7, 'window': [1794797100, 1794798000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794797100&to=1794798000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794797100&to=1794798000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798613, 'age_seconds': 149, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 70, c4xx 70, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 27233, 'stale_s': 6}
- 2026-11-16 05:18:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 71, 'c4xx': 71, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 2, 'rejecting': True, 'usual_per_window': 0.1, 'baseline_days': 7, 'window': [1794797100, 1794798000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794797100&to=1794798000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794797100&to=1794798000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798613, 'age_seconds': 149, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794798755, 'resolved_at': 1794806332, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 71, c4xx 71, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 34810, 'stale_s': 7583}
- 2026-11-16 14:26:30 UTCreopened x-ray · severity=warning was=warning title=feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2 level=medium was_level=medium finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 62, 'c4xx': 62, 'share_pct': 100.0, 'c401': 0, 'c403': 61, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.21, 'baseline_days': 7, 'window': [1794837300, 1794838200], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794837300&to=1794838200&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794837300&to=1794838200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794839045, 'age_seconds': 154, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794839190, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 71, c4xx 71, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794839190, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 67671, 'stale_s': 9}
- 2026-11-16 16:32:01 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 62, 'c4xx': 62, 'share_pct': 100.0, 'c401': 0, 'c403': 61, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.21, 'baseline_days': 7, 'window': [1794837300, 1794838200], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794837300&to=1794838200&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794837300&to=1794838200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794839045, 'age_seconds': 154, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794839190, 'resolved_at': 1794846721, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 62, c4xx 62, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794839190, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 75201, 'stale_s': 7539}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794839045 |
| age_seconds | 154 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |