feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call
critical
4xx spike (403/401/404…)
master
resolved
impact 65
opened 6d ago · last observed 1d ago · observed 57× · #17148 · a923a4f4b90bf1e4f833
What happened template
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see.
low confidence
claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /studies/:id/feed/week |
| service | feedsvc |
| calls | 24 |
| c4xx | 23 |
| share_pct | 95.83 |
| c401 | 0 |
| c403 | 23 |
| other_4xx | 0 |
| rejecting | yes |
| usual_per_window | 60.30 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /studies/:id/feed/week |
| window | 2026-11-17 06:10:00 UTC → 2026-11-17 06:25:00 UTC |
The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.
- 2026-11-12 18:03:36 UTCfinding · critical feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
- 2026-11-12 18:03:36 UTCthis finding feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call
- 2026-11-13 19:37:11 UTCfinding · info feedsvc request rate 27.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-11-15 22:28:32 UTCdeploy automated deployed feedsvc
- 2026-11-16 01:17:17 UTCdeploy automated deployed feedsvc
- 2026-11-16 07:43:55 UTCfinding · info feedsvc Neo4j load 83 q/min vs a normal 0 q/min ± 20 q/min for Mon 18:00 UTC (4 of the last 4 checks)
- 2026-11-16 11:17:28 UTCdeploy automated deployed feedsvc
- 2026-11-16 18:04:02 UTCdeploy automated deployed feedsvc
- 2026-11-16 22:26:02 UTCdeploy automated deployed feedsvc
- 2026-11-17 02:53:50 UTCdeploy automated deployed feedsvc
- 2026-11-17 05:43:34 UTCfinding · critical feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2
History
- 2026-11-16 07:34:54 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed/week: permission denied — 580 of 580 calls (100 %; 403×580) in the last 15 min — 257× its usual 2.3 level=high was_level=high finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 580 of 580 calls (100 %; 403×580) in the last 15 min — 257× its usual 2.3', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 580, 'c4xx': 580, 'share_pct': 100.0, 'c401': 0, 'c403': 580, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 2.26, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794813000&to=1794813900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814423, 'age_seconds': 78, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 66.8, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794814494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 74 of 75 calls (99 %; 403×74) in the last 15 min — 36× its usual 2.1. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 75, c4xx 74, share_pct 98.66666666666667, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 307885, 'stale_s': 7}
- 2026-11-16 08:15:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 775 of 775 calls (100 %; 403×775) in the last 15 min — 343× its usual 2.3', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 775, 'c4xx': 775, 'share_pct': 100.0, 'c401': 0, 'c403': 775, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 2.26, 'baseline_days': 7, 'window': [1794813900, 1794814800], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794813900&to=1794814800&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794813900&to=1794814800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794815405, 'age_seconds': 118, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 73.0, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794815516, 'resolved_at': 1794816941, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 775 of 775 calls (100 %; 403×775) in the last 15 min — 343× its usual 2.3. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 775, c4xx 775, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 310333, 'stale_s': 1433}
- 2026-11-16 08:53:28 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed/week: permission denied — 45 of 45 calls (100 %; 403×45) in the last 15 min — 10× its usual 4.3 level=high was_level=high finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 45 of 45 calls (100 %; 403×45) in the last 15 min — 10× its usual 4.3', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 45, 'c4xx': 45, 'share_pct': 100.0, 'c401': 0, 'c403': 45, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 4.29, 'baseline_days': 7, 'window': [1794817800, 1794818700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794817800&to=1794818700&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794817800&to=1794818700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794819084, 'age_seconds': 130, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 58.8, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794819208, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 19, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 775 of 775 calls (100 %; 403×775) in the last 15 min — 343× its usual 2.3. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 775, c4xx 775, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 312597, 'stale_s': 5}
- 2026-11-16 11:24:39 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 902 of 904 calls (100 %; 403×902) in the last 15 min — 193× its usual 4.7', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 904, 'c4xx': 902, 'share_pct': 99.77876106194691, 'c401': 0, 'c403': 902, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 4.67, 'baseline_days': 7, 'window': [1794821700, 1794822600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794821700&to=1794822600&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794821700&to=1794822600'}], 'provenance': {'source': 'rollup', 'collected_at': 1794823281, 'age_seconds': 1108, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 74.2, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794824378, 'resolved_at': 1794828279, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 22, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 902 of 904 calls (100 %; 403×902) in the last 15 min — 193× its usual 4.7. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 904, c4xx 902, share_pct 99.77876106194691, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 321671, 'stale_s': 3909}
- 2026-11-16 12:06:52 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed/week: permission denied — 70 of 71 calls (99 %; 403×70) in the last 15 min — 6× its usual 11.0 level=high was_level=high finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 70 of 71 calls (99 %; 403×70) in the last 15 min — 6× its usual 11.0', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 71, 'c4xx': 70, 'share_pct': 98.59154929577464, 'c401': 0, 'c403': 70, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 11.0, 'baseline_days': 7, 'window': [1794828000, 1794828900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794828000&to=1794828900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794828000&to=1794828900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794829599, 'age_seconds': 1222, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 65.5, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794830812, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 23, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 902 of 904 calls (100 %; 403×902) in the last 15 min — 193× its usual 4.7. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 904, c4xx 902, share_pct 99.77876106194691, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 324205, 'stale_s': 9}
- 2026-11-16 19:23:44 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 411 of 412 calls (100 %; 403×411) in the last 15 min — 8× its usual 52.7', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 412, 'c4xx': 411, 'share_pct': 99.75728155339806, 'c401': 0, 'c403': 411, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 52.7, 'baseline_days': 7, 'window': [1794854400, 1794855300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794854400&to=1794855300&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794854400&to=1794855300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794855670, 'age_seconds': 182, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 75.1, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794855844, 'resolved_at': 1794857024, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 55, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 411 of 412 calls (100 %; 403×411) in the last 15 min — 8× its usual 52.7. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 412, c4xx 411, share_pct 99.75728155339806, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 350415, 'stale_s': 1187}
- 2026-11-17 02:35:20 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed/week: permission denied — 84 of 84 calls (100 %; 403×84) in the last 15 min — it rejects nearly every call level=high was_level=high finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 84 of 84 calls (100 %; 403×84) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 84, 'c4xx': 84, 'share_pct': 100.0, 'c401': 0, 'c403': 84, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 60.1, 'baseline_days': 7, 'window': [1794881100, 1794882000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794881100&to=1794882000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794881100&to=1794882000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794882595, 'age_seconds': 333, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 61.8, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794882920, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 411 of 412 calls (100 %; 403×411) in the last 15 min — 8× its usual 52.7. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 412, c4xx 411, share_pct 99.75728155339806, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794882920, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 376312, 'stale_s': 8}
- 2026-11-17 03:35:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 84 of 84 calls (100 %; 403×84) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 84, 'c4xx': 84, 'share_pct': 100.0, 'c401': 0, 'c403': 84, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 60.1, 'baseline_days': 7, 'window': [1794881100, 1794882000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794881100&to=1794882000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794881100&to=1794882000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794882595, 'age_seconds': 333, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 66.8, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794882920, 'resolved_at': 1794886558, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 84 of 84 calls (100 %; 403×84) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 84, c4xx 84, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794882920, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 379951, 'stale_s': 3647}
- 2026-11-17 06:42:35 UTCreopened x-ray · severity=critical was=critical title=feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call level=high was_level=high finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 24, 'c4xx': 23, 'share_pct': 95.83333333333333, 'c401': 0, 'c403': 23, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 60.3, 'baseline_days': 7, 'window': [1794895800, 1794896700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794895800&to=1794896700&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794895800&to=1794896700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794897339, 'age_seconds': 425, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 60.3, 'status': 'open', 'first_seen': 1794506616, 'last_seen': 1794897755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 84 of 84 calls (100 %; 403×84) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 84, c4xx 84, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794897755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 391148, 'stale_s': 9}
- 2026-11-17 07:09:11 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17148, 'fingerprint': 'a923a4f4b90bf1e4f833', 'kind': 'endpoint.4xx_spike', 'severity': 'critical', 'cluster': 'master', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call', 'facts': {'endpoint': 'GET /studies/:id/feed/week', 'service': 'feedsvc', 'calls': 24, 'c4xx': 23, 'share_pct': 95.83333333333333, 'c401': 0, 'c403': 23, 'other_4xx': 0, 'rejecting': True, 'usual_per_window': 60.3, 'baseline_days': 7, 'window': [1794895800, 1794896700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/week'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=master&from=1794895800&to=1794896700&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=master&from=1794895800&to=1794896700'}], 'provenance': {'source': 'rollup', 'collected_at': 1794897339, 'age_seconds': 425, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 65.3, 'status': 'resolved', 'first_seen': 1794506616, 'last_seen': 1794897755, 'resolved_at': 1794899351, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/week', 'level': 'high', 'told_level': None, 'episode_at': 1794897755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 392744, 'stale_s': 1605}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794897339 |
| age_seconds | 425 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |