Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 6 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 04:57:54.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2

warning 4xx spike (403/401/404…) test27 resolved impact 38 opened 3d ago · last observed 2d ago · observed 3× · #17327 · d7e8dcd0ad71283b5c40

What happened template

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 62, c4xx 62, share_pct 100.0, c401 0.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

The facts

endpointGET /studies/:id/feed
servicefeedsvc
calls62
c4xx62
share_pct100.00
c4010
c40361
other_4xx1
rejectingyes
usual_per_window0.21
baseline_days7
window_min15
severity_cappedcritical → warning: test27 is a test-tier cluster
discriminatorGET /studies/:id/feed
window2026-11-16 13:55:00 UTC → 2026-11-16 14:10:00 UTC

The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.

History

  • 2026-11-15 19:38:48 UTCopened x-ray · severity=warning title=feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 70, 'c4xx': 70, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794770100, 1794771000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794770100&to=1794771000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794770100&to=1794771000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771526, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794771528, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794771528, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 6, 'stale_s': 6}
  • 2026-11-15 21:43:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 70, 'c4xx': 70, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794770100, 1794771000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794770100&to=1794771000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794770100&to=1794771000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794771526, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794771528, 'resolved_at': 1794779020, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 70, c4xx 70, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794771528, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 7498, 'stale_s': 7498}
  • 2026-11-16 03:12:35 UTCreopened x-ray · severity=warning was=warning title=feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1 level=medium was_level=medium finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 71, 'c4xx': 71, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 2, 'rejecting': True, 'usual_per_window': 0.1, 'baseline_days': 7, 'window': [1794797100, 1794798000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794797100&to=1794798000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794797100&to=1794798000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798613, 'age_seconds': 149, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 70 of 70 calls (100 %; 403×69, other 4xx×1) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 70, c4xx 70, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 27233, 'stale_s': 6}
  • 2026-11-16 05:18:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 71, 'c4xx': 71, 'share_pct': 100.0, 'c401': 0, 'c403': 69, 'other_4xx': 2, 'rejecting': True, 'usual_per_window': 0.1, 'baseline_days': 7, 'window': [1794797100, 1794798000], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794797100&to=1794798000&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794797100&to=1794798000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794798613, 'age_seconds': 149, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794798755, 'resolved_at': 1794806332, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 71, c4xx 71, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794798755, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 34810, 'stale_s': 7583}
  • 2026-11-16 14:26:30 UTCreopened x-ray · severity=warning was=warning title=feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2 level=medium was_level=medium finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 62, 'c4xx': 62, 'share_pct': 100.0, 'c401': 0, 'c403': 61, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.21, 'baseline_days': 7, 'window': [1794837300, 1794838200], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794837300&to=1794838200&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794837300&to=1794838200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794839045, 'age_seconds': 154, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794771528, 'last_seen': 1794839190, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 71 of 71 calls (100 %; 403×69, other 4xx×2) in the last 15 min — 682× its usual 0.1. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 71, c4xx 71, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794839190, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 67671, 'stale_s': 9}
  • 2026-11-16 16:32:01 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17327, 'fingerprint': 'd7e8dcd0ad71283b5c40', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'test27', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed', 'service': 'feedsvc', 'calls': 62, 'c4xx': 62, 'share_pct': 100.0, 'c401': 0, 'c403': 61, 'other_4xx': 1, 'rejecting': True, 'usual_per_window': 0.21, 'baseline_days': 7, 'window': [1794837300, 1794838200], 'window_min': 15, 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': 'GET /studies/:id/feed'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=test27&from=1794837300&to=1794838200&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=test27&from=1794837300&to=1794838200'}], 'provenance': {'source': 'rollup', 'collected_at': 1794839045, 'age_seconds': 154, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.5, 'status': 'resolved', 'first_seen': 1794771528, 'last_seen': 1794839190, 'resolved_at': 1794846721, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 62, c4xx 62, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed', 'level': 'medium', 'told_level': None, 'episode_at': 1794839190, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 75201, 'stale_s': 7539}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794839045
age_seconds154
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike