Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:39:22.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2

warning 4xx spike (403/401/404…) automation resolved impact 31 opened 1d ago · last observed 1d ago · observed 6× · #17497 · a53207ef156d5030e292

What happened template

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 30, c4xx 30, share_pct 100.0, c401 0.

An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.

The facts

endpointGET /studies/:id/feed/news
servicefeedsvc
calls30
c4xx30
share_pct100.00
c4010
c4030
other_4xx30
rejectingyes
usual_per_window0.23
baseline_days7
window_min15
discriminatorGET /studies/:id/feed/news
window2026-11-17 07:10:00 UTC → 2026-11-17 07:25:00 UTC

The story — No feedsvc deploy or hand change is recorded in the hour before this opened. Everything below is feedsvc only.

History

  • 2026-11-17 02:44:54 UTCopened x-ray · severity=warning title=feedsvc GET /studies/:id/feed/news: 4xx — 21 of 21 calls (100 %; other 4xx×21) in the last 15 min — it usually has none finding={'id': 17497, 'fingerprint': 'a53207ef156d5030e292', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/news: 4xx — 21 of 21 calls (100 %; other 4xx×21) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed/news', 'service': 'feedsvc', 'calls': 21, 'c4xx': 21, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 21, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794882000, 1794882900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/news'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794882000&to=1794882900&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=automation&from=1794882000&to=1794882900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794883363, 'age_seconds': 138, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.4, 'status': 'open', 'first_seen': 1794883494, 'last_seen': 1794883494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'GET /studies/:id/feed/news', 'level': 'medium', 'told_level': None, 'episode_at': 1794883494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 7, 'stale_s': 7}
  • 2026-11-17 03:35:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17497, 'fingerprint': 'a53207ef156d5030e292', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/news: 4xx — 23 of 23 calls (100 %; other 4xx×23) in the last 15 min — it usually has none', 'facts': {'endpoint': 'GET /studies/:id/feed/news', 'service': 'feedsvc', 'calls': 23, 'c4xx': 23, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 23, 'rejecting': True, 'usual_per_window': 0.0, 'baseline_days': 7, 'window': [1794883500, 1794884400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/news'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794883500&to=1794884400&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=automation&from=1794883500&to=1794884400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794884967, 'age_seconds': 220, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.7, 'status': 'resolved', 'first_seen': 1794883494, 'last_seen': 1794885178, 'resolved_at': 1794886558, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 23 of 23 calls (100 %; other 4xx×23) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 23, c4xx 23, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/news', 'level': 'medium', 'told_level': None, 'episode_at': 1794883494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 3073, 'stale_s': 1389}
  • 2026-11-17 07:09:11 UTCreopened x-ray · severity=warning was=warning title=feedsvc GET /studies/:id/feed/news: 4xx — 33 of 33 calls (100 %; other 4xx×33) in the last 15 min — 191× its usual 0.2 level=medium was_level=medium finding={'id': 17497, 'fingerprint': 'a53207ef156d5030e292', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/news: 4xx — 33 of 33 calls (100 %; other 4xx×33) in the last 15 min — 191× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed/news', 'service': 'feedsvc', 'calls': 33, 'c4xx': 33, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 33, 'rejecting': True, 'usual_per_window': 0.17, 'baseline_days': 7, 'window': [1794897600, 1794898500], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/news'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794897600&to=1794898500&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=automation&from=1794897600&to=1794898500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794898950, 'age_seconds': 410, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.3, 'status': 'open', 'first_seen': 1794883494, 'last_seen': 1794899351, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 23 of 23 calls (100 %; other 4xx×23) in the last 15 min — it usually has none. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 23, c4xx 23, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/news', 'level': 'medium', 'told_level': None, 'episode_at': 1794899351, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 15866, 'stale_s': 9}
  • 2026-11-17 08:04:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17497, 'fingerprint': 'a53207ef156d5030e292', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'feedsvc', 'title': 'feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2', 'facts': {'endpoint': 'GET /studies/:id/feed/news', 'service': 'feedsvc', 'calls': 30, 'c4xx': 30, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 30, 'rejecting': True, 'usual_per_window': 0.23, 'baseline_days': 7, 'window': [1794899400, 1794900300], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /studies/:id/feed/news'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794899400&to=1794900300&service=feedsvc&status=4xx'}, {'label': 'feedsvc service page', 'href': '/service/feedsvc?cluster=automation&from=1794899400&to=1794900300'}], 'provenance': {'source': 'rollup', 'collected_at': 1794900779, 'age_seconds': 571, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 31.3, 'status': 'resolved', 'first_seen': 1794883494, 'last_seen': 1794901340, 'resolved_at': 1794902649, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 6, 'narrative': '4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 30, c4xx 30, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /studies/:id/feed/news', 'level': 'medium', 'told_level': None, 'episode_at': 1794899351, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 19164, 'stale_s': 1318}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794900779
age_seconds571
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_4xx_spike