Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 03:10:46.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 68detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
info Neo4j load anomaly test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 26d ago · 261×

docproc Neo4j load 87 q/min vs a normal 11 q/min ± 1 q/min for Mon 19:00 UTC (3 of the last 4 checks)

docproc's Neo4j load is 87 q/min on test27, above a normal of 11 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 2d ago · since 85d ago · 2174×

workplansvc Neo4j load 849 q/min vs a normal 111 q/min ± 41 q/min for Mon 19:00 UTC (3 of the last 4 checks)

workplansvc's Neo4j load is 849 q/min on test27, above a normal of 111 q/min ± 41 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 17.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Error never seen before test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 2d ago · 4×

chat-agent: a new error shape, 5× in the last hour, never seen in 90 days (status_tool: body raised, surfaced to the LLM as a tool error)

chat-agent logged “status_tool: body raised, surfaced to the LLM as a tool error” 5 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 17:58 UTC, raised at hitl/decorator.py:630). Example trace: 382e740286d47f73922200a7d6ba4554.

warning 4xx spike (403/401/404…) test27 Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 2d ago · since 6d ago · 3×

execsvc POST /api/trialexec/v2/study/:id/activity/details: permission denied — 276 of 622 calls (44 %; 403×276) in the last 15 min — 522× its usual 0.5

4xx spike (403/401/404…): execsvc POST /api/trialexec/v2/study/:id/activity/details: permission denied — 276 of 622 calls (44 %; 403×276) in the last 15 min — 522× its usual 0.5. Facts: endpoint POST /api/trialexec/v2/study/:id/activity/details, service execsvc, calls 622, c4xx 276, share_pct 44.37299035369775, c401 0.

info Service outlier test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 2d ago · since 84d ago · 3064×

virtual-assistant: Neo4j queries per request 47.5 — 22.2× the fleet's usual 2.1 (also p95 latency (ms) 5.58 s)

virtual-assistant differs from the other 9 services on 2 measures over six hours: Neo4j queries per request 47.5 against the peers' 2.1 (18.5 MADs above); p95 latency (ms) 5.58 s against the peers' 518 ms (7.0 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

onboardingsvc 5 targets worst: warning GET /api/organizations/myCro: permission denied — 85 of 85 calls (100 %; 403×85) in the last 15 min — it usually has none, GET /api/myprofile/visibility/v2: permission denied — 114 of 125 calls (91 %; 403×114) in the last 15 min — 8× its usual 15.1, POST /api/sites: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 118× its usual 0.3, GET /api/organizations/tildainc: permission denied — 39 of 43 calls (91 %; 403×39) in the last 15 min — 69× its usual 0.6, GET /api/organizations/teammatesDemoSandbox: permission denied — 38 of 38 calls (100 %; 403×38) in the last 15 min — it usually has none
warning 4xx spike (403/401/404…) test27 resolved last seen · since · 3×

onboardingsvc GET /api/organizations/myCro: permission denied — 85 of 85 calls (100 %; 403×85) in the last 15 min — it usually has none

4xx spike (403/401/404…): onboardingsvc GET /api/organizations/myCro: permission denied — 85 of 85 calls (100 %; 403×85) in the last 15 min — it usually has none. Facts: endpoint GET /api/organizations/myCro, service onboardingsvc, calls 85, c4xx 85, share_pct 100.0, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen · since · 36×

onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 114 of 125 calls (91 %; 403×114) in the last 15 min — 8× its usual 15.1

4xx spike (403/401/404…): onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 114 of 125 calls (91 %; 403×114) in the last 15 min — 8× its usual 15.1. Facts: endpoint GET /api/myprofile/visibility/v2, service onboardingsvc, calls 125, c4xx 114, share_pct 91.2, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen · since · 8×

onboardingsvc POST /api/sites: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 118× its usual 0.3

4xx spike (403/401/404…): onboardingsvc POST /api/sites: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 118× its usual 0.3. Facts: endpoint POST /api/sites, service onboardingsvc, calls 31, c4xx 31, share_pct 100.0, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen · since · 26×

onboardingsvc GET /api/organizations/tildainc: permission denied — 39 of 43 calls (91 %; 403×39) in the last 15 min — 69× its usual 0.6

4xx spike (403/401/404…): onboardingsvc GET /api/organizations/tildainc: permission denied — 39 of 43 calls (91 %; 403×39) in the last 15 min — 69× its usual 0.6. Facts: endpoint GET /api/organizations/tildainc, service onboardingsvc, calls 43, c4xx 39, share_pct 90.69767441860465, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen · since · 1×

onboardingsvc GET /api/organizations/teammatesDemoSandbox: permission denied — 38 of 38 calls (100 %; 403×38) in the last 15 min — it usually has none

4xx spike (403/401/404…): onboardingsvc GET /api/organizations/teammatesDemoSandbox: permission denied — 38 of 38 calls (100 %; 403×38) in the last 15 min — it usually has none. Facts: endpoint GET /api/organizations/teammatesDemoSandbox, service onboardingsvc, calls 38, c4xx 38, share_pct 100.0, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen 2d ago · since 3d ago · 3×

feedsvc GET /studies/:id/feed/week: 4xx — 54 of 54 calls (100 %; other 4xx×54) in the last 15 min — 267× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: 4xx — 54 of 54 calls (100 %; other 4xx×54) in the last 15 min — 267× its usual 0.2. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 54, c4xx 54, share_pct 100.0, c401 0.

warning 4xx spike (403/401/404…) test27 resolved last seen 2d ago · since 3d ago · 3×

feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 62 of 62 calls (100 %; 403×61, other 4xx×1) in the last 15 min — 289× its usual 0.2. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 62, c4xx 62, share_pct 100.0, c401 0.

info Neo4j load anomaly test27 Owner: Satish Ashok (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 69d ago · 100×

regsvc Neo4j load 31 q/min vs a normal 1 q/min ± 1 q/min for Mon 05:00 UTC (3 of the last 4 checks)

regsvc's Neo4j load is 31 q/min on test27, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 50.2, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

virtual-assistant: a new error shape 3 targets worst: warning 26× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e, 27× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at, 53× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er
warning Error never seen before test27 last seen · since · 3×

virtual-assistant: a new error shape, 26× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e)

virtual-assistant logged “Token refresh failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. ” 26 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 04:26 UTC, raised at api/token_manager.py:155). Example trace: 35b8e6f94a1746fd82c6b2ccc79fce59.

warning Error never seen before test27 last seen · since · 3×

virtual-assistant: a new error shape, 27× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at)

virtual-assistant logged “Error fetching document info: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login at” 27 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 04:26 UTC, raised at document_subscriber/document_events.py:165). Example trace: 35b8e6f94a1746fd82c6b2ccc79fce59.

warning Error never seen before test27 last seen · since · 3×

virtual-assistant: a new error shape, 53× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)

virtual-assistant logged “HTTP request failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. W” 53 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 04:26 UTC, raised at api/token_manager.py:87). Example trace: 35b8e6f94a1746fd82c6b2ccc79fce59.