Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 04:10:43.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 118detectors over stored rollups and live sources

What X-ray is prepared to defend about test26 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test26
just test26 · nothing ticked means test26, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 10 warning · 8 info detectors
warning Error never seen before test26 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 3d ago · since 3d ago · 3×

virtual-assistant: a new error shape, 12× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e)

virtual-assistant logged “Token refresh failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. ” 12 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 00:00 UTC, raised at api/token_manager.py:155). Example trace: 5379da8caeb98fbc98bc36a9dc219e8f.

warning Error never seen before test26 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 3d ago · since 3d ago · 3×

virtual-assistant: a new error shape, 25× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)

virtual-assistant logged “HTTP request failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. W” 25 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 00:00 UTC, raised at api/token_manager.py:87). Example trace: 5379da8caeb98fbc98bc36a9dc219e8f.

info Service outlier test26 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 3d ago · since 76d ago · 316×

chat-agent: p99 latency (ms) 15.00 s — 76.9× the fleet's usual 195 ms (also p95 latency (ms) 8.12 s)

chat-agent differs from the other 5 services on 2 measures over six hours: p99 latency (ms) 15.00 s against the peers' 195 ms (87.5 MADs above); p95 latency (ms) 8.12 s against the peers' 175 ms (61.7 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Service outlier test26 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 3d ago · since 84d ago · 1051×

studiosvc: p99 latency (ms) 24.50 s — 12.6× the fleet's usual 1.94 s

studiosvc's p99 latency (ms) is 24.50 s against the 7 other services' median of 1.94 s (8.7 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

warning Query shape degrading test26 Owner: Vishal Karedla (backup: Taylor Futral) resolved last seen 3d ago · since 34d ago · 8×

dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)

A dashrsvc query shape on test26 has p95 1.7 s in the second half of the window against 19 ms in the first (89.8×, 272 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Error never seen before test26 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 4d ago · since 4d ago · 3×

studiosvc: a new error shape, ≥4357× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)

studiosvc logged “error on pubsub receive — rpc error: code = NotFound desc = Resource not found (resource=study-site-binding-test26-studiosvcsub).” 4,357 times in the last hour and never before in 90 days on test26 (first seen 2026-11-14 14:23 UTC, raised at pubsublib/googlepubsub.go:554). Example trace: none carried.