Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 09:57:43.
› →|
updated 23h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 68detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
warning Background job failing test27 last seen 23h ago · since 16d ago · 22×

pubsub.onboarding-test27-regsvc-sub: 19 failed runs in the last 6 h — failed to create triggers: failed to fetch existing triggers: failed to fetch triggers: Error from A

The background job pubsub.onboarding-test27-regsvc-sub on test27 failed 19 runs recently (528 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning Background job failing test27 last seen 23h ago · since 21d ago · 34×

studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^

The background job studioai.unknown on test27 failed 5 runs recently (532 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Service outlier test27 Owner: Prasad Limaye (backup: Harish Kumar) last seen 23h ago · since 25d ago · 895×

docsvc: Neo4j queries per request 107.5 — 18.6× the fleet's usual 5.8 (also p95 latency (ms) 2.77 s)

docsvc differs from the other 7 services on 2 measures over six hours: Neo4j queries per request 107.5 against the peers' 5.8 (13.1 MADs above); p95 latency (ms) 2.77 s against the peers' 451 ms (3.6 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Service outlier test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 23h ago · since 25d ago · 687×

docproc: Neo4j queries per request 53.3 — 9.2× the fleet's usual 5.8

docproc's Neo4j queries per request is 53.3 against the 7 other services' median of 5.8 (6.1 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Neo4j load anomaly test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 23h ago · since 70d ago · 344×

studiosvc Neo4j load 4,216 q/min vs a normal 0 q/min ± 20 q/min for Tue 10:00 UTC (4 of the last 4 checks)

studiosvc's Neo4j load is 4,216 q/min on test27, above a normal of 0 q/min ± 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 210.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Service outlier test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 23h ago · since 83d ago · 2744×

studiosvc: Neo4j queries per request 161.0 — 27.8× the fleet's usual 5.8 (also p95 latency (ms) 9.24 s; p99 latency (ms) 18.30 s)

studiosvc differs from the other 7 services on 3 measures over six hours: Neo4j queries per request 161.0 against the peers' 5.8 (19.9 MADs above); p95 latency (ms) 9.24 s against the peers' 451 ms (13.7 MADs above); p99 latency (ms) 18.30 s against the peers' 1.01 s (12.1 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info N+1 fan-out test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 23h ago · since 83d ago · 329×

studiosvc runs 270 foreground Neo4j queries per request (180,895 queries / 670 calls, last hour)

studiosvc on test27 averaged 270.0 foreground Neo4j queries per API request over the last hour (180,895 queries for 670 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

info Neo4j load anomaly test27 last seen 23h ago · since 84d ago · 1794×

test27 (whole cluster) Neo4j load 4,441 q/min vs a normal 239 q/min ± 184 q/min for Tue 10:00 UTC (4 of the last 4 checks)

the whole test27 cluster's Neo4j load is 4,441 q/min on test27, above a normal of 239 q/min ± 184 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 22.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.