Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 20:19:07.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 51detectors over stored rollups and live sources

What X-ray is prepared to defend about production right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
production
just production · nothing ticked means production, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 7 warning · 8 info detectors
warning Background job failing production last seen 1d ago · since 1d ago · 8×

execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?

The background job execsvc.auto-query-reeval-sweep on production failed 11 runs recently (425 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Neo4j load anomaly production Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 26d ago · 260×

virtual-assistant Neo4j load 413 q/min vs a normal 4 q/min ± 4 q/min for Tue 10:00 UTC (3 of the last 4 checks)

virtual-assistant's Neo4j load is 413 q/min on production, above a normal of 4 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 95.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Service outlier production Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 26d ago · 1186×

virtual-assistant: Neo4j queries per request 83.8 — 14.0× the fleet's usual 6.0

virtual-assistant's Neo4j queries per request is 83.8 against the 7 other services' median of 6.0 (11.9 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

warning Autoscaler blind production last seen 1d ago · since 49d ago · 539×

ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h

Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s.

info Neo4j load anomaly production Owner: Vishal Karedla (backup: Taylor Futral) last seen 1d ago · since 71d ago · 95×

dashrsvc Neo4j load 61 q/min vs a normal 0 q/min ± 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)

dashrsvc's Neo4j load is 61 q/min on production, above a normal of 0 q/min ± 0 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 205.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Background job failing production last seen 1d ago · since 72d ago · 572×

pubsub.dashboards-production-dashsub: 14 failed runs in the last 6 h — message type retired, no longer ingested

The background job pubsub.dashboards-production-dashsub on production failed 14 runs recently (123 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Service outlier production Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 84d ago · 1838×

execsvc: Neo4j queries per request 93.6 — 15.7× the fleet's usual 6.0

execsvc's Neo4j queries per request is 93.6 against the 7 other services' median of 6.0 (13.4 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Service outlier production Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 84d ago · 2245×

docproc: Neo4j queries per request 47.1 — 7.9× the fleet's usual 6.0

docproc's Neo4j queries per request is 47.1 against the 7 other services' median of 6.0 (6.3 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.