Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-23 05:59:32.
› →|
updated 4d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 18detectors over stored rollups and live sources

What X-ray is prepared to defend about delta right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
delta
just delta · nothing ticked means delta, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 1 critical · 10 warning · 7 info detectors
critical Error never seen before delta Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 4d ago · since 4d ago · 3×

execsvc: a new error shape, 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2)

execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Example trace: 9bff744b81af07042b1e3e7de4963df4.

First look low confidence
Cause not clear from logs and recent changes; execsvc returned 500 "could not save Careplan activity V2" 9× between 09:46 and 09:54 UTC on 2026-11-17 (new shape, logged at observability/middleware.go:281).
Checked: no execsvc deploys in the 24 h before onset (running build deployed 2026-11-16 05:11 UTC by [email protected]), no other findings at the same time.
Alongside it: "no result from branch evaluation" (branchV2.go:77, 2×), "Question object not found for question" (questionV2.go:1863, 3×, new at 09:53), and pubsub "transaction was never committed" (2×).
The code shown (careplan.go:1044) is the 403 branch; the 500 comes from a later error path after execution.SaveActivityV2, whose underlying error is not in the bundle.
Next step: open the example trace to read the wrapped error from SaveActivityV2. Then check whether it matches the branch-evaluation or missing-question errors for that protocol. · its evidence →

warning Pod keeps restarting delta last seen 4d ago · since 14d ago · 9×

neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2

neo4j-differential-backup-delta in neo4j-system on delta has restarted in place 4 times since 2026-11-17 10:15 UTC (4 in the last 30 minutes, 4 in 24 hours), most recently at 2026-11-17 10:25 UTC. Kubernetes' account of it: 104 sched events, 3 crash events. Last event: BackOff — “Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe31”. It is running image 5.26.28-diff-backup-2. Containers restarting: neo4j-differential-backup ×4. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

warning Error budget burning delta Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 4d ago · since 15d ago · 16×

tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective

tmfsync is spending its error budget 8.4× faster than the objective allows over the last hour (0.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 26 bad events against 6,583 calls in the window.