Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 08:50:14.
› →|
updated 22h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 6

stored age unknown all 6detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
warning Error budget burning master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 22h ago · since 83d ago · 233×

execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective

execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.

warning Service gone silent master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 76d ago · 108×

docsvc on master has served nothing for 1.7 h; at this hour it normally serves about 144 calls/hour

docsvc on master has served no API calls for 100 minutes (last call 2026-11-16 17:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 144 calls an hour, 72 in the window checked. It carried 10.1% of the cluster's calls over the last week. A deploy landed just before the silence began — see the facts. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.

warning Error budget burning master Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 10d ago · 63×

docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective

docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.

critical Pod keeps restarting master resolved last seen 2d ago · since 2d ago · 5×

workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926

workplan-agent in tilda on master has restarted in place 13 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 13 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 47 sched events, 12 crash events, 4 probe events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)”. It is running image suresh-eng8690-8981379926. Containers restarting: workplan-agent ×7, istio-proxy ×6. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

First look low confidence
Cause not clear from logs and recent changes; workplan-agent (ns tilda) restarted 14 times in 15 min since 13:40 UTC: 47 unschedulable, 12 crash and 4 probe events, both istio-proxy and workplan-agent containers (last event: BackOff on istio-proxy).
The same image tag suresh-eng8690-8981379926 is failing at the same time on workplansvc (crash loop) and virtual-assistant (unschedulable).
workplansvc logs show a new error since 13:46: otel resource detection cannot reach the metadata server 169.254.169.254 (main.go:64, observability.go:133).
A node went NotReady and cilium-agent logged a foreign-IP error at 13:17, but nothing links either to this pod.
No deploy of these three services shows in the 24 h before. The argocd collector was stale for 224 min, so a rollout of this branch-named image may be missing.
Next step: run kubectl describe on pod workplan-agent-7f4d75b46c-8w7x9 to get the unschedulable reason, and check who rolled out tag suresh-eng8690-8981379926 to master. · its evidence →

critical Pod keeps restarting master resolved last seen 2d ago · since 2d ago · 5×

virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926

virtual-assistant in tilda on master has restarted in place 17 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 17 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 27 sched events, 16 crash events, 2 probe events. Last event: FailedScheduling — “0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no ”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, clamd ×7, virtual-assistant ×2. The workload carried 5.6% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

First look low confidence
Cause not clear from logs and recent changes; virtual-assistant (ns tilda, master) restarted 15 times in 15 min from 13:40 UTC. Dominant reason is unschedulable (sched 26, crash 16, probe 2).
Last FailedScheduling: 0/7 nodes available: 5 Insufficient memory, 3 Insufficient cpu, 1 untolerated taint, 1 affinity mismatch. Preemption did not help.
The same branch image suresh-eng8690-8981379926 hit workplan-agent (unschedulable) and workplansvc (crash loop, otel errors reaching 169.254.169.254) at the same time.
None of the 12 deploys checked is this image. The argocd collector was 224 min stale, so the rollout may simply be missing from the bundle.
Next step: find who rolled suresh-eng8690-8981379926 into tilda on master. Compare its CPU/memory requests with free node capacity, then roll back to the previous image or add node capacity. · its evidence →

critical Pod keeps restarting master resolved last seen 2d ago · since 2d ago · 4×

workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926

workplansvc in tilda on master has restarted in place 15 times since 2026-11-15 13:40 UTC (8 in the last 30 minutes, 15 in 24 hours), most recently at 2026-11-15 13:50 UTC. Kubernetes' account of it: 27 crash events, 17 sched events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, workplansvc ×7. The workload carried 42.9% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

First look medium confidence
workplansvc (ns tilda) on master is crash-looping: 14 restarts in 15 min on 1 pod (workplansvc 7, istio-proxy 7), image suresh-eng8690-8981379926.
Started 13:40 UTC. From 13:46 workplansvc/main.go:64 exits on 'observability init failed': otel resource detection cannot reach the GCE metadata server (dial tcp 169.254.169.254, from observability/observability.go:133), 6x each, never seen in 90 days.
The istio-proxy sidecar is also in BackOff. A node went NotReady and cilium logged foreign-IP errors at 13:17, so pod networking to the metadata server is the likely break.
No workplansvc deploy is in the 24 h list. The same image runs on workplan-agent and virtual-assistant, which are also restarting (unschedulable).
Next: check istio-proxy logs and egress to 169.254.169.254 from the pod. Make tracing init non-fatal at main.go:64 so the service starts without otel.
Suspect: GCE metadata server 169.254.169.254 unreachable · its evidence →