Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:55:39.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 69detectors over stored rollups and live sources

What X-ray is prepared to defend about delta right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
delta
just delta · nothing ticked means delta, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 1 critical · 10 warning · 7 info detectors
execsvc: a new error shape 7 targets worst: critical 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2, 55× in the last hour, never seen in 90 days (evalResult is not a bool, 150× in the last hour, never seen in 90 days (no result from branch evaluation, 6× in the last hour, never seen in 90 days (message was never published because transaction was never committed, 46× in the last hour, never seen in 90 days (evalResult is not a bool, 46× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva and 1 more
critical Error never seen before delta last seen · since · 3×

execsvc: a new error shape, 9× in the last hour, never seen in 90 days (Request completed — ? · could not save Careplan activity V2)

execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Example trace: 9bff744b81af07042b1e3e7de4963df4.

First look low confidence
Cause not clear from logs and recent changes; execsvc returned 500 "could not save Careplan activity V2" 9× between 09:46 and 09:54 UTC on 2026-11-17 (new shape, logged at observability/middleware.go:281).
Checked: no execsvc deploys in the 24 h before onset (running build deployed 2026-11-16 05:11 UTC by [email protected]), no other findings at the same time.
Alongside it: "no result from branch evaluation" (branchV2.go:77, 2×), "Question object not found for question" (questionV2.go:1863, 3×, new at 09:53), and pubsub "transaction was never committed" (2×).
The code shown (careplan.go:1044) is the 403 branch; the 500 comes from a later error path after execution.SaveActivityV2, whose underlying error is not in the bundle.
Next step: open the example trace to read the wrapped error from SaveActivityV2. Then check whether it matches the branch-evaluation or missing-question errors for that protocol. · its evidence →

critical Error never seen before delta resolved last seen · since · 2×

execsvc: a new error shape, 55× in the last hour, never seen in 90 days (evalResult is not a bool)

execsvc logged "evalResult is not a bool" 55 times within a one-hour window ending at execlib/branchV2.go:96, with no prior occurrences in the preceding 90 days (first seen 2026-11-16 18:51 UTC). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.

First look low confidence
Cause not clear from logs and recent changes; execsvc logged 'evalResult is not a bool' 55× (18:51–19:21 UTC) at execlib/branchV2.go:96, alongside 'no result from branch evaluation' 150× at branchV2.go:91 since 18:42 (finding 17479).
The same two errors already ran from 14:20 UTC at branchV2.go:77/82, last at 18:31, via the participant auto-query path (activityV2.go:882).
The line numbers moved by 14 between 18:31 and 18:42, which suggests a new execsvc build started then, but no image deploy is recorded.
The only recorded deploy is config-only (6db9855a7d34, [email protected], 05:11 UTC, 820 min before), and nothing links it to these errors.
The error strings were not found in the default-branch code, so the running code is unknown.
Next step: confirm which execsvc build has run since about 18:40, then read execlib/branchV2.go:91-96 in it to see why branch evaluation returns no or non-bool results. · its evidence →

critical Error never seen before delta resolved last seen · since · 3×

execsvc: a new error shape, 150× in the last hour, never seen in 90 days (no result from branch evaluation)

execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.

First look low confidence
Cause not clear from logs and recent changes; this looks like an existing error that moved, not a new one.
execsvc logged 'no result from branch evaluation' 101× at execlib/branchV2.go:91 from 18:42 to 19:13 UTC, plus 'evalResult is not a bool' 38× at branchV2.go:96 from 18:51 UTC.
The same two messages came from branchV2.go:77 (106×, seen since Nov 7) and :82 (4×) until 18:31 UTC, then stopped. The activityV2.go:882 wrapper lines also stopped at 18:31.
That 14-line shift suggests a new execsvc build rolled out around 18:31–18:42, but the only recorded deploy is [email protected] at 05:11 UTC, 810 min earlier. Code and commits could not be read because the GitHub budget ran out.
Next step: find what execsvc version started running around 18:31–18:42 UTC, diff branchV2.go, and check why branch evaluation returns no result (failing since Nov 7). · its evidence →

warning Error never seen before delta resolved last seen · since · 2×

execsvc: a new error shape, 6× in the last hour, never seen in 90 days (message was never published because transaction was never committed)

execsvc logged “message was never published because transaction was never committed” 6 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:28 UTC, raised at pubsublib/pubsub.go:131). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.

warning Error never seen before delta resolved last seen · since · 3×

execsvc: a new error shape, 46× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)

execsvc logged “failed to evaluate and raise auto query for participant question — evalResult is not a bool” 46 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:20 UTC, raised at execution/activityV2.go:882). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.

critical Error never seen before delta resolved last seen · since · 3×

execsvc: a new error shape, 247× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )

execsvc logged “failed to evaluate and raise auto query for participant question — no result from branch evaluation for branch, protocolId=? activityRefId=?” 247 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 14:20 UTC, raised at execution/activityV2.go:882). Example trace: 7bce6fd5fb57a5f7ec3d2edce7a8fdb0.

First look low confidence
Cause not clear from logs and recent changes; execsvc logged 122 'failed to evaluate and raise auto query for participant question — no result from branch evaluation' errors (execution/activityV2.go:882) from 14:20 to 14:26 UTC on 2026-11-16.
Alongside it, 19 'evalResult is not a bool' errors (execlib/branchV2.go:82) also began at 14:20, plus 1 auto query re-evaluation failure at 14:23.
The inner error 'no result from branch evaluation' (execlib/branchV2.go:77) has existed since 2026-11-07. Only the auto-query wrapper at activityV2.go:882 is new.
The only execsvc deploy was a config change (6db9855a7d34) by [email protected] at 05:11 UTC. That is 548 min before onset, and nothing links it to these errors. Code and commits could not be read because the GitHub budget ran out.
Next step: open the example trace and check the branch and auto-query definitions for protocolId 77bcfccb-cc59-4aea-bff8-0991eaa0c788. Look for a protocol edit published around 14:20 that makes branches return no result or a non-bool. · its evidence →

warning Pod keeps restarting delta last seen 1d ago · since 10d ago · 9×

neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2

neo4j-differential-backup-delta in neo4j-system on delta has restarted in place 4 times since 2026-11-17 10:15 UTC (4 in the last 30 minutes, 4 in 24 hours), most recently at 2026-11-17 10:25 UTC. Kubernetes' account of it: 104 sched events, 3 crash events. Last event: BackOff — “Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe31”. It is running image 5.26.28-diff-backup-2. Containers restarting: neo4j-differential-backup ×4. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

warning Error budget burning delta Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 12d ago · 16×

tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective

tmfsync is spending its error budget 8.4× faster than the objective allows over the last hour (0.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 26 bad events against 6,583 calls in the window.