Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:14:07.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 98detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
critical Error-rate anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 1d ago · 1×

execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)

execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.

First look low confidence
Cause not clear from logs and recent changes; POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls, putting execsvc 5xx at 89.19% against a normal 0.00%, since 2026-11-17 04:05 UTC.
All 700 error lines are one failure: "could not match participant question to activity graph" (execution/activity_details_v2.go:2007), sample questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with empty refId, variableName and title.
Checked: the 5 execsvc deploys in the prior 24 h were config-only autosyncs (last d8d21151cd28 at 01:33 UTC, 151 min before onset) and brought no commits, so nothing links them to the failure.
The empty refId/variableName/title points at study or activity-graph data rather than code; a smaller episode on 2026-11-09 (finding 16947) hit the same endpoint and was resolved.
Next step: open the trace, find the study whose question 5571182f-... is missing from its activity graph, and check what changed in that study's design around 04:05 UTC. · its evidence →

execsvc: a new error shape 8 targets worst: critical 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor and 2 more
critical Error never seen before master resolved last seen · since · 2×

execsvc: a new error shape, 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to)

execsvc logged “could not match participant questionnaire/question/table/doc-upload to activity graph” 787 times in the last hour and never before in 90 days on master (first seen 2026-11-17 04:05 UTC, raised at execution/activity_details_v2.go:2007). Example trace: b4f239ac36290842023cca51c51d0e33.

First look low confidence
Cause not clear from logs and recent changes; GET activity status in execsvc returned 500 about 700 times between 04:05 and 04:16 UTC on 2026-11-17, logged as 'could not match participant question to activity graph'.
One question appears in the samples, questionId="5571182f-6c21-491e-b8c6-51fce3166e1e", with empty refId, variableName and title, so it may be one participant record polled repeatedly.
The handler error (handlers/activity_details_v2.go:457) has existed since 2026-11-04. Only the log line at execution/activity_details_v2.go:2007 is new.
The 5 deploys in the prior 24 h were autosync config-only with no image commits. The latest, d8d21151cd28, landed 151 min before onset, and nothing links any of them to this error.
Next step: look up question 5571182f-6c21-491e-b8c6-51fce3166e1e and its activity graph, and check why it has no refId, variableName or title. Start from the example trace.
This is also driving the error budget burn on execsvc (18x over 1 h). · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at tildadb/util.go:1004). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors, all at 16:49 UTC: 5× 'failed to run cypher in transaction' (tildadb/util.go:1004), plus 14× util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12+5× document copy (doclib/copyDocument.go:74) and 5× scope binders (doclib/context.go:335).
The errors have not recurred since then (last seen 16:49 per error lines).
The 5 deploys in the prior 24 h were config-only autosync; the closest was 15:39 (69 min before), with no image or commit change and nothing tying it to these files.
The util.go:993 deadlock shape has existed since 2026-11-15, so deadlocks are not new; what is new is the document-copy write path hitting them.
Next step: open the example trace and check which concurrent document copies (bulk doc write vs. context scope binders) took ExclusiveLocks on the same nodes, then confirm the copy path retries on TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)

execsvc logged “failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[tran” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/context.go:335). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors at 16:49 UTC while copying documents.
This finding is 5× 'failed to add document to scope binders' (doclib/context.go:335). Same burst: 14× tildadb/util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12× + 5× copy failures (doclib/copyDocument.go:74).
All lines are from 16:49 UTC only. The util.go:993 deadlock shape was already seen on 2026-11-15, so deadlocks are not new.
Checked: the 5 deploys in the prior 24 h were config-only autosyncs (latest 15:39, 69 min before) with no new commits. Nothing links them to this burst.
Next step: open the example trace to see which concurrent document copies collided on the same nodes. Check whether the copy path retries Neo4j TransientErrors. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could)

execsvc logged “failed to create document copy — could not add document context: could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetec” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC (5x here at doclib/copyDocument.go:74 via context.go:335 scope binders).
The same burst produced 12x bulk-write deadlocks (createDocument.go:1015, copyDocument.go:74), 14x at tildadb/util.go:993 and 5x at util.go:1004, opening 4 sibling findings.
The deadlock shape at util.go:993 was already seen on 2026-11-15 07:38, so deadlocks predate this burst; the new part is which callers hit them.
Checked: the 5 deploys in the prior 24 h were autosync config-only changes (the latest at 15:39, 69 min before onset), with no image or commit change and nothing tying them to these writes.
Next step: open the example trace to see which concurrent copy transactions took the locks on the scope binders.
Also check whether copyDocument retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to bulk-write document nodes — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/createDocument.go:1015). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors in one minute, 16:49 UTC on 2026-11-16.
Counts: 12x 'failed to bulk-write document nodes' (doclib/createDocument.go:1015), 12x copy failures (doclib/copyDocument.go:74), 14x tildadb/util.go:993, 5x scope binders (doclib/context.go:335).
All come from document-copy writes contending for the same lock; findings 17467/17469/17470/17471 are the same burst.
Checked: 5 autosync config-only deploys in the prior 24 h, the last at 15:39 UTC (70 min before), with no commits and nothing tying them to these writes. The util.go:993 deadlock shape was already seen on 2026-11-15 07:38.
Next step: open the example trace to see which concurrent copy requests collided, then check whether createDocument.go:1015 retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)

execsvc logged “failed to create document copy — could not write document node and core relationships: bulk doc write: Neo4jError: Neo.TransientError.Transaction.DeadlockDetect” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC, last seen 16:55 UTC.
Counts: 12× copyDocument.go:74 (bulk doc write), 5× copyDocument.go:74 (context scope binders), 14× tildadb/util.go:993.
The util.go:993 deadlock shape has existed since 2026-11-15 07:38, so lock contention is not new; only the document-copy paths are.
Checked: 5 autosync deploys in the prior 24 h were config-only (no image or commit change, the last one 69 min before onset), with nothing linking them to the error. This finding has no earlier history.
Next step: open the example trace and find which concurrent transaction held the ExclusiveLock during the 16:49 copy burst.
Also check whether doclib retries Neo.TransientError, since this error class is retryable. · its evidence →

warning Error never seen before master last seen · since · 3×

execsvc: a new error shape, 44× in the last hour, never seen in 90 days (pubsub not initialized, dropping onboarding event)

execsvc logged “pubsub not initialized, dropping onboarding event” 44 times in the last hour and never before in 90 days on master (first seen 2026-11-16 07:26 UTC, raised at health/util.go:145). Example trace: df85a2172dcd3a30d3260b366d781790.

warning Error never seen before master resolved last seen · since · 2×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)

execsvc logged “failed to evaluate and raise auto query for participant question — evalResult is not a bool” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-15 07:38 UTC, raised at execution/activityV2.go:901). Example trace: fa1e73c1fad65714b4c0802b8fef431e.

critical Pod cannot start master resolved last seen 1d ago · since 10d ago · 176×

x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod

Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794883816.

First look low confidence
Cause not clear from logs and recent changes; CronJob pod x-ray-weaviate-stats-29914690-fpppt (x-ray) has been in ImagePullBackOff for 34 min (49 events) since 2026-11-17 02:23 UTC.
The same failure has recurred every night since 2026-11-08, opening around 02:23–02:28 UTC and clearing by about 05:00–05:11, with 166 occurrences in total.
An earlier first look recorded the failing image as us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5e023f3.
None of the 12 deploys in the prior 24 h touched x-ray, and none of the cluster error lines mention this pull.
The feedsvc 403s at the same time look unrelated.
Next step: read the pull error in the Kubernetes events (not found or unauthorized), then check that this tag exists in the registry and the CronJob's pull access. · its evidence →

critical Endpoint failing master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 33d ago · 11×

execsvc POST /api/trialexec/v2/study/:id/activity/status: 473 of 473 calls failed (100 %) in the last 15 min

execsvc's POST /api/trialexec/v2/study/:id/activity/status on master failed 473 of 473 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

First look low confidence
Cause not clear from logs and recent changes; execsvc POST /api/trialexec/v2/study/:id/activity/status failed 655 of 655 calls (100 %, all 5xx) in 15 min. Onset was 04:00 UTC; errors start at 04:05 UTC.
Each failure is a 500 from activity_details_v2.go:458: 'could not match participant question to activity graph', with an empty refId, variableName and title (sample questionId 5571182f-6c21-491e-b8c6-51fce3166e1e). Sampled failures are all org smotest1.
The 5 execsvc deploys in the prior 24 h were config-only autosyncs with no image change. The last was at 01:33 UTC, 146 min before onset. No commit links to the failure.
The same error has recurred since 2026-10-16 across 5 earlier episodes. A new shape from execution/activity_details_v2.go:2007 appeared at 04:05 UTC.
Next step: check the smotest1 study data for question 5571182f-6c21-491e-b8c6-51fce3166e1e and why it has no match in the activity graph, starting at execution/activity_details_v2.go:2007. · its evidence →

critical Error-rate anomaly master resolved last seen 1d ago · since 55d ago · 9×

master (whole cluster) server-error rate 57.64% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)

the whole master cluster's error rate is 57.6% on master, above a normal of 0.0% ± 1.0% for this hour of the week (3 of the last 4 five-minute buckets, z = 57.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

First look low confidence
Cause not clear from logs and recent changes; execsvc POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls (cluster 5xx 57.64% vs normal 0.00%) since 2026-11-17 04:05 UTC.
All 700 error lines in the window say "could not match participant question to activity graph" (handlers/activity_details_v2.go:457, execution/activity_details_v2.go:2007), for questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with an empty refId, variableName and title.
The only execsvc deploy was an autosync config change d8d21151cd28 at 01:33 UTC, 151 min before onset, with no new image; nothing ties it to this error.
This message was already seen on 2026-10-16 and 2026-11-04, so bad study data is more likely than new code.
Next step: check which study and activity graph reference question 5571182f-6c21-491e-b8c6-51fce3166e1e, and why the graph has no matching refId or variableName. · its evidence →

info Service outlier master Owner: Prasad Limaye (backup: Harish Kumar) last seen 1d ago · since 84d ago · 2941×

docsvc: Neo4j queries per request 5.1 — 2.1× the fleet's usual 2.4

docsvc's Neo4j queries per request is 5.1 against the 6 other services' median of 2.4 (3.7 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Latency anomaly master resolved last seen 1d ago · since 70d ago · 679×

master (whole cluster) p95 2.0 min vs a normal 368 ms ± 473 ms for Tue 03:00 UTC (3 of the last 4 checks)

the whole master cluster's p95 is 120.0 s on master, above a normal of 368 ms ± 473 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 252.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Service outlier master Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 84d ago · 957×

execsvc: p99 latency (ms) 7.08 s — 12.9× the fleet's usual 550 ms (also p95 latency (ms) 4.01 s)

execsvc differs from the other 5 services on 2 measures over six hours: p99 latency (ms) 7.08 s against the peers' 550 ms (56.1 MADs above); p95 latency (ms) 4.01 s against the peers' 468 ms (7.8 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Traffic anomaly master resolved last seen 1d ago · since 85d ago · 637×

master (whole cluster) request rate 7.6 req/min vs a normal 0.1 req/min ± 0.1 req/min for Tue 02:00 UTC (3 of the last 4 checks)

the whole master cluster's request rate is 7.6 req/min on master, above a normal of 0.1 req/min ± 0.1 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 50.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.