Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 20:13:45.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 26detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
critical Pod cannot start master last seen 1d ago · since 1d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
Suspect: x-ray-services-master sync 556a243c8ac9 (autosync) · its evidence →

feedsvc GET /studies/:id/ 3 targets worst: critical feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2, feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call, feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
critical 4xx spike (403/401/404…) master last seen · since · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 174, c4xx 166, share_pct 95.40229885057471, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/news returned 403 on 44 of 47 calls (94 %) in 15 min, against a usual 0.01 per window. Onset was 2026-11-17 05:20 UTC, and the first failing request in the log is at 05:27. Every sampled 403 comes from one org, "Teammates Demo Sandbox", and returns fast (13–21 ms). That points to a permission check on that org or its users · its evidence →

critical 4xx spike (403/401/404…) master resolved last seen · since · 57×

feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see. · its evidence →

critical 4xx spike (403/401/404…) master resolved last seen · since · 59×

feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies. · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 26d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j database 'automation' (project tilda-trial-dev) had 37 failed Cypher queries in the last 15 min, 366 occurrences since 2026-10-23 19:21 UTC.
Checked: 12 autosync config deploys in the 24 h before onset (closest: execsvc 57 min and studiosvc 55 min before). None has an image tag, commit or error line tying it to Neo4j.
None of the 6 error shapes in the bundle comes from neo4j or names a Cypher query. execsvc 'no result from branch evaluation' first appeared 2026-11-13, weeks after onset.
No earlier episodes and no other findings opened at the same time.
Next step: open the Neo4j error logs for this window, read the Cypher error text, and find which client service sends the failing queries to 'automation'. · its evidence →

critical Background job failing master last seen 1d ago · since 26d ago · 545×

workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?

The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

First look low confidence
workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?.
Cause not clear from logs and recent changes; checked: error lines (none stored), deploys in the 24 h before (12), commits in the deploy's range (0), this finding's history, other findings at the same time (1). · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 14d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; Neo4j shared, database matrix_b: 35 Cypher queries failed in the last 15 min (213 occurrences since onset).
Failing since 2026-11-04 14:19 UTC, still seen 2026-11-17 10:21 UTC; matrix_a also started failing 17 min later (7 queries).
Checked: 12 deploys in the 24 h before onset (11 config syncs at revision c8a04003ea3d by [email protected] 205 min before, dashrsvc autosync 67 min before). Nothing links them to Neo4j.
None of the stored cluster error shapes are Neo4j or Cypher errors, and no single service owns the queries.
Next step: open the Neo4j error logs for the window to get the failing Cypher error codes and the client issuing them. · its evidence →

critical 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 15d ago · 25×

studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8

4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 36, share_pct 73.46938775510205, c401 36.

First look low confidence
Cause not clear from logs and recent changes; studiosvc GET /api/studio/events returned 401 on 21 of 49 calls (43 %) in 15 min, 28× its usual 0.8 per window.
Onset 15:25 UTC on 2026-11-16; the first 401 in the request log is at 15:28 UTC. This finding has recurred many times since 2026-11-03, and earlier episodes cleared on their own in about 20–30 min.
Checked: 0 stored error lines, 4 config-only autosync deploys (the last at 08:05 UTC, 7 h before) with no image or commit change, and the SSE handler. The handler returns 401 when authlib.GetRequestContext fails and logs a warning, but no such lines are stored.
Next step: open one of the 401 traces to see whether the auth context or Organization header is missing, and compare with the open sibling finding on /api/studio/triggers (100 % 401). · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 22d ago · 365×

Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min

Neo4j queries failing: Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min. Facts: env shared, database test26, failed 36, window_min 15, project tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j, database test26 (project tilda-trial-dev): 662 Cypher queries failed in 15 min, 234 occurrences, this episode since 2026-11-10 19:08 UTC (first seen 2026-10-27, last 2026-11-16 06:22).
Checked: 12 autosync config deploys (ash-nazg 991526a739f8 at 18:05, 670c5dce81a4 at 18:35) landed 33–63 min before onset. No error line, changed file or code ties them to Neo4j.
No stored error line comes from Neo4j. Top shapes are fluentbit, cilium and execsvc branch evaluation, all starting days later.
The same finding recurred 2026-10-27→11-10 with no cause found, so this looks chronic to test26 rather than tied to a deploy.
Next step: open the Neo4j error logs for test26 to see the actual Cypher error and which client sends the failing queries. · its evidence →

critical Error-rate anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 1d ago · 1×

execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)

execsvc shows an 89.2% error rate on master, significantly above the normal 0.0% ± 1.0% baseline for this weekly time slot. The anomaly spans 3 of the last 4 five-minute buckets (z = 89.2), based on 4 weeks of historical data. The failing endpoint is POST /api/trialexec/v2/study/:id/activity/status, which experienced 787 errors out of 787 total calls. User impact depends on endpoint traffic distribution.

First look low confidence
Cause not clear from logs and recent changes; POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls, putting execsvc 5xx at 89.19% against a normal 0.00%, since 2026-11-17 04:05 UTC.
All 700 error lines are one failure: "could not match participant question to activity graph" (execution/activity_details_v2.go:2007), sample questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with empty refId, variableName and title.
Checked: the 5 execsvc deploys in the prior 24 h were config-only autosyncs (last d8d21151cd28 at 01:33 UTC, 151 min before onset) and brought no commits, so nothing links them to the failure.
The empty refId/variableName/title points at study or activity-graph data rather than code; a smaller episode on 2026-11-09 (finding 16947) hit the same endpoint and was resolved.
Next step: open the trace, find the study whose question 5571182f-... is missing from its activity graph, and check what changed in that study's design around 04:05 UTC. · its evidence →

execsvc: a new error shape 6 targets worst: critical 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor
critical Error never seen before master resolved last seen · since · 2×

execsvc: a new error shape, 787× in the last hour, never seen in 90 days (could not match participant questionnaire/question/table/doc-upload to)

execsvc logged “could not match participant questionnaire/question/table/doc-upload to activity graph” 787 times in the last hour and never before in 90 days on master (first seen 2026-11-17 04:05 UTC, raised at execution/activity_details_v2.go:2007). Example trace: b4f239ac36290842023cca51c51d0e33.

First look low confidence
Cause not clear from logs and recent changes; GET activity status in execsvc returned 500 about 700 times between 04:05 and 04:16 UTC on 2026-11-17, logged as 'could not match participant question to activity graph'.
One question appears in the samples, questionId="5571182f-6c21-491e-b8c6-51fce3166e1e", with empty refId, variableName and title, so it may be one participant record polled repeatedly.
The handler error (handlers/activity_details_v2.go:457) has existed since 2026-11-04. Only the log line at execution/activity_details_v2.go:2007 is new.
The 5 deploys in the prior 24 h were autosync config-only with no image commits. The latest, d8d21151cd28, landed 151 min before onset, and nothing links any of them to this error.
Next step: look up question 5571182f-6c21-491e-b8c6-51fce3166e1e and its activity graph, and check why it has no refId, variableName or title. Start from the example trace.
This is also driving the error budget burn on execsvc (18x over 1 h). · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at tildadb/util.go:1004). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors, all at 16:49 UTC: 5× 'failed to run cypher in transaction' (tildadb/util.go:1004), plus 14× util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12+5× document copy (doclib/copyDocument.go:74) and 5× scope binders (doclib/context.go:335).
The errors have not recurred since then (last seen 16:49 per error lines).
The 5 deploys in the prior 24 h were config-only autosync; the closest was 15:39 (69 min before), with no image or commit change and nothing tying it to these files.
The util.go:993 deadlock shape has existed since 2026-11-15, so deadlocks are not new; what is new is the document-copy write path hitting them.
Next step: open the example trace and check which concurrent document copies (bulk doc write vs. context scope binders) took ExclusiveLocks on the same nodes, then confirm the copy path retries on TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)

execsvc logged “failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[tran” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/context.go:335). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors at 16:49 UTC while copying documents.
This finding is 5× 'failed to add document to scope binders' (doclib/context.go:335). Same burst: 14× tildadb/util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12× + 5× copy failures (doclib/copyDocument.go:74).
All lines are from 16:49 UTC only. The util.go:993 deadlock shape was already seen on 2026-11-15, so deadlocks are not new.
Checked: the 5 deploys in the prior 24 h were config-only autosyncs (latest 15:39, 69 min before) with no new commits. Nothing links them to this burst.
Next step: open the example trace to see which concurrent document copies collided on the same nodes. Check whether the copy path retries Neo4j TransientErrors. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could)

execsvc logged “failed to create document copy — could not add document context: could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetec” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC (5x here at doclib/copyDocument.go:74 via context.go:335 scope binders).
The same burst produced 12x bulk-write deadlocks (createDocument.go:1015, copyDocument.go:74), 14x at tildadb/util.go:993 and 5x at util.go:1004, opening 4 sibling findings.
The deadlock shape at util.go:993 was already seen on 2026-11-15 07:38, so deadlocks predate this burst; the new part is which callers hit them.
Checked: the 5 deploys in the prior 24 h were autosync config-only changes (the latest at 15:39, 69 min before onset), with no image or commit change and nothing tying them to these writes.
Next step: open the example trace to see which concurrent copy transactions took the locks on the scope binders.
Also check whether copyDocument retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)

execsvc logged “failed to bulk-write document nodes — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/createDocument.go:1015). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors in one minute, 16:49 UTC on 2026-11-16.
Counts: 12x 'failed to bulk-write document nodes' (doclib/createDocument.go:1015), 12x copy failures (doclib/copyDocument.go:74), 14x tildadb/util.go:993, 5x scope binders (doclib/context.go:335).
All come from document-copy writes contending for the same lock; findings 17467/17469/17470/17471 are the same burst.
Checked: 5 autosync config-only deploys in the prior 24 h, the last at 15:39 UTC (70 min before), with no commits and nothing tying them to these writes. The util.go:993 deadlock shape was already seen on 2026-11-15 07:38.
Next step: open the example trace to see which concurrent copy requests collided, then check whether createDocument.go:1015 retries Neo.TransientError. · its evidence →

critical Error never seen before master last seen · since · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)

execsvc logged “failed to create document copy — could not write document node and core relationships: bulk doc write: Neo4jError: Neo.TransientError.Transaction.DeadlockDetect” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/copyDocument.go:74). Example trace: f91346437409440b0cc533d78b9297f2.

First look low confidence
Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC, last seen 16:55 UTC.
Counts: 12× copyDocument.go:74 (bulk doc write), 5× copyDocument.go:74 (context scope binders), 14× tildadb/util.go:993.
The util.go:993 deadlock shape has existed since 2026-11-15 07:38, so lock contention is not new; only the document-copy paths are.
Checked: 5 autosync deploys in the prior 24 h were config-only (no image or commit change, the last one 69 min before onset), with nothing linking them to the error. This finding has no earlier history.
Next step: open the example trace and find which concurrent transaction held the ExclusiveLock during the 16:49 copy burst.
Also check whether doclib retries Neo.TransientError, since this error class is retryable. · its evidence →