Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:14:35.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 9

stored age unknown all 9detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
critical Neo4j queries failing master last seen 1d ago · since 26d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j database 'automation' (project tilda-trial-dev) had 37 failed Cypher queries in the last 15 min, 366 occurrences since 2026-10-23 19:21 UTC.
Checked: 12 autosync config deploys in the 24 h before onset (closest: execsvc 57 min and studiosvc 55 min before). None has an image tag, commit or error line tying it to Neo4j.
None of the 6 error shapes in the bundle comes from neo4j or names a Cypher query. execsvc 'no result from branch evaluation' first appeared 2026-11-13, weeks after onset.
No earlier episodes and no other findings opened at the same time.
Next step: open the Neo4j error logs for this window, read the Cypher error text, and find which client service sends the failing queries to 'automation'. · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 14d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; Neo4j shared, database matrix_b: 35 Cypher queries failed in the last 15 min (213 occurrences since onset).
Failing since 2026-11-04 14:19 UTC, still seen 2026-11-17 10:21 UTC; matrix_a also started failing 17 min later (7 queries).
Checked: 12 deploys in the 24 h before onset (11 config syncs at revision c8a04003ea3d by [email protected] 205 min before, dashrsvc autosync 67 min before). Nothing links them to Neo4j.
None of the stored cluster error shapes are Neo4j or Cypher errors, and no single service owns the queries.
Next step: open the Neo4j error logs for the window to get the failing Cypher error codes and the client issuing them. · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 22d ago · 365×

Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min

Neo4j queries failing: Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min. Facts: env shared, database test26, failed 36, window_min 15, project tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j, database test26 (project tilda-trial-dev): 662 Cypher queries failed in 15 min, 234 occurrences, this episode since 2026-11-10 19:08 UTC (first seen 2026-10-27, last 2026-11-16 06:22).
Checked: 12 autosync config deploys (ash-nazg 991526a739f8 at 18:05, 670c5dce81a4 at 18:35) landed 33–63 min before onset. No error line, changed file or code ties them to Neo4j.
No stored error line comes from Neo4j. Top shapes are fluentbit, cilium and execsvc branch evaluation, all starting days later.
The same finding recurred 2026-10-27→11-10 with no cause found, so this looks chronic to test26 rather than tied to a deploy.
Next step: open the Neo4j error logs for test26 to see the actual Cypher error and which client sends the failing queries. · its evidence →

warning Query shape regressed at a deploy master Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 2d ago · 17×

workplansvc shape 252AE86D: p95 184 ms → 313 ms across automated's deploy at 2026-11-16 18:21:05 UTC (p = 1.3e-11, 201/259 runs)

workplansvc shape 252AE86D ran slower after automated deployed workplansvc at 2026-11-16 18:21 UTC: p95 184 ms → 313 ms (1.7×) over 201 vs 259 executions, and the two latency distributions differ with p = 1.26e-11. That is the deploy being tested, not blamed: the statement itself may be unchanged and the data larger.

warning Query shape degrading master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 2d ago · since 2d ago · 3×

studiosvc shape 8D3802C7: p95 1.73 s now vs 746 ms earlier (2.3×, 232 runs)

A studiosvc query shape on master has p95 1.7 s in the second half of the window against 746 ms in the first (2.3×, 232 executions). The shape reads: “MATCH (proto:FormV2 {tildaid: $protoId})-[pr:PROTOCOL_TO_GRAPH_NODE_REF]->(ref:GraphNodeRef) MATCH (ref)-[:GRAPH_NODE_RE…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.