Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 14:39:36.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 98detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
info Traffic anomaly master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 68d ago · 309×

studiosvc request rate 73.4 req/min vs a normal 2.6 req/min ± 3.9 req/min for Tue 10:00 UTC (3 of the last 4 checks)

studiosvc's request rate is 73.4 req/min on master, above a normal of 2.6 req/min ± 3.9 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 18.4, 4 weeks of history). automated changed studiosvc 56 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Error budget burning master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 83d ago · 233×

execsvc is burning its error budget at 0.0× (1 h) / 8.3× (6 h) against a 95.00% objective

execsvc is spending its error budget 0.0× faster than the objective allows over the last hour (8.3× over six hours). The objective is 95.0%, derived — a baseline, not a promise. 787 bad events against 1,905 calls in the window.

info Service outlier master Owner: Ojas Kale (backup: Taylor Futral) last seen 1d ago · since 26d ago · 771×

tusker: Neo4j queries per request 31.7 — 5.5× the fleet's usual 5.7

tusker's Neo4j queries per request is 31.7 against the 12 other services' median of 5.7 (3.8 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info N+1 fan-out master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 83d ago · 177×

docsvc runs 45 foreground Neo4j queries per request (17,242 queries / 384 calls, last hour)

docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (17,242 queries for 384 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

info Service outlier master Owner: Vishal Karedla (backup: Taylor Futral) last seen 1d ago · since 26d ago · 1393×

dashql: p95 latency (ms) 4.33 s — 3.0× the fleet's usual 1.46 s

dashql's p95 latency (ms) is 4.33 s against the 9 other services' median of 1.46 s (3.9 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info N+1 fan-out master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 21d ago · 23×

execsvc runs 26 foreground Neo4j queries per request (3,407 queries / 130 calls, last hour)

execsvc on master averaged 26.2 foreground Neo4j queries per API request over the last hour (3,407 queries for 130 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

critical Neo4j queries failing master last seen 1d ago · since 22d ago · 365×

Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min

Neo4j queries failing: Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min. Facts: env shared, database test26, failed 36, window_min 15, project tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j, database test26 (project tilda-trial-dev): 662 Cypher queries failed in 15 min, 234 occurrences, this episode since 2026-11-10 19:08 UTC (first seen 2026-10-27, last 2026-11-16 06:22).
Checked: 12 autosync config deploys (ash-nazg 991526a739f8 at 18:05, 670c5dce81a4 at 18:35) landed 33–63 min before onset. No error line, changed file or code ties them to Neo4j.
No stored error line comes from Neo4j. Top shapes are fluentbit, cilium and execsvc branch evaluation, all starting days later.
The same finding recurred 2026-10-27→11-10 with no cause found, so this looks chronic to test26 rather than tied to a deploy.
Next step: open the Neo4j error logs for test26 to see the actual Cypher error and which client sends the failing queries. · its evidence →

info Traffic anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 84d ago · 258×

execsvc request rate 6.8 req/min vs a normal 0.2 req/min ± 0.3 req/min for Tue 05:00 UTC (3 of the last 4 checks)

execsvc's request rate is 6.8 req/min on master, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 22.3, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.