Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 14:42:44.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 89detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
warning Query shape regressed at a deploy master Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 1d ago · 17×

workplansvc shape 252AE86D: p95 184 ms → 313 ms across automated's deploy at 2026-11-16 18:21:05 UTC (p = 1.3e-11, 201/259 runs)

workplansvc shape 252AE86D ran slower after automated deployed workplansvc at 2026-11-16 18:21 UTC: p95 184 ms → 313 ms (1.7×) over 201 vs 259 executions, and the two latency distributions differ with p = 1.26e-11. That is the deploy being tested, not blamed: the statement itself may be unchanged and the data larger.

info Service outlier master Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 82d ago · 164×

enrollsvc: Neo4j queries per request 3.1 — 2.4× the fleet's usual 1.3

enrollsvc's Neo4j queries per request is 3.1 against the 7 other services' median of 1.3 (5.1 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Service outlier master Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 26d ago · 1407×

docproc: Neo4j queries per request 76.8 — 33.7× the fleet's usual 2.3

docproc's Neo4j queries per request is 76.8 against the 8 other services' median of 2.3 (44.3 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Latency anomaly master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 15d ago · 570×

studiosvc p95 2.0 min vs a normal 369 ms ± 242 ms for Mon 18:00 UTC (3 of the last 4 checks)

studiosvc's p95 is 120.0 s on master, above a normal of 369 ms ± 242 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 493.5, 4 weeks of history). automated changed studiosvc 54 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Neo4j load anomaly master resolved last seen 1d ago · since 2d ago · 29×

feedsvc Neo4j load 83 q/min vs a normal 0 q/min ± 20 q/min for Mon 18:00 UTC (4 of the last 4 checks)

feedsvc's Neo4j load is 83 q/min on master, above a normal of 0 q/min ± 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 4.2, 4 weeks of history). automated changed feedsvc 50 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Service gone silent master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 76d ago · 108×

docsvc on master has served nothing for 1.7 h; at this hour it normally serves about 144 calls/hour

docsvc on master has served no API calls for 100 minutes (last call 2026-11-16 17:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 144 calls an hour, 72 in the window checked. It carried 10.1% of the cluster's calls over the last week. A deploy landed just before the silence began — see the facts. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.

info Latency anomaly master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 26d ago · 30×

docsvc p95 1.69 s vs a normal 494 ms ± 213 ms for Mon 17:00 UTC (3 of the last 4 checks)

docsvc's p95 is 1.7 s on master, above a normal of 494 ms ± 213 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 5.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Neo4j load anomaly master Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 1d ago · since 26d ago · 88×

virtual-assistant Neo4j load 192 q/min vs a normal 1 q/min ± 1 q/min for Mon 16:00 UTC (3 of the last 4 checks)

virtual-assistant's Neo4j load is 192 q/min on master, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 128.7, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.