Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 21:33:14.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 68detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
info Traffic anomaly test27 Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 83d ago · 14×

execsvc request rate 143.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 07:00 UTC (4 of the last 4 checks)

execsvc's request rate is 143.8 req/min on test27, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 28.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test27 Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 84d ago · 218×

execsvc Neo4j load 591 q/min vs a normal 0 q/min ± 20 q/min for Tue 07:00 UTC (4 of the last 4 checks)

execsvc's Neo4j load is 591 q/min on test27, above a normal of 0 q/min ± 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 29.5, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 1d ago · since 84d ago · 2099×

virtual-assistant Neo4j load 94 q/min vs a normal 2 q/min ± 1 q/min for Tue 07:00 UTC (3 of the last 4 checks)

virtual-assistant's Neo4j load is 94 q/min on test27, above a normal of 2 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 103.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Service outlier test27 Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) last seen 1d ago · since 26d ago · 171×

enrollsvc: Neo4j queries per request 42.0 — 7.1× the fleet's usual 5.9 (also p95 latency (ms) 1.98 s)

enrollsvc differs from the other 6 services on 2 measures over six hours: Neo4j queries per request 42.0 against the peers' 5.9 (6.3 MADs above); p95 latency (ms) 1.98 s against the peers' 404 ms (3.6 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Neo4j load anomaly test27 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 26d ago · 158×

docsvc Neo4j load 7,364 q/min vs a normal 1 q/min ± 1 q/min for Tue 06:00 UTC (3 of the last 4 checks)

docsvc's Neo4j load is 7,364 q/min on test27, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 12415.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Service gone silent test27 Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 69d ago · 93×

onboardingsvc on test27 has served nothing for 40 min; at this hour it normally serves about 132 calls/hour

onboardingsvc on test27 has served no API calls for 40 minutes (last call 2026-11-17 05:00 UTC); at this hour of the week its baseline — 5 weeks of history — expects about 132 calls an hour, 66 in the window checked. It carried 10.6% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.

warning 4xx spike (403/401/404…) test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 16d ago · 128×

studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3

4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 84, c4xx 84, share_pct 100.0, c401 84.

info Service outlier test27 Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 26d ago · 475×

execsvc: p95 latency (ms) 3.05 s — 9.7× the fleet's usual 315 ms

execsvc's p95 latency (ms) is 3.05 s against the 5 other services' median of 315 ms (5.9 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Service outlier test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 84d ago · 3365×

workplansvc: Neo4j queries per request 969.5 — 235.1× the fleet's usual 4.1

workplansvc's Neo4j queries per request is 969.5 against the 6 other services' median of 4.1 (175.8 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

warning Query shape degrading test27 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 70d ago · 141×

docsvc shape 00000000: p95 1.65 s now vs 69 ms earlier (23.9×, 1,858 runs)

A docsvc query shape on test27 has p95 1.6 s in the second half of the window against 69 ms in the first (23.9×, 1,858 executions). The shape reads: “MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.