Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 12:47:28.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 100detectors over stored rollups and live sources

What X-ray is prepared to defend about test26 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test26
just test26 · nothing ticked means test26, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 10 warning · 8 info detectors
warning Query shape degrading test26 resolved last seen 1d ago · since 20d ago · 14×

feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)

A feedsvc query shape on test26 has p95 4.8 s in the second half of the window against 28 ms in the first (172.7×, 3,394 executions). The shape reads: “RETURN EXISTS { MATCH (cfg2:WorkplanCfg) WHERE cfg2.progressPending = true AND coalesce(cfg2.lastViewedAtMs, ?) > (times…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

info Traffic anomaly test26 resolved last seen 1d ago · since 84d ago · 1255×

test26 (whole cluster) request rate 56.4 req/min vs a normal 0.5 req/min ± 0.7 req/min for Tue 07:00 UTC (4 of the last 4 checks)

the whole test26 cluster's request rate is 56.4 req/min on test26, above a normal of 0.5 req/min ± 0.7 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 75.4, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Background job failing test26 resolved last seen 1d ago · since 39d ago · 99×

pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch

The background job pubsub.study-smo-binding-test26-enrollsub on test26 failed 5 runs recently (4,845 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Neo4j load anomaly test26 Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 23d ago · 95×

enrollsvc Neo4j load 43 q/min vs a normal 3 q/min ± 4 q/min for Tue 06:00 UTC (3 of the last 4 checks)

enrollsvc's Neo4j load is 43 q/min on test26, above a normal of 3 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 8.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Traffic anomaly test26 resolved last seen 1d ago · since 21d ago · 3×

feedsvc request rate 25.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 01:00 UTC (4 of the last 4 checks)

feedsvc's request rate is 25.8 req/min on test26, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 5.2, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test26 resolved last seen 1d ago · since 21d ago · 27×

feedsvc Neo4j load 93 q/min vs a normal 0 q/min ± 20 q/min for Tue 00:00 UTC (3 of the last 4 checks)

feedsvc's Neo4j load is 93 q/min on test26, above a normal of 0 q/min ± 20 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 4.6, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Query shape degrading test26 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 75d ago · 168×

docsvc shape 00000000: p95 1.29 s now vs 55 ms earlier (23.2×, 941 runs)

A docsvc query shape on test26 has p95 1.3 s in the second half of the window against 55 ms in the first (23.2×, 941 executions). The shape reads: “MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.i…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.