Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:25:08.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 39detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
warning Background job failing test27 last seen 1d ago · since 16d ago · 22×

pubsub.onboarding-test27-regsvc-sub: 19 failed runs in the last 6 h — failed to create triggers: failed to fetch existing triggers: failed to fetch triggers: Error from A

The background job pubsub.onboarding-test27-regsvc-sub on test27 failed 19 runs recently (528 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning Background job failing test27 last seen 1d ago · since 21d ago · 34×

studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^

The background job studioai.unknown on test27 failed 5 runs recently (532 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning Pod keeps restarting test27 resolved last seen 1d ago · since 1d ago · 1×

jss-server on test27 has restarted 9 times in 10 min — crash loop, image spreadsheet-agent-edits-d3202c0

jss-server in tilda on test27 has restarted in place 9 times since 2026-11-17 10:05 UTC (9 in the last 30 minutes, 9 in 24 hours), most recently at 2026-11-17 10:15 UTC. Kubernetes' account of it: 11 crash events, 3 sched events. Last event: BackOff — “Back-off restarting failed container jss-server in pod jss-server-5968bc86d8-pptmg_tilda(fda1a1d7-5f21-42bd-b73d-7d92a2cf2383)”. It is running image spreadsheet-agent-edits-d3202c0. Containers restarting: jss-server ×9. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

warning Query shape degrading test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 1d ago · 4×

workplansvc shape 50B96E05: p95 1.45 s now vs 488 ms earlier (3.0×, 223 runs)

A workplansvc query shape on test27 has p95 1.4 s in the second half of the window against 488 ms in the first (3.0×, 223 executions). The shape reads: “MATCH (wp:WorkPlanInstance)-[:WORKPLAN_INSTANCE_TO_MILESTONE]->(:WorkPlanMilestone)-[:REL_WORKPLAN_INSTANCE_MILESTONE_TO…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Query shape degrading test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 3d ago · since 3d ago · 1×

workplansvc shape 6947E55E: p95 1.20 s now vs 493 ms earlier (2.4×, 272 runs)

A workplansvc query shape on test27 has p95 1.2 s in the second half of the window against 493 ms in the first (2.4×, 272 executions). The shape reads: “MATCH (wp:WorkPlanInstance)-[:WORKPLAN_INSTANCE_TO_MILESTONE]->(:WorkPlanMilestone)-[:REL_WORKPLAN_INSTANCE_MILESTONE_TO…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Service gone silent test27 Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 69d ago · 93×

onboardingsvc on test27 has served nothing for 40 min; at this hour it normally serves about 132 calls/hour

onboardingsvc on test27 has served no API calls for 40 minutes (last call 2026-11-17 05:00 UTC); at this hour of the week its baseline — 5 weeks of history — expects about 132 calls an hour, 66 in the window checked. It carried 10.6% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.

warning 4xx spike (403/401/404…) test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 16d ago · 128×

studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3

4xx spike (403/401/404…): studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3. Facts: endpoint GET /api/studio/triggers, service studiosvc, calls 84, c4xx 84, share_pct 100.0, c401 84.