Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 20:48:27.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 143detectors over stored rollups and live sources

What X-ray is prepared to defend about automation right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
automation
just automation · nothing ticked means automation, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 7 critical · 20 warning · 13 info detectors
warning Out-of-memory kills automation last seen 1d ago · since 1d ago · 6×

the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz

Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.

info Latency anomaly automation Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 15d ago · 198×

studiosvc p95 26.46 s vs a normal 485 ms ± 437 ms for Tue 10:00 UTC (3 of the last 4 checks)

studiosvc's p95 is 26.5 s on automation, above a normal of 485 ms ± 437 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 59.4, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Traffic anomaly automation Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 17d ago · 1922×

tmfsync request rate 20.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (4 of the last 4 checks)

tmfsync's request rate is 20.0 req/min on automation, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (4 of the last 4 five-minute buckets, z = 4.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Background job failing automation last seen 1d ago · since 26d ago · 158×

studioai.unknown: 7 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in _handle_organization_creation activity_id = cr

The background job studioai.unknown on automation failed 7 runs recently (655 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

critical Background job failing automation last seen 1d ago · since 26d ago · 447×

pubsub.tilda-study-automation-regsvc-sub: 26 failed runs in the last 6 h — Plan definition not found for study ?

The background job pubsub.tilda-study-automation-regsvc-sub on automation failed 26 runs recently (201 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Service outlier automation Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 26d ago · 2821×

studiosvc: p99 latency (ms) ≥ 2.0 min — 75.2× the fleet's usual 1.60 s (also p95 latency (ms) 42.23 s; Neo4j queries per request 27.7)

studiosvc differs from the other 13 services on 3 measures over six hours: p99 latency (ms) ≥ 2.0 min against the peers' 1.60 s (76.5 MADs above); p95 latency (ms) 42.23 s against the peers' 671 ms (71.4 MADs above); Neo4j queries per request 27.7 against the peers' 4.1 (5.1 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Service outlier automation Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 26d ago · 2210×

docproc: Neo4j queries per request 34.7 — 8.4× the fleet's usual 4.1

docproc's Neo4j queries per request is 34.7 against the 13 other services' median of 4.1 (6.7 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.