Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 01:48:03.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 68detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
warning Background job failing test27 resolved last seen 3d ago · since 58d ago · 200×

pubsub.study-site-binding-test27-regsub: 5 failed runs in the last 6 h — internal code issue: Trying to commit a transaction which never started

The background job pubsub.study-site-binding-test27-regsub on test27 failed 5 runs recently (15 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Service outlier test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 3d ago · since 3d ago · 44×

ontologysvc: p95 latency (ms) 2.94 s — 8.1× the fleet's usual 364 ms (also p99 latency (ms) 4.59 s)

ontologysvc differs from the other 5 services on 2 measures over six hours: p95 latency (ms) 2.94 s against the peers' 364 ms (14.8 MADs above); p99 latency (ms) 4.59 s against the peers' 495 ms (7.5 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

tusker: a new error shape, 9× in the last hour, never seen in 90 days (failed to 3 targets worst: warning process db result — Neo4jError: Neo.TransientError.Transacti, write relationship cypher — Neo4jError: Neo.TransientError.T, setup site inbox — Neo4jError: Neo.TransientError.Transactio
warning Error never seen before test27 resolved last seen · since · 3×

tusker: a new error shape, 9× in the last hour, never seen in 90 days (failed to process db result — Neo4jError: Neo.TransientError.Transacti)

tusker logged “failed to process db result — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire ExclusiveLoc” 9 times in the last hour and never before in 90 days on test27 (first seen 2026-11-15 06:38 UTC, raised at tildadb/util.go:993). Example trace: 7294ed6c4e4116e90c91cfc7debb02b6.

warning Error never seen before test27 resolved last seen · since · 3×

tusker: a new error shape, 9× in the last hour, never seen in 90 days (failed to write relationship cypher — Neo4jError: Neo.TransientError.T)

tusker logged “failed to write relationship cypher — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire Excl” 9 times in the last hour and never before in 90 days on test27 (first seen 2026-11-15 06:38 UTC, raised at tildadb/api.go:1653). Example trace: 7294ed6c4e4116e90c91cfc7debb02b6.

warning Error never seen before test27 resolved last seen · since · 3×

tusker: a new error shape, 9× in the last hour, never seen in 90 days (failed to setup site inbox — Neo4jError: Neo.TransientError.Transactio)

tusker logged “failed to setup site inbox — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't acquire ExclusiveLock” 9 times in the last hour and never before in 90 days on test27 (first seen 2026-11-15 06:38 UTC, raised at pubsub/pubsub.go:330). Example trace: 7294ed6c4e4116e90c91cfc7debb02b6.

warning Error budget burning test27 Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 3d ago · since 26d ago · 210×

onboardingsvc is burning its error budget at 7.3× (1 h) / 7.4× (6 h) against a 99.00% objective

onboardingsvc is spending its error budget 7.3× faster than the objective allows over the last hour (7.4× over six hours). The objective is 99.0%, derived — a baseline, not a promise. 69 bad events against 929 calls in the window.

info Latency anomaly test27 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 4d ago · since 68d ago · 69×

docsvc p95 4.47 s vs a normal 91 ms ± 63 ms for Sat 18:00 UTC (3 of the last 4 checks)

docsvc's p95 is 4.5 s on test27, above a normal of 91 ms ± 63 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 69.2, 2 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Service outlier test27 Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) last seen 4d ago · since 4d ago · 49×

patientsvc: p95 latency (ms) 9.09 s — 42.8× the fleet's usual 212 ms

patientsvc's p95 latency (ms) is 9.09 s against the 5 other services' median of 212 ms (31.0 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

warning Query shape degrading test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 4d ago · since 13d ago · 8×

studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)

A studiosvc query shape on test27 has p95 2.2 s in the second half of the window against 128 ms in the first (16.8×, 865 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Error never seen before test27 Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 4d ago · since 4d ago · 3×

studiosvc: a new error shape, ≥13006× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)

studiosvc logged “error on pubsub receive — rpc error: code = NotFound desc = Resource not found (resource=study-site-binding-test27-studiosvcsub).” 13,006 times in the last hour and never before in 90 days on test27 (first seen 2026-11-14 14:23 UTC, raised at pubsublib/googlepubsub.go:554). Example trace: none carried.

info Latency anomaly test27 Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 4d ago · since 76d ago · 74×

execsvc p95 1.21 s vs a normal 98 ms ± 100 ms for Sat 12:00 UTC (3 of the last 4 checks)

execsvc's p95 is 1.2 s on test27, above a normal of 98 ms ± 100 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 11.1, 2 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.