Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 07:13:42.
› →|
updated 20h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Home automation

What needs you first, on the cluster you selected — arranged the way you left it. ?Every card draws on data X-ray already has: findings, the headline numbers, time series with their normal band, forecasts, notifications, deploys, error budgets, tests, evals, your saved views. Cards follow the selected cluster and time range unless you pin a cluster in a card's settings. The layout is saved for you alone; nobody else's Home changes.

sign in to arrange this page

Broken right now · automation

Cannot confirm automation is healthy — neo4j errors last ran cleanly Mon 16 Nov 06:57; endpoint 4xx spike last ran cleanly Mon 16 Nov 06:57; endpoint failing last ran cleanly Mon 16 Nov 06:57; errors novel last ran cleanly Mon 16 Nov 06:42; request data stops at Tue 17 Nov 10:25 — detectors

Journeys not judged: request data for automation stops at Tue 17 Nov 10:25.

Deployed in the last 2 hours · ArgoCD syncs

Nothing deployed to automation in the last 2 hours.

Since you last looked

1 opened 0 escalated 43 resolved 28 reopened 40 open now · 7 critical · 13 info since 1d ago · on automation

Open findings, ranked

27 warning or worse on automation · 13 informational kept off the board · all findings

critical Background job failing automation last seen 20h ago · since 25d ago · 447×

pubsub.tilda-study-automation-regsvc-sub: 26 failed runs in the last 6 h — Plan definition not found for study ?

The background job pubsub.tilda-study-automation-regsvc-sub on automation failed 26 runs recently (201 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failu…

critical Error never seen before automation Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) last seen 1d ago · since 1d ago · 3×

onboardingsvc: a new error shape, 5× in the last hour, never seen in 90 days (Could not begin transaction — Neo4jError: Neo.ClientError.Database.Dat)

onboardingsvc logged “Could not begin transaction — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Database does not exist. Database name: ?.)” 5 times in the last hour and never before in 90 days on automation (…

critical Error never seen before automation Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 1d ago · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )

execsvc logged “Request completed — ? · {?:" failed to create Actor for Study ? Actor {? [{{ 90a172de-995e-422f-93ff-01f4” 5 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:12 U…

critical Error never seen before automation Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 1d ago · 3×

execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )

execsvc logged “Request completed — ? · {?:" failed to create Actor for Study ? Actor {? [{{ 5c4ac420-2d4d-4efe-b649-985c” 5 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:11 U…

critical Error never seen before automation Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 1d ago · 3×

docproc: a new error shape, 6× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)

docproc logged “Failed to create document processing AIRun — aborting pipeline (refuse to run untracked) — docsvc update failed: HTTP ?” 6 times in the last hour and never before in 90 days on automation (first seen 2026…

critical Error never seen before automation Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 1d ago · 3×

docproc: a new error shape, 5× in the last hour, never seen in 90 days (Recovery failed — AIRun creation failed for document ? (document proce)

docproc logged “Recovery failed — AIRun creation failed for document ? (document processing): docsvc update failed: HTTP ?” 5 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:11 …

critical Error never seen before automation Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) last seen 3d ago · since 3d ago · 3×

onboardingsvc: a new error shape, 115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us)

onboardingsvc logged “error on pubsub receive — rpc error: code = PermissionDenied desc = User not authorized to perform this action.” 115 times in the last hour and never before in 90 days on automation (first seen 2026…

warning Background job failing automation last seen 20h ago · since 74d ago · 801×

pubsub.tilda-study-automation-execsvc-sub: 8 failed runs in the last 6 h — failed to get study &{{? ? ? ? false ?} E2E CRO Study Humacyte ? ? enrolling ? ? ? ? false true fals

The background job pubsub.tilda-study-automation-execsvc-sub on automation failed 8 runs recently (32 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severi…

Fleet at a glance

What is running

detectorson · last pass never
notificationsdry-run (outbox only) · alerts
Ask X-rayready
master ingest20h ago
automation ingest20h ago
nexus ingest20h ago
production ingest20h ago
delta ingest20h ago
test27 ingest20h ago
test26 ingest20h ago