Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 48 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 07:13:40.
› →|
updated 20h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Home master

What needs you first, on the cluster you selected — arranged the way you left it. ?Every card draws on data X-ray already has: findings, the headline numbers, time series with their normal band, forecasts, notifications, deploys, error budgets, tests, evals, your saved views. Cards follow the selected cluster and time range unless you pin a cluster in a card's settings. The layout is saved for you alone; nobody else's Home changes.

sign in to arrange this page

Broken right now · master

critical Error never seen before Owner: Madhava Mukkamala (backup: Shashank Agrawal) since Mon 16 Nov 16:55

execsvc · new error: failed to create document copy — could not add document context: could not writ… · 5× in the last hour, never seen in 90 days

started 75 min after deploy 29620eb70 by autosync

First look: Cause not clear from logs and recent changes; execsvc document copies failed on Neo4j DeadlockDetected in one burst at 16:49 UTC (5x here at doclib/copyDocument.go:74 via context.go:335 scope binders…

critical Error never seen before Owner: Madhava Mukkamala (backup: Shashank Agrawal) since Mon 16 Nov 16:55

execsvc · new error: failed to run cypher in transaction — Neo4jError: Neo.TransientError.Transactio… · 5× in the last hour, never seen in 90 days

started 75 min after deploy 29620eb70 by autosync

First look: Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors, all at 16:49 UTC: 5× 'failed to run cypher in transaction' (tildadb/util.go:1004), plus 14× util.go…

Journeys not judged: request data for master stops at Tue 17 Nov 10:25.

Deployed in the last 2 hours · ArgoCD syncs

Nothing deployed to master in the last 2 hours.

Since you last looked

0 opened 4 escalated 9 resolved 11 reopened 39 open now · 12 critical · 13 info since 1d ago · on master

Open findings, ranked

26 warning or worse on master · 13 informational kept off the board · all findings

critical Neo4j queries failing master last seen 20h ago · since 25d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence cause not clear from logs and recent changes · read it →

critical Neo4j queries failing master last seen 20h ago · since 13d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence cause not clear from logs and recent changes · read it →

critical Neo4j queries failing master last seen 1d ago · since 22d ago · 365×

Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min

Neo4j queries failing: Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min. Facts: env shared, database test26, failed 36, window_min 15, project tilda-trial-dev.

First look low confidence cause not clear from logs and recent changes · read it →

critical 4xx spike (403/401/404…) master last seen 20h ago · since 1d ago · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, ca…

First look low confidence cause not clear from logs and recent changes · read it →

critical Pod cannot start master last seen 20h ago · since 1d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence most likely x-ray-services-master sync 556a243c8ac9 (autosync) · read it →

critical Background job failing master last seen 20h ago · since 25d ago · 545×

workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?

The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that …

First look low confidence cause not clear from logs and recent changes · read it →

critical Error never seen before master Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 1d ago · 3×

execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)

execsvc logged “failed to create document copy — could not write document node and core relationships: bulk doc write: Neo4jError: Neo.TransientError.Transaction.DeadlockDetect” 12 times in the last hour and never before…

First look low confidence cause not clear from logs and recent changes · read it →

Fleet at a glance

What is running

detectorson · last pass never
notificationsdry-run (outbox only) · alerts
Ask X-rayready
master ingest20h ago
automation ingest20h ago
nexus ingest20h ago
production ingest20h ago
delta ingest20h ago
test27 ingest20h ago
test26 ingest20h ago