Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:36:33.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Incidents 0 happening now on master

What is wrong right now, by what it means for people. ?critical: users are affected now, or data is at risk — about zero a week. high: degraded or getting worse — fewer than five a week. medium: worth a look today, grouped. low: context, in the daily digest. An incident is judged on its current state and closes itself when it recovers; one that comes back within a day at no higher level is the same incident.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
0 now 0 critical 0 high 0 medium 0 low
Nothing happening now on master. 39 older findings still open below.
Open, but not observed for an hour 39 nothing has resolved them yet; they send nothing new unless they come back at a higher level
LevelIncidentWhatForSeenGoes to
high x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods Pod cannot start 1d 1d ago in-app / digest only
high feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2 4xx spike (403/401/404…) 1d 1d ago in-app / digest only
high x-ray-pulls (x-ray) image pull failing for at least 117 min — 1 pod Pod cannot start 9d 1d ago in-app / digest only
high Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min Neo4j queries failing 25d 1d ago in-app / digest only
high workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ? Background job failing 26d 1d ago in-app / digest only
high pubsub.training-assignment-master-trainingsub: 21 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj Background job failing 71d 1d ago in-app / digest only
high Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min Neo4j queries failing 13d 1d ago in-app / digest only
high Neo4j shared Neo4j, database test26: 36 Cypher queries failed in the last 15 min Neo4j queries failing 22d 1d ago in-app / digest only
medium pubsub.dashboards-master-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationForDocument, id ? Background job failing 26d 1d ago in-app / digest only
medium Neo4j shared Neo4j, database matrix_a: 7 Cypher queries failed in the last 15 min Neo4j queries failing 13d 1d ago in-app / digest only
medium execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T) Error never seen before 1d 1d ago in-app / digest only
medium execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop) Error never seen before 1d 1d ago in-app / digest only
medium execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could) Error never seen before 1d 1d ago in-app / digest only
medium execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T) Error never seen before 1d 1d ago in-app / digest only
medium execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor) Error never seen before 1d 1d ago in-app / digest only
medium Neo4j shared Neo4j, database master: 29 Cypher queries failed in the last 15 min Neo4j queries failing 21d 1d ago in-app / digest only
medium execsvc: a new error shape, 44× in the last hour, never seen in 90 days (pubsub not initialized, dropping onboarding event) Error never seen before 2d 2d ago in-app / digest only
medium Neo4j shared Neo4j, database test27: 24 Cypher queries failed in the last 15 min Neo4j queries failing 20d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 5× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e) Error never seen before 2d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 7× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at) Error never seen before 2d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 12× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er) Error never seen before 2d 2d ago in-app / digest only
medium studioai-master: a new error shape, 6× in the last hour, never seen in 90 days (Token refresh failed: Login failed: {?:?,?:?}) Error never seen before 2d 2d ago in-app / digest only
medium Neo4j shared Neo4j, database nexus: 10 Cypher queries failed in the last 15 min Neo4j queries failing 14d 2d ago in-app / digest only
low feedsvc request rate 27.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks) Traffic anomaly 4d 1d ago in-app / digest only
low master AI spend projects to $1,754 this month, 1.6× last month's $1,067 AI spend running high 13d 1d ago in-app / digest only
low ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h Autoscaler blind 26d 1d ago in-app / digest only
low virtual-assistant: p99 latency (ms) 17.64 s — 4.6× the fleet's usual 3.87 s Service outlier 26d 1d ago in-app / digest only
low chat-agent: p99 latency (ms) 1.5 min — 23.7× the fleet's usual 3.87 s (also p95 latency (ms) 8.25 s) Service outlier 26d 1d ago in-app / digest only
low workplansvc: Neo4j queries per request 622.4 — 140.7× the fleet's usual 4.4 Service outlier 26d 1d ago in-app / digest only
low studiosvc Neo4j load 118 q/min vs a normal 3 q/min ± 5 q/min for Tue 10:00 UTC (3 of the last 4 checks) Neo4j load anomaly 70d 1d ago in-app / digest only
low studiosvc: p95 latency (ms) 1.8 min — 85.2× the fleet's usual 1.28 s (also p99 latency (ms) ≥ 2.0 min) Service outlier 83d 1d ago in-app / digest only
low execsvc Neo4j load 24 q/min vs a normal 0 q/min ± 1 q/min for Tue 10:00 UTC (3 of the last 4 checks) Neo4j load anomaly 84d 1d ago in-app / digest only
low tusker: Neo4j queries per request 31.7 — 5.5× the fleet's usual 5.7 Service outlier 26d 1d ago in-app / digest only
low dashql: p95 latency (ms) 4.33 s — 3.0× the fleet's usual 1.46 s Service outlier 26d 1d ago in-app / digest only
low feedsvc: Neo4j queries per request 15.6 — 3.9× the fleet's usual 4.0 Service outlier 7d 1d ago in-app / digest only
low docsvc: Neo4j queries per request 5.1 — 2.1× the fleet's usual 2.4 Service outlier 84d 1d ago in-app / digest only
low execsvc: p99 latency (ms) 7.08 s — 12.9× the fleet's usual 550 ms (also p95 latency (ms) 4.01 s) Service outlier 83d 1d ago in-app / digest only
low workplansvc shape 252AE86D: p95 184 ms → 313 ms across automated's deploy at 2026-11-16 18:21:05 UTC (p = 1.3e-11, 201/259 runs) Query shape regressed at a deploy 1d 1d ago in-app / digest only
low docproc: Neo4j queries per request 76.8 — 33.7× the fleet's usual 2.3 Service outlier 26d 1d ago in-app / digest only