Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 02:48:02.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Incidents 0 happening now on automation

What is wrong right now, by what it means for people. ?critical: users are affected now, or data is at risk — about zero a week. high: degraded or getting worse — fewer than five a week. medium: worth a look today, grouped. low: context, in the daily digest. An incident is judged on its current state and closes itself when it recovers; one that comes back within a day at no higher level is the same incident.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
0 now 0 critical 0 high 0 medium 0 low
Nothing happening now on automation. 40 older findings still open below.
Open, but not observed for an hour 40 nothing has resolved them yet; they send nothing new unless they come back at a higher level
LevelIncidentWhatForSeenGoes to
high pubsub.tilda-study-automation-regsvc-sub: 26 failed runs in the last 6 h — Plan definition not found for study ? Background job failing 26d 1d ago in-app / digest only
high onboardingsvc: a new error shape, 115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us) Error never seen before 4d 4d ago in-app / digest only
medium the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz Out-of-memory kills 1d 1d ago in-app / digest only
medium studioai.unknown: 7 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in _handle_organization_creation activity_id = cr Background job failing 26d 1d ago in-app / digest only
medium workplan.workplan_config: 12 failed runs in the last 6 h — A listed condition would be lost: milestone ? has ? review or checkpoint steps, so the decision it n Background job failing 29d 1d ago in-app / digest only
medium pubsub.workplan-config-response-automation-workplansvc-sub: 12 failed runs in the last 6 h — A listed condition would be lost: milestone ? has ? review or checkpoint steps, so the decision it n Background job failing 29d 1d ago in-app / digest only
medium pubsub.tilda-study-automation-execsvc-sub: 8 failed runs in the last 6 h — failed to get study &{{? ? ? ? false ?} E2E CRO Study Humacyte ? ? enrolling ? ? ? ? false true fals Background job failing 75d 1d ago in-app / digest only
medium pubsub.dashboards-automation-dashsub: 17 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ? Background job failing 75d 1d ago in-app / digest only
medium workplansvc on automation has served nothing for 1.2 h; at this hour it normally serves about 318 calls/hour Service gone silent 1d 1d ago in-app / digest only
medium pubsub.patient-signup-automation-notifsub: 7 failed runs in the last 6 h — job not found: objLabel Job: get with empty tildaId requires a struct with at least one non-inherite Background job failing 75d 1d ago in-app / digest only
medium exportsvc: a new error shape, 7× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) Error never seen before 2d 2d ago in-app / digest only
medium onboardingsvc: a new error shape, 5× in the last hour, never seen in 90 days (Could not begin transaction — Neo4jError: Neo.ClientError.Database.Dat) Error never seen before 2d 2d ago in-app / digest only
medium virtual-assistant: a new error shape, 23× in the last hour, never seen in 90 days ([PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to) Error never seen before 2d 2d ago in-app / digest only
medium execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor ) Error never seen before 2d 2d ago in-app / digest only
medium docproc: a new error shape, 5× in the last hour, never seen in 90 days (Recovery failed — AIRun creation failed for document ? (document proce) Error never seen before 2d 2d ago in-app / digest only
medium execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor ) Error never seen before 2d 2d ago in-app / digest only
medium docproc: a new error shape, 6× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse) Error never seen before 2d 2d ago in-app / digest only
medium execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga) Error never seen before 2d 2d ago in-app / digest only
medium execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat) Error never seen before 2d 2d ago in-app / digest only
medium docproc: a new error shape, 5× in the last hour, never seen in 90 days (Neo4j health check failed for neo4j://db2.neo4j.shared:?) Error never seen before 2d 2d ago in-app / digest only
medium exportsvc: a new error shape, 7× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) Error never seen before 2d 2d ago in-app / digest only
medium exportsvc: a new error shape, 7× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) Error never seen before 2d 2d ago in-app / digest only
medium exportsvc: a new error shape, 7× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) Error never seen before 3d 3d ago in-app / digest only
low automation AI spend projects to $2,236 this month, 3.8× last month's $585 AI spend running high 13d 1d ago in-app / digest only
low HorizontalPodAutoscaler/ontologysvc-hpa (matrix) cannot read its scaling metric — 26× in 2 h Autoscaler blind 14d 1d ago in-app / digest only
low HorizontalPodAutoscaler/docproc-hpa (matrix) cannot read its scaling metric — 33× in 2 h Autoscaler blind 14d 1d ago in-app / digest only
low studiosvc p95 26.46 s vs a normal 485 ms ± 437 ms for Tue 10:00 UTC (3 of the last 4 checks) Latency anomaly 15d 1d ago in-app / digest only
low tmfsync request rate 20.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (4 of the last 4 checks) Traffic anomaly 17d 1d ago in-app / digest only
low studiosvc: p99 latency (ms) ≥ 2.0 min — 75.2× the fleet's usual 1.60 s (also p95 latency (ms) 42.23 s; Neo4j queries per request 27.7) Service outlier 26d 1d ago in-app / digest only
low docproc: Neo4j queries per request 34.7 — 8.4× the fleet's usual 4.1 Service outlier 26d 1d ago in-app / digest only
low pods on an unready node: 2 affected in 30 min Node problem 50d 1d ago in-app / digest only
low regsvc: Neo4j queries per request 169.6 — 35.3× the fleet's usual 4.8 Service outlier 84d 1d ago in-app / digest only
low virtual-assistant: p99 latency (ms) 9.79 s — 6.8× the fleet's usual 1.43 s Service outlier 26d 1d ago in-app / digest only
low docsvc: 5xx rate (%) 0.55% — 7.5× the fleet's usual 0.07% Service outlier 26d 1d ago in-app / digest only
low workplansvc: 5xx rate (%) 0.92% — 12.6× the fleet's usual 0.07% Service outlier 83d 1d ago in-app / digest only
low finsvc: 5xx rate (%) 0.80% — 10.8× the fleet's usual 0.07% Service outlier 84d 1d ago in-app / digest only
low onboardingsvc: 5xx rate (%) 0.34% — 6.3× the fleet's usual 0.05% Service outlier 83d 2d ago in-app / digest only
low execsvc: 5xx rate (%) 0.31% — 6.6× the fleet's usual 0.05% Service outlier 84d 2d ago in-app / digest only
low dashql: p95 latency (ms) 1.27 s — 4.9× the fleet's usual 260 ms Service outlier 26d 2d ago in-app / digest only
low tusker: Neo4j queries per request 30.2 — 4.2× the fleet's usual 7.2 Service outlier 83d 3d ago in-app / digest only