Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 12:52:48.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Noise on automation, last 7 days

How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
critical
6.0/week
6 in 7 d · budget ≤ 1/week
service.silent 6
high
26.0/week
26 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 10
endpoint.failing 9
error.novel 5
job.failing 2
medium
73.0/week
73 in 7 d · budget grouped
error.novel 29
job.failing 18
endpoint.4xx_spike 8
service.silent 8
scaler.failing 4
budget.burning 3
pod.unstable 1
node.unhealthy 1
low
46.0/week
46 in 7 d · budget digest
anomaly.neo4j 16
anomaly.traffic 12
anomaly.latency 9
node.unhealthy 4
budget.burning 3
n1.fanout 2

critical + high as messages (30-minute correlation): 19.0/week · medium as hourly messages: 29 in 7 d · damped as the same incident: 433

critical + high incidents 19

high1d ago execsvc POST /api/trialexec/study/:id/subject/:id: 4 of 5 calls failed (80 %) in the last 15 min
high1d ago execsvc POST /api/trialexec/careplan/:id/visit/:id/task: 6 of 9 calls failed (67 %) in the last 15 min
high1d ago exportsvc: a new error shape, 101× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe)
high1d ago execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min
high2d ago tusker PUT /api/tasks/bulk: 4 of 8 calls failed (50 %) in the last 15 min, 2 more were 4xx
high2d ago studiosvc GET /api/studio/triggers: permission denied — 33 of 33 calls (100 %; 403×33) in the last 15 min — 46× its usual 0.7 + 6 more in the same half hour
critical2d ago studiosvc on automation has served nothing for 2.1 h; at this hour it normally serves about 480 calls/hour + 2 more in the same half hour
high2d ago exportsvc: a new error shape, 114× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe)
critical2d ago docsvc on automation has served nothing for 4 h; at this hour it normally serves about 72 calls/hour
high3d ago docsvc POST /api/documents/folders/manager: 4 of 6 calls failed (67 %) in the last 15 min + 3 more in the same half hour
high3d ago execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min
high3d ago studiosvc GET /api/studio/triggers: permission denied — 21 of 21 calls (100 %; 403×21) in the last 15 min — 25× its usual 0.8
high3d ago pubsub.tilda-study-automation-regsvc-sub: 21 failed runs in the last 6 h — Plan definition not found for study ?
high3d ago exportsvc: a new error shape, 109× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe)
high3d ago onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min
critical3d ago onboardingsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 118 calls/hour + 2 more in the same half hour
high3d ago onboardingsvc: a new error shape, ≥115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us)
high3d ago onboardingsvc: a new error shape, ≥8000× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
critical4d ago execsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 106 calls/hour

What each rule sent

Rulemessagesby status