Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-23 06:45:02.
› →|
updated 4d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Noise on automation, last 7 days

How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
critical
6.0/week
6 in 7 d · budget ≤ 1/week
service.silent 6
high
21.0/week
21 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 9
endpoint.failing 8
error.novel 2
job.failing 2
medium
60.0/week
60 in 7 d · budget grouped
error.novel 23
job.failing 17
endpoint.4xx_spike 8
scaler.failing 4
budget.burning 3
service.silent 2
node.unhealthy 1
pod.unstable 1
low
38.0/week
38 in 7 d · budget digest
anomaly.neo4j 13
anomaly.traffic 12
anomaly.latency 7
budget.burning 3
node.unhealthy 2
n1.fanout 1

critical + high as messages (30-minute correlation): 13.0/week · medium as hourly messages: 25 in 7 d · damped as the same incident: 322

critical + high incidents 13

high5d ago execsvc POST /api/trialexec/study/:id/subject/:id: 4 of 5 calls failed (80 %) in the last 15 min
high5d ago execsvc POST /api/trialexec/careplan/:id/visit/:id/task: 6 of 9 calls failed (67 %) in the last 15 min
high5d ago pubsub.tilda-study-automation-regsvc-sub: 20 failed runs in the last 6 h — Plan definition not found for study ?
high5d ago exportsvc: a new error shape, 101× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe)
high5d ago execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min
high5d ago tusker PUT /api/tasks/bulk: 4 of 8 calls failed (50 %) in the last 15 min, 2 more were 4xx
high5d ago studiosvc GET /api/studio/triggers: permission denied — 33 of 33 calls (100 %; 403×33) in the last 15 min — 46× its usual 0.7 + 6 more in the same half hour
high6d ago feedsvc GET /studies/:id/feed: permission denied — 30 of 50 calls (60 %; 403×30) in the last 15 min — 17× its usual 1.8 + 1 more in the same half hour
critical6d ago studiosvc on automation has served nothing for 2.1 h; at this hour it normally serves about 480 calls/hour + 3 more in the same half hour
high6d ago pubsub.dashboards-automation-dashsub: 20 failed runs in the last 6 h — no data found for label RecommendationForDocument, id ?
high6d ago exportsvc: a new error shape, 114× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe)
high6d ago onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 18 of 18 calls failed (100 %) in the last 15 min
critical6d ago onboardingsvc on automation has served nothing for 4 h; at this hour it normally serves about 90 calls/hour + 4 more in the same half hour

What each rule sent

Rulemessagesby status