Noise on automation, last 7 days
How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.
critical
6.0/week
6 in 7 d · budget ≤ 1/week
service.silent 6
high
21.0/week
21 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 9
endpoint.failing 8
error.novel 2
job.failing 2
medium
60.0/week
60 in 7 d · budget grouped
error.novel 23
job.failing 17
endpoint.4xx_spike 8
scaler.failing 4
budget.burning 3
service.silent 2
node.unhealthy 1
pod.unstable 1
low
38.0/week
38 in 7 d · budget digest
anomaly.neo4j 13
anomaly.traffic 12
anomaly.latency 7
budget.burning 3
node.unhealthy 2
n1.fanout 1
critical + high as messages (30-minute correlation): 13.0/week · medium as hourly messages: 25 in 7 d · damped as the same incident: 322
critical + high incidents 13
| high | 5d ago | execsvc POST /api/trialexec/study/:id/subject/:id: 4 of 5 calls failed (80 %) in the last 15 min |
| high | 5d ago | execsvc POST /api/trialexec/careplan/:id/visit/:id/task: 6 of 9 calls failed (67 %) in the last 15 min |
| high | 5d ago | pubsub.tilda-study-automation-regsvc-sub: 20 failed runs in the last 6 h — Plan definition not found for study ? |
| high | 5d ago | exportsvc: a new error shape, 101× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) |
| high | 5d ago | execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min |
| high | 5d ago | tusker PUT /api/tasks/bulk: 4 of 8 calls failed (50 %) in the last 15 min, 2 more were 4xx |
| high | 5d ago | studiosvc GET /api/studio/triggers: permission denied — 33 of 33 calls (100 %; 403×33) in the last 15 min — 46× its usual 0.7 + 6 more in the same half hour |
| high | 6d ago | feedsvc GET /studies/:id/feed: permission denied — 30 of 50 calls (60 %; 403×30) in the last 15 min — 17× its usual 1.8 + 1 more in the same half hour |
| critical | 6d ago | studiosvc on automation has served nothing for 2.1 h; at this hour it normally serves about 480 calls/hour + 3 more in the same half hour |
| high | 6d ago | pubsub.dashboards-automation-dashsub: 20 failed runs in the last 6 h — no data found for label RecommendationForDocument, id ? |
| high | 6d ago | exportsvc: a new error shape, 114× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) |
| high | 6d ago | onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 18 of 18 calls failed (100 %) in the last 15 min |
| critical | 6d ago | onboardingsvc on automation has served nothing for 4 h; at this hour it normally serves about 90 calls/hour + 4 more in the same half hour |
What each rule sent
| Rule | messages | by status |
|---|