Noise on automation, last 7 days
How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.
critical
6.0/week
6 in 7 d · budget ≤ 1/week
service.silent 6
high
26.0/week
26 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 10
endpoint.failing 9
error.novel 5
job.failing 2
medium
73.0/week
73 in 7 d · budget grouped
error.novel 29
job.failing 18
endpoint.4xx_spike 8
service.silent 8
scaler.failing 4
budget.burning 3
pod.unstable 1
node.unhealthy 1
low
46.0/week
46 in 7 d · budget digest
anomaly.neo4j 16
anomaly.traffic 12
anomaly.latency 9
node.unhealthy 4
budget.burning 3
n1.fanout 2
critical + high as messages (30-minute correlation): 19.0/week · medium as hourly messages: 29 in 7 d · damped as the same incident: 433
critical + high incidents 19
| high | 1d ago | execsvc POST /api/trialexec/study/:id/subject/:id: 4 of 5 calls failed (80 %) in the last 15 min |
| high | 1d ago | execsvc POST /api/trialexec/careplan/:id/visit/:id/task: 6 of 9 calls failed (67 %) in the last 15 min |
| high | 1d ago | exportsvc: a new error shape, 101× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) |
| high | 1d ago | execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min |
| high | 2d ago | tusker PUT /api/tasks/bulk: 4 of 8 calls failed (50 %) in the last 15 min, 2 more were 4xx |
| high | 2d ago | studiosvc GET /api/studio/triggers: permission denied — 33 of 33 calls (100 %; 403×33) in the last 15 min — 46× its usual 0.7 + 6 more in the same half hour |
| critical | 2d ago | studiosvc on automation has served nothing for 2.1 h; at this hour it normally serves about 480 calls/hour + 2 more in the same half hour |
| high | 2d ago | exportsvc: a new error shape, 114× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) |
| critical | 2d ago | docsvc on automation has served nothing for 4 h; at this hour it normally serves about 72 calls/hour |
| high | 3d ago | docsvc POST /api/documents/folders/manager: 4 of 6 calls failed (67 %) in the last 15 min + 3 more in the same half hour |
| high | 3d ago | execsvc POST /api/trialexec/careplan/:id/task: 6 of 8 calls failed (75 %) in the last 15 min |
| high | 3d ago | studiosvc GET /api/studio/triggers: permission denied — 21 of 21 calls (100 %; 403×21) in the last 15 min — 25× its usual 0.8 |
| high | 3d ago | pubsub.tilda-study-automation-regsvc-sub: 21 failed runs in the last 6 h — Plan definition not found for study ? |
| high | 3d ago | exportsvc: a new error shape, 109× in the last hour, never seen in 90 days (Cache not found for activity update: Cache not found for study ?. Expe) |
| high | 3d ago | onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min |
| critical | 3d ago | onboardingsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 118 calls/hour + 2 more in the same half hour |
| high | 3d ago | onboardingsvc: a new error shape, ≥115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us) |
| high | 3d ago | onboardingsvc: a new error shape, ≥8000× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n) |
| critical | 4d ago | execsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 106 calls/hour |
What each rule sent
| Rule | messages | by status |
|---|