Noise on production, last 7 days
How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.
critical
0.0/week
0 in 7 d · budget ≤ 1/week
high
9.0/week
9 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 3
job.failing 2
log.pattern 1
pod.cannot_start 1
error.novel 1
endpoint.failing 1
medium
16.0/week
16 in 7 d · budget grouped
job.failing 4
scaler.failing 3
log.pattern 2
error.novel 2
budget.burning 2
node.unhealthy 1
pod.cannot_start 1
service.silent 1
low
25.0/week
25 in 7 d · budget digest
anomaly.neo4j 10
anomaly.traffic 7
anomaly.latency 4
n1.fanout 3
budget.burning 1
critical + high as messages (30-minute correlation): 7.0/week · medium as hourly messages: 13 in 7 d · damped as the same incident: 103
critical + high incidents 7
| high | 1d ago | execsvc.auto-query-reeval: 20 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=? |
| high | 1d ago | pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested + 1 more in the same half hour |
| high | 1d ago | execsvc: a new error shape, 203× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no ) |
| high | 2d ago | docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none + 1 more in the same half hour |
| high | 2d ago | virtual-assistant GET /api/v1/va/sse/stream: permission denied — 20 of 22 calls (91 %; 401×20) in the last 15 min — 232× its usual 0.1 |
| high | 2d ago | x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod |
| high | 3d ago | ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal |
What each rule sent
| Rule | messages | by status |
|---|