Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 12:18:21.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Noise on production, last 7 days

How loud each level is against what it is allowed to be. Nobody reads a channel that cries wolf. ?Replayed from this cluster's finding events exactly as the level rules apply them: each event's level at its own moment, a reopen within a day at no higher level is the same incident, an escalation inside a level is not news. critical and high that land within 30 minutes of each other are one incident — one message.

Dry-run. Messages are composed and logged under Deliveries; nothing external is sent unless a channel was armed on its own.
critical
0.0/week
0 in 7 d · budget ≤ 1/week
high
9.0/week
9 in 7 d · budget ≤ 5/week
endpoint.4xx_spike 3
job.failing 2
log.pattern 1
pod.cannot_start 1
error.novel 1
endpoint.failing 1
medium
16.0/week
16 in 7 d · budget grouped
job.failing 4
scaler.failing 3
log.pattern 2
error.novel 2
budget.burning 2
node.unhealthy 1
pod.cannot_start 1
service.silent 1
low
25.0/week
25 in 7 d · budget digest
anomaly.neo4j 10
anomaly.traffic 7
anomaly.latency 4
n1.fanout 3
budget.burning 1

critical + high as messages (30-minute correlation): 7.0/week · medium as hourly messages: 13 in 7 d · damped as the same incident: 103

critical + high incidents 7

high1d ago execsvc.auto-query-reeval: 20 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
high1d ago pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested + 1 more in the same half hour
high1d ago execsvc: a new error shape, 203× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
high2d ago docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 63 calls (44 %; 403×28) in the last 15 min — it usually has none + 1 more in the same half hour
high2d ago virtual-assistant GET /api/v1/va/sse/stream: permission denied — 20 of 22 calls (91 %; 401×20) in the last 15 min — 232× its usual 0.1
high2d ago x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod
high3d ago ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal

What each rule sent

Rulemessagesby status