notifsvc on automation has served nothing for 1.2 h; at this hour it normally serves about 78 calls/hour
warning
Service gone silent
automation
resolved
impact 32
Owner: Murali Krishna Velamati (backup: Sambasiva Bandarupalli)
opened 69d ago · last observed 4d ago · observed 497× · #1399 · 48ff53f1a2ee5ea7601f
What happened template
notifsvc on automation has served no API calls for 70 minutes (last call 2026-11-14 23:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 78 calls an hour, 39 in the window checked. It carried 2.3% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.
A service that normally serves requests at this hour has served none for the sustain window — down, or cut off.
The facts
| service | notifsvc |
| silent_minutes | 70 |
| silent_since | 1,794,697,500 |
| last_call_at | 1,794,697,200 |
| expected_calls_in_window | 39.00 |
| expected_per_hour | 78.00 |
| baseline_weeks | 4 |
| buckets_checked | 6 |
| share_of_api_calls_7d | 0.02 |
| discriminator | |
| window | 2026-11-14 23:45:00 UTC → 2026-11-15 00:15:00 UTC |
What is failing in notifsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from notifsvc. In this finding's window (2026-11-14 23:45:00 UTC → 2026-11-15 00:15:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 597× | CreateMessage CreateFromPatientMsg failed — Patient not found : object UnknownSender {deltimestamp:$a_deltimestamp,mob… since 2026-10-31 00:21:43 UTC |
| 232× | could not process HandlePreScreenNotfications event — job not found: objLabel Job: get with empty tildaId requires a str… since 2026-09-15 00:34:02 UTC |
| 2× | ListPolicies query failed — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for dat… since 2026-10-17 19:03:04 UTC |
| 2× | Job completed — could not get policies from the DB: Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get… since 2026-10-05 19:03:35 UTC |
| 2× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-11-02 19:05:06 UTC |
The story — No notifsvc deploy or hand change is recorded in the hour before this opened. Everything below is notifsvc only.
- 2026-09-10 06:23:35 UTCthis finding notifsvc on automation has served nothing for 1.2 h; at this hour it normally serves about 78 calls/hour
- 2026-10-19 19:16:26 UTCfinding · critical notifsvc GET /api/v1/notif/uievents: permission denied — 39 of 39 calls (100 %; 403×39) in the last 15 min — 15× its usual 2.6
- 2026-11-14 18:05:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:51 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:57 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-14 21:11:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1399, 'fingerprint': '48ff53f1a2ee5ea7601f', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'notifsvc', 'title': 'notifsvc on automation has served nothing for 11 h; at this hour it normally serves about 138 calls/hour', 'facts': {'service': 'notifsvc', 'silent_minutes': 675, 'silent_since': 1794585900, 'last_call_at': 1794585600, 'expected_calls_in_window': 69.0, 'expected_per_hour': 138.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.0225, 'recent_deploys': [], 'window': [1794624600, 1794626400], 'discriminator': ''}, 'evidence': [{'label': 'notifsvc overview', 'href': '/service/notifsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'notifsvc logs', 'href': '/logs?cluster=automation&service=notifsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794626883, 'age_seconds': 89, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 51.3, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794626952, 'resolved_at': 1794690719, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 495, 'narrative': "notifsvc on automation has served no API calls for 675 minutes (last call 2026-11-13 16:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 138 calls an hour, 69 in the window checked. It carried 2.2% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794605231, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5669327, 'stale_s': 63790}
- 2026-11-15 00:18:41 UTCreopened x-ray · severity=warning was=critical title=notifsvc on automation has served nothing for 1.1 h; at this hour it normally serves about 104 calls/hour level=medium was_level=critical finding={'id': 1399, 'fingerprint': '48ff53f1a2ee5ea7601f', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'notifsvc', 'title': 'notifsvc on automation has served nothing for 1.1 h; at this hour it normally serves about 104 calls/hour', 'facts': {'service': 'notifsvc', 'silent_minutes': 65, 'silent_since': 1794697500, 'last_call_at': 1794697200, 'expected_calls_in_window': 52.0, 'expected_per_hour': 104.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.0227, 'recent_deploys': [], 'window': [1794699600, 1794701400], 'discriminator': ''}, 'evidence': [{'label': 'notifsvc overview', 'href': '/service/notifsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'notifsvc logs', 'href': '/logs?cluster=automation&service=notifsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794701884, 'age_seconds': 62, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 26.8, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794701921, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 496, 'narrative': "notifsvc on automation has served no API calls for 675 minutes (last call 2026-11-13 16:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 138 calls an hour, 69 in the window checked. It carried 2.2% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794701921, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5680532, 'stale_s': 26}
- 2026-11-15 02:26:17 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1399, 'fingerprint': '48ff53f1a2ee5ea7601f', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'notifsvc', 'title': 'notifsvc on automation has served nothing for 1.2 h; at this hour it normally serves about 78 calls/hour', 'facts': {'service': 'notifsvc', 'silent_minutes': 70, 'silent_since': 1794697500, 'last_call_at': 1794697200, 'expected_calls_in_window': 39.0, 'expected_per_hour': 78.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.0227, 'recent_deploys': [], 'window': [1794699900, 1794701700], 'discriminator': ''}, 'evidence': [{'label': 'notifsvc overview', 'href': '/service/notifsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'notifsvc logs', 'href': '/logs?cluster=automation&service=notifsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794702289, 'age_seconds': 104, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794702354, 'resolved_at': 1794709577, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 497, 'narrative': "notifsvc on automation has served no API calls for 70 minutes (last call 2026-11-14 23:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 78 calls an hour, 39 in the window checked. It carried 2.3% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794701921, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5688186, 'stale_s': 7247}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794702289 |
| age_seconds | 104 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | service_silent |