execsvc on automation has served nothing for 2.5 h; at this hour it normally serves about 60 calls/hour
critical
Service gone silent
automation
resolved
impact 53
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 70d ago · last observed 3d ago · observed 821× · #1398 · 31fa164cbba1fd4a23b7
What happened template
execsvc on automation has served no API calls for 150 minutes (last call 2026-11-15 23:00 UTC); at this hour of the week its baseline — 5 weeks of history — expects about 60 calls an hour, 30 in the window checked. It carried 17.5% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.
A service that normally serves requests at this hour has served none for the sustain window — down, or cut off.
The facts
| service | execsvc |
| silent_minutes | 150 |
| silent_since | 1,794,783,900 |
| last_call_at | 1,794,783,600 |
| expected_calls_in_window | 30.00 |
| expected_per_hour | 60.00 |
| baseline_weeks | 5 |
| buckets_checked | 6 |
| share_of_api_calls_7d | 0.18 |
| discriminator | |
| window | 2026-11-16 01:05:00 UTC → 2026-11-16 01:35:00 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 01:05:00 UTC → 2026-11-16 01:35:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-09-10 06:23:35 UTCthis finding execsvc on automation has served nothing for 2.5 h; at this hour it normally serves about 60 calls/hour
- 2026-09-11 21:11:41 UTCfinding · critical execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min
- 2026-10-31 04:53:55 UTCfinding · warning execsvc GET /api/trialexec/study/:id/actor: 4xx — 21 of 60 calls (35 %; other 4xx×21) in the last 15 min — 39× its usual 0.5
- 2026-11-10 14:33:32 UTCfinding · info execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)
- 2026-11-14 14:16:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:17:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:28:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:29:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:35:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:36:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:41:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:41:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:04:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:08:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:09:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:18:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:18:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:46:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:47:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:52:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:53:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 00:00:00 UTCerrors execsvc 5xx peaked at 32 in one five-minute bucket (median 0 in this window)
- 2026-11-15 07:16:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:28:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:43:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:59:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-14 11:17:38 UTCreopened x-ray · severity=critical was=critical title=execsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 106 calls/hour level=critical was_level=critical finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 2.1 h; at this hour it normally serves about 106 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 125, 'silent_since': 1794647100, 'last_call_at': 1794646800, 'expected_calls_in_window': 53.0, 'expected_per_hour': 106.0, 'baseline_weeks': 5, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1826, 'recent_deploys': [], 'window': [1794652800, 1794654600], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794655030, 'age_seconds': 50, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 48.5, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794655058, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 795, 'narrative': "execsvc on automation has served no API calls for 135 minutes (last call 2026-11-14 06:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 1,024 calls an hour, 512 in the window checked. It carried 18.5% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794655058, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5633665, 'stale_s': 22}
- 2026-11-14 13:24:15 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 4 h; at this hour it normally serves about 1,350 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 235, 'silent_since': 1794647100, 'last_call_at': 1794646800, 'expected_calls_in_window': 675.0, 'expected_per_hour': 1350.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1797, 'recent_deploys': [], 'window': [1794659400, 1794661200], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794661638, 'age_seconds': 113, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.4, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794661730, 'resolved_at': 1794662655, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 811, 'narrative': "execsvc on automation has served no API calls for 235 minutes (last call 2026-11-14 09:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 1,350 calls an hour, 675 in the window checked. It carried 18.0% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794655058, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5641264, 'stale_s': 949}
- 2026-11-14 19:26:42 UTCreopened x-ray · severity=critical was=critical title=execsvc on automation has served nothing for 5 h; at this hour it normally serves about 84 calls/hour level=critical was_level=critical finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 5 h; at this hour it normally serves about 84 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 310, 'silent_since': 1794665100, 'last_call_at': 1794664800, 'expected_calls_in_window': 42.0, 'expected_per_hour': 84.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1773, 'recent_deploys': [], 'window': [1794681900, 1794683700], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794684252, 'age_seconds': 172, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 48.4, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794684402, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 812, 'narrative': "execsvc on automation has served no API calls for 235 minutes (last call 2026-11-14 09:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 1,350 calls an hour, 675 in the window checked. It carried 18.0% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794684402, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5663009, 'stale_s': 22}
- 2026-11-14 19:44:51 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 5 h; at this hour it normally serves about 84 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 310, 'silent_since': 1794665100, 'last_call_at': 1794664800, 'expected_calls_in_window': 42.0, 'expected_per_hour': 84.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1773, 'recent_deploys': [], 'window': [1794681900, 1794683700], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794684252, 'age_seconds': 172, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.4, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794684402, 'resolved_at': 1794685491, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 812, 'narrative': "execsvc on automation has served no API calls for 310 minutes (last call 2026-11-14 14:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 84 calls an hour, 42 in the window checked. It carried 17.7% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794684402, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5664097, 'stale_s': 1110}
- 2026-11-14 20:41:47 UTCreopened x-ray · severity=warning was=critical title=execsvc on automation has served nothing for 30 min; at this hour it normally serves about 78 calls/hour level=medium was_level=critical finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 30 min; at this hour it normally serves about 78 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 30, 'silent_since': 1794686700, 'last_call_at': 1794686400, 'expected_calls_in_window': 39.0, 'expected_per_hour': 78.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1772, 'recent_deploys': [], 'window': [1794686700, 1794688500], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794688848, 'age_seconds': 114, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.9, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794688907, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 813, 'narrative': "execsvc on automation has served no API calls for 310 minutes (last call 2026-11-14 14:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 84 calls an hour, 42 in the window checked. It carried 17.7% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794688907, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5667546, 'stale_s': 54}
- 2026-11-14 21:03:50 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 30 min; at this hour it normally serves about 78 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 30, 'silent_since': 1794686700, 'last_call_at': 1794686400, 'expected_calls_in_window': 39.0, 'expected_per_hour': 78.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1772, 'recent_deploys': [], 'window': [1794686700, 1794688500], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794688848, 'age_seconds': 114, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 33.9, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794688907, 'resolved_at': 1794690230, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 813, 'narrative': "execsvc on automation has served no API calls for 30 minutes (last call 2026-11-14 20:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 78 calls an hour, 39 in the window checked. It carried 17.7% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794688907, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5668837, 'stale_s': 1345}
- 2026-11-15 00:18:41 UTCreopened x-ray · severity=warning was=warning title=execsvc on automation has served nothing for 1.1 h; at this hour it normally serves about 2,510 calls/hour level=medium was_level=medium finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 1.1 h; at this hour it normally serves about 2,510 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 65, 'silent_since': 1794697500, 'last_call_at': 1794697200, 'expected_calls_in_window': 1255.0, 'expected_per_hour': 2510.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1774, 'recent_deploys': [], 'window': [1794699600, 1794701400], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794701884, 'age_seconds': 63, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.9, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794701921, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 814, 'narrative': "execsvc on automation has served no API calls for 30 minutes (last call 2026-11-14 20:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 78 calls an hour, 39 in the window checked. It carried 17.7% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794701921, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5680532, 'stale_s': 26}
- 2026-11-15 00:49:50 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 1.3 h; at this hour it normally serves about 767 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 80, 'silent_since': 1794697500, 'last_call_at': 1794697200, 'expected_calls_in_window': 383.5, 'expected_per_hour': 767.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1775, 'recent_deploys': [], 'window': [1794700500, 1794702300], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794702763, 'age_seconds': 87, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 33.9, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794702828, 'resolved_at': 1794703790, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 816, 'narrative': "execsvc on automation has served no API calls for 80 minutes (last call 2026-11-14 23:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 767 calls an hour, 384 in the window checked. It carried 17.8% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794701921, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5682399, 'stale_s': 986}
- 2026-11-15 05:22:28 UTCreopened x-ray · severity=warning was=warning title=execsvc on automation has served nothing for 1.2 h; at this hour it normally serves about 291 calls/hour level=medium was_level=medium finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 1.2 h; at this hour it normally serves about 291 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 70, 'silent_since': 1794715500, 'last_call_at': 1794715200, 'expected_calls_in_window': 145.5, 'expected_per_hour': 291.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1778, 'recent_deploys': [], 'window': [1794717900, 1794719700], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794720125, 'age_seconds': 42, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 28.9, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794720148, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 817, 'narrative': "execsvc on automation has served no API calls for 80 minutes (last call 2026-11-14 23:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 767 calls an hour, 384 in the window checked. It carried 17.8% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794720148, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5698752, 'stale_s': 19}
- 2026-11-15 21:16:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 1.3 h; at this hour it normally serves about 97 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 80, 'silent_since': 1794715500, 'last_call_at': 1794715200, 'expected_calls_in_window': 48.5, 'expected_per_hour': 97.0, 'baseline_weeks': 4, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1778, 'recent_deploys': [], 'window': [1794718500, 1794720300], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794720869, 'age_seconds': 94, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 33.9, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794720896, 'resolved_at': 1794777405, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 819, 'narrative': "execsvc on automation has served no API calls for 80 minutes (last call 2026-11-15 04:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 97 calls an hour, 48 in the window checked. It carried 17.8% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794720148, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5756011, 'stale_s': 56530}
- 2026-11-16 01:36:14 UTCreopened x-ray · severity=critical was=warning title=execsvc on automation has served nothing for 2.4 h; at this hour it normally serves about 60 calls/hour level=critical was_level=medium finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 2.4 h; at this hour it normally serves about 60 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 145, 'silent_since': 1794783900, 'last_call_at': 1794783600, 'expected_calls_in_window': 30.0, 'expected_per_hour': 60.0, 'baseline_weeks': 5, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1751, 'recent_deploys': [], 'window': [1794790800, 1794792600], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794792930, 'age_seconds': 93, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 48.3, 'status': 'open', 'first_seen': 1789021415, 'last_seen': 1794792974, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 820, 'narrative': "execsvc on automation has served no API calls for 80 minutes (last call 2026-11-15 04:00 UTC); at this hour of the week its baseline — 4 weeks of history — expects about 97 calls an hour, 48 in the window checked. It carried 17.8% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5771608, 'stale_s': 49}
- 2026-11-16 02:02:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1398, 'fingerprint': '31fa164cbba1fd4a23b7', 'kind': 'service.silent', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc on automation has served nothing for 2.5 h; at this hour it normally serves about 60 calls/hour', 'facts': {'service': 'execsvc', 'silent_minutes': 150, 'silent_since': 1794783900, 'last_call_at': 1794783600, 'expected_calls_in_window': 30.0, 'expected_per_hour': 60.0, 'baseline_weeks': 5, 'buckets_checked': 6, 'share_of_api_calls_7d': 0.1751, 'recent_deploys': [], 'window': [1794791100, 1794792900], 'discriminator': ''}, 'evidence': [{'label': 'execsvc overview', 'href': '/service/execsvc?cluster=automation&range=24h'}, {'label': 'Reliability', 'href': '/reliability?cluster=automation&range=24h'}, {'label': 'execsvc logs', 'href': '/logs?cluster=automation&service=execsvc&range=24h'}], 'provenance': {'source': 'rollup', 'collected_at': 1794793383, 'age_seconds': 69, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.3, 'status': 'resolved', 'first_seen': 1789021415, 'last_seen': 1794793430, 'resolved_at': 1794794547, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 821, 'narrative': "execsvc on automation has served no API calls for 150 minutes (last call 2026-11-15 23:00 UTC); at this hour of the week its baseline — 5 weeks of history — expects about 60 calls an hour, 30 in the window checked. It carried 17.5% of the cluster's calls over the last week. Silence is not measured downtime: the pods may be up and unused, or crashed, or cut off upstream — Reliability and Pods say which. It resolves on its own once the service serves again.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'critical', 'told_level': None, 'episode_at': 1794792974, 'chat_thread': None, 'kind_label': 'Service gone silent', 'group': 'Reliability', 'age_s': 5773156, 'stale_s': 1141}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794793383 |
| age_seconds | 69 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | service_silent |