virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926
critical
Pod keeps restarting
master
resolved
impact 58
opened 2d ago · last observed 2d ago · observed 5× · #17318 · cf20847104b60ddf0e58
What happened template
virtual-assistant in tilda on master has restarted in place 17 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 17 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 27 sched events, 16 crash events, 2 probe events. Last event: FailedScheduling — “0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no ”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, clamd ×7, virtual-assistant ×2. The workload carried 5.6% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.
A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; virtual-assistant (ns tilda, master) restarted 15 times in 15 min from 13:40 UTC. Dominant reason is unschedulable (sched 26, crash 16, probe 2).
Last FailedScheduling: 0/7 nodes available: 5 Insufficient memory, 3 Insufficient cpu, 1 untolerated taint, 1 affinity mismatch. Preemption did not help.
The same branch image suresh-eng8690-8981379926 hit workplan-agent (unschedulable) and workplansvc (crash loop, otel errors reaching 169.254.169.254) at the same time.
None of the 12 deploys checked is this image. The argocd collector was 224 min stale, so the rollout may simply be missing from the bundle.
Next step: find who rolled suresh-eng8690-8981379926 into tilda on master. Compare its CPU/memory requests with free node capacity, then roll back to the previous image or add node capacity.
low confidence
claude-opus-5-5 · 2d ago · $0.06 — every link above is evidence it read; nothing else was claimed.
The facts
| workload | virtual-assistant |
| namespace | tilda |
| restarts_30m | 0 |
| restarts_1h | 17 |
| restarts_24h | 17 |
| restarts_this_run | 17 |
| run_minutes | 15 |
| since | 1,794,750,000 |
| last_restart_at | 1,794,750,900 |
| pods | 1 |
| containers | {"clamd": 7, "istio-proxy": 8, "virtual-assistant": 2} |
| reasons | {"sched": 27, "probe": 2, "crash": 16} |
| dominant_reason | unschedulable |
| image | us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926 |
| last_event | {"reason": "FailedScheduling", "message": "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", "at": 1794751016} |
| share_of_api_calls_24h | 0.06 |
| discriminator | |
| window | 2026-11-15 13:40:00 UTC → 2026-11-15 14:20:01 UTC |
The story — 12 changes landed in the hour before this opened — adjacency, not proof.
- 2026-11-15 13:09:11 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:15 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:20 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:24 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:29 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:33 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:38 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:20:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-15 13:28:12 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:28:17 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-12 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:45:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (otel resource detection failed — error detecting resource: partial res)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability: failed to initialize tracing — create otel resource: er)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability init failed — observability: init tracing: create otel r)
- 2026-11-15 13:51:32 UTCfinding · critical workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCfinding · critical workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCthis finding virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:56:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 14:21:03 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:21:08 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-13 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:30 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:35 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:40 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:44 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:49 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:53 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:58 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:26:02 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
History
- 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=virtual-assistant on master has restarted 7 times in 5 min — unschedulable, image suresh-eng8690-8981379926 finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 7 times in 5 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 7, 'restarts_1h': 7, 'restarts_24h': 7, 'restarts_this_run': 7, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'clamd': 2, 'istio-proxy': 5}, 'reasons': {'sched': 19, 'probe': 2, 'crash': 15}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod virtual-assistant-58bd84fbdc-4fnfx_tilda(9ae12790-07ea-4fc2-9f63-849c6cfe347c)', 'at': 1794750426}, 'share_of_api_calls_24h': 0.0562, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 33.1, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
- 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=virtual-assistant on master has restarted 15 times in 15 min — unschedulable, image suresh-eng8690-8981379926 level=low was_level=low finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 15 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 15, 'restarts_1h': 15, 'restarts_24h': 15, 'restarts_this_run': 15, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'clamd': 7, 'istio-proxy': 6, 'virtual-assistant': 2}, 'reasons': {'sched': 26, 'probe': 2, 'crash': 16}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'FailedScheduling', 'message': "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", 'at': 1794751016}, 'share_of_api_calls_24h': 0.0562, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The virtual-assistant workload in the tilda namespace has restarted 7 times since 2026-11-15 13:40 UTC, including 7 restarts in the last 30 minutes and 7 in 24 hours, most recently at 2026-11-15 13:45 UTC. Kubernetes recorded 19 scheduling events, 15 crash events, and 2 probe events. The last event was a BackOff failure restarting the istio-proxy container in pod virtual-assistant-58bd84fbdc-4fnfx_tilda(9ae12790-07ea-4fc2-9f63-849c6cfe347c). The workload runs image suresh-eng8690-8981379926 with istio-proxy restarting 5 times and clamd 2 times. It handled 5.6% of cluster API calls in the last 24 hours. Restarts are counted from container restart counters only; crashes count, but rollouts and spot preemptions do not. Resolution occurs automatically after 30 quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
- 2026-11-15 14:40:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 0, 'restarts_1h': 17, 'restarts_24h': 17, 'restarts_this_run': 17, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'clamd': 7, 'istio-proxy': 8, 'virtual-assistant': 2}, 'reasons': {'sched': 27, 'probe': 2, 'crash': 16}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'FailedScheduling', 'message': "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", 'at': 1794751016}, 'share_of_api_calls_24h': 0.0559, 'recent_deploys': [], 'window': [1794750000, 1794752401], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 58.5, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752401, 'resolved_at': 1794753640, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "virtual-assistant in tilda on master has restarted in place 17 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 17 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 27 sched events, 16 crash events, 2 probe events. Last event: FailedScheduling — “0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no ”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, clamd ×7, virtual-assistant ×2. The workload carried 5.6% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2968, 'stale_s': 1259}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes |
| detector | pod_unstable |