workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926
critical
Pod keeps restarting
master
resolved
impact 41
opened 3d ago · last observed 2d ago · observed 5× · #17319 · e50cd05f14c4d47d50f5
What happened template
workplan-agent in tilda on master has restarted in place 13 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 13 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 47 sched events, 12 crash events, 4 probe events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)”. It is running image suresh-eng8690-8981379926. Containers restarting: workplan-agent ×7, istio-proxy ×6. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.
A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; workplan-agent (ns tilda) restarted 14 times in 15 min since 13:40 UTC: 47 unschedulable, 12 crash and 4 probe events, both istio-proxy and workplan-agent containers (last event: BackOff on istio-proxy).
The same image tag suresh-eng8690-8981379926 is failing at the same time on workplansvc (crash loop) and virtual-assistant (unschedulable).
workplansvc logs show a new error since 13:46: otel resource detection cannot reach the metadata server 169.254.169.254 (main.go:64, observability.go:133).
A node went NotReady and cilium-agent logged a foreign-IP error at 13:17, but nothing links either to this pod.
No deploy of these three services shows in the 24 h before. The argocd collector was stale for 224 min, so a rollout of this branch-named image may be missing.
Next step: run kubectl describe on pod workplan-agent-7f4d75b46c-8w7x9 to get the unschedulable reason, and check who rolled out tag suresh-eng8690-8981379926 to master.
low confidence
claude-opus-5-5 · 3d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| workload | workplan-agent |
| namespace | tilda |
| restarts_30m | 0 |
| restarts_1h | 13 |
| restarts_24h | 13 |
| restarts_this_run | 13 |
| run_minutes | 15 |
| since | 1,794,750,000 |
| last_restart_at | 1,794,750,900 |
| pods | 1 |
| containers | {"istio-proxy": 6, "workplan-agent": 7} |
| reasons | {"sched": 47, "crash": 12, "probe": 4} |
| dominant_reason | unschedulable |
| image | us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926 |
| last_event | {"reason": "BackOff", "message": "Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)", "at": 1794751000} |
| share_of_api_calls_24h | 0.00 |
| discriminator | |
| window | 2026-11-15 13:40:00 UTC → 2026-11-15 14:20:01 UTC |
The story — 12 changes landed in the hour before this opened — adjacency, not proof.
- 2026-11-15 13:09:11 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:15 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:20 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:24 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:29 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:33 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:09:38 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:20:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-15 13:28:12 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:28:17 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-12 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 13:45:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (otel resource detection failed — error detecting resource: partial res)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability: failed to initialize tracing — create otel resource: er)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability init failed — observability: init tracing: create otel r)
- 2026-11-15 13:51:32 UTCfinding · critical virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCfinding · critical workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926
- 2026-11-15 13:51:32 UTCthis finding workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926
- 2026-11-15 13:56:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:56:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 14:21:03 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/test-gaps-agent by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:21:08 UTChand change [email protected] changed batch/v1/namespaces/default/jobs/test-gaps-agent-manual-1115-13 by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:30 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:35 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:40 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:44 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-ingester by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:49 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:53 UTChand change [email protected] changed apps/v1/namespaces/default/deployments/x-ray-log-alerter by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:25:58 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-argocd-collector by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 14:26:02 UTChand change [email protected] changed batch/v1/namespaces/default/cronjobs/x-ray-compass by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
History
- 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=workplan-agent on master has restarted 6 times in 5 min — unschedulable, image suresh-eng8690-8981379926 finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 6 times in 5 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 6, 'restarts_1h': 6, 'restarts_24h': 6, 'restarts_this_run': 6, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'istio-proxy': 4, 'workplan-agent': 2}, 'reasons': {'sched': 47, 'crash': 11, 'probe': 3}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'Killing', 'message': 'Container workplan-agent failed liveness probe, will be restarted', 'at': 1794750434}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 23.1, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
- 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=workplan-agent on master has restarted 14 times in 15 min — unschedulable, image suresh-eng8690-8981379926 level=low was_level=low finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 14 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 14, 'restarts_1h': 14, 'restarts_24h': 14, 'restarts_this_run': 14, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'istio-proxy': 7, 'workplan-agent': 7}, 'reasons': {'sched': 47, 'crash': 12, 'probe': 4}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)', 'at': 1794751000}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 40.6, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The workplan-agent workload in the tilda namespace has restarted 6 times since 2026-11-15 13:40 UTC, most recently at 13:45 UTC. Kubernetes recorded 47 scheduling events, 11 crashes, and 3 probe failures. The last event was a liveness probe failure triggering a container restart. The workload runs image suresh-eng8690-8981379926 with istio-proxy (×4) and workplan-agent (×2) containers restarting. It carried 0.0% of cluster API calls in 24 hours. Note: restart counts exclude rollouts and preemptions; crashes count. Resolution occurs after thirty quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
- 2026-11-15 14:40:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17319, 'fingerprint': 'e50cd05f14c4d47d50f5', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplan-agent', 'title': 'workplan-agent on master has restarted 13 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplan-agent', 'namespace': 'tilda', 'restarts_30m': 0, 'restarts_1h': 13, 'restarts_24h': 13, 'restarts_this_run': 13, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'istio-proxy': 6, 'workplan-agent': 7}, 'reasons': {'sched': 47, 'crash': 12, 'probe': 4}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplan-agent:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)', 'at': 1794751000}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794750000, 1794752401], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplan-agent error logs', 'href': '/logs?cluster=master&service=workplan-agent&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 41.1, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752401, 'resolved_at': 1794753640, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "workplan-agent in tilda on master has restarted in place 13 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 13 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 47 sched events, 12 crash events, 4 probe events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplan-agent-7f4d75b46c-8w7x9_tilda(5371afad-548c-4493-99b5-94989841e269)”. It is running image suresh-eng8690-8981379926. Containers restarting: workplan-agent ×7, istio-proxy ×6. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2968, 'stale_s': 1259}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes |
| detector | pod_unstable |