Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:26:09.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926

critical Pod keeps restarting master resolved impact 65 opened 2d ago · last observed 2d ago · observed 4× · #17317 · b807db4266bc4a7340a1

What happened template

workplansvc in tilda on master has restarted in place 15 times since 2026-11-15 13:40 UTC (8 in the last 30 minutes, 15 in 24 hours), most recently at 2026-11-15 13:50 UTC. Kubernetes' account of it: 27 crash events, 17 sched events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, workplansvc ×7. The workload carried 42.9% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.

First look — investigated before anyone looked

workplansvc (ns tilda) on master is crash-looping: 14 restarts in 15 min on 1 pod (workplansvc 7, istio-proxy 7), image suresh-eng8690-8981379926.
Started 13:40 UTC. From 13:46 workplansvc/main.go:64 exits on 'observability init failed': otel resource detection cannot reach the GCE metadata server (dial tcp 169.254.169.254, from observability/observability.go:133), 6x each, never seen in 90 days.
The istio-proxy sidecar is also in BackOff. A node went NotReady and cilium logged foreign-IP errors at 13:17, so pod networking to the metadata server is the likely break.
No workplansvc deploy is in the 24 h list. The same image runs on workplan-agent and virtual-assistant, which are also restarting (unschedulable).
Next: check istio-proxy logs and egress to 169.254.169.254 from the pod. Make tracing init non-fatal at main.go:64 so the service starts without otel.

medium confidence dependency GCE metadata server 169.254.169.254 unreachable

claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

workloadworkplansvc
namespacetilda
restarts_30m8
restarts_1h15
restarts_24h15
restarts_this_run15
run_minutes10
since1,794,750,000
last_restart_at1,794,750,600
pods1
containers{"istio-proxy": 8, "workplansvc": 7}
reasons{"sched": 17, "crash": 27}
dominant_reasoncrash loop
imageus-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926
last_event{"reason": "BackOff", "message": "Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)", "at": 1794750986}
share_of_api_calls_24h0.43
discriminator
window2026-11-15 13:40:00 UTC → 2026-11-15 14:13:43 UTC

The story — 12 changes landed in the hour before this opened — adjacency, not proof.

History

  • 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=workplansvc on master has restarted 8 times in 5 min — crash loop, image suresh-eng8690-8981379926 finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 8 times in 5 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 8, 'restarts_1h': 8, 'restarts_24h': 8, 'restarts_this_run': 8, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'istio-proxy': 4, 'workplansvc': 4}, 'reasons': {'sched': 16, 'crash': 25}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750430}, 'share_of_api_calls_24h': 0.4223, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 40.4, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
  • 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926 level=high was_level=medium finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 14 times in 15 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 14, 'restarts_1h': 14, 'restarts_24h': 14, 'restarts_this_run': 14, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'workplansvc': 7, 'istio-proxy': 7}, 'reasons': {'sched': 17, 'crash': 27}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750986}, 'share_of_api_calls_24h': 0.422, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 64.4, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The workplansvc workload in the tilda namespace has restarted 8 times since 2026-11-15 13:40 UTC, most recently at 13:45 UTC. Kubernetes logs 25 crash events and 16 scheduling events. The last event shows BackOff restarting the failed istio-proxy container in pod workplansvc-6787748cc8-nvgzf_tilda (911901ce-cd4e-43ab-ab4f-d47978721f92). It runs image suresh-eng8690-8981379926 with istio-proxy ×4 and workplansvc ×4 containers restarting. This workload carried 42.2% of cluster API calls in 24 hours. Note: restarts are counted from container restart counters only, excluding rollouts or spot preemptions, and resolve automatically after thirty quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'high', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
  • 2026-11-15 14:33:42 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17317, 'fingerprint': 'b807db4266bc4a7340a1', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/workplansvc', 'title': 'workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926', 'facts': {'workload': 'workplansvc', 'namespace': 'tilda', 'restarts_30m': 8, 'restarts_1h': 15, 'restarts_24h': 15, 'restarts_this_run': 15, 'run_minutes': 10, 'since': 1794750000, 'last_restart_at': 1794750600, 'pods': 1, 'containers': {'istio-proxy': 8, 'workplansvc': 7}, 'reasons': {'sched': 17, 'crash': 27}, 'dominant_reason': 'crash loop', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/workplansvc:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)', 'at': 1794750986}, 'share_of_api_calls_24h': 0.4286, 'recent_deploys': [], 'window': [1794750000, 1794752023], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'workplansvc error logs', 'href': '/logs?cluster=master&service=workplansvc&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 65.0, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752023, 'resolved_at': 1794753222, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': "workplansvc in tilda on master has restarted in place 15 times since 2026-11-15 13:40 UTC (8 in the last 30 minutes, 15 in 24 hours), most recently at 2026-11-15 13:50 UTC. Kubernetes' account of it: 27 crash events, 17 sched events. Last event: BackOff — “Back-off restarting failed container istio-proxy in pod workplansvc-6787748cc8-nvgzf_tilda(911901ce-cd4e-43ab-ab4f-d47978721f92)”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, workplansvc ×7. The workload carried 42.9% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'high', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2549, 'stale_s': 1218}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
notepod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes
detectorpod_unstable