Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 16:55:05.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926

critical Pod keeps restarting master resolved impact 58 opened 3d ago · last observed 3d ago · observed 5× · #17318 · cf20847104b60ddf0e58

What happened template

virtual-assistant in tilda on master has restarted in place 17 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 17 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 27 sched events, 16 crash events, 2 probe events. Last event: FailedScheduling — “0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no ”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, clamd ×7, virtual-assistant ×2. The workload carried 5.6% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.

A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; virtual-assistant (ns tilda, master) restarted 15 times in 15 min from 13:40 UTC. Dominant reason is unschedulable (sched 26, crash 16, probe 2).
Last FailedScheduling: 0/7 nodes available: 5 Insufficient memory, 3 Insufficient cpu, 1 untolerated taint, 1 affinity mismatch. Preemption did not help.
The same branch image suresh-eng8690-8981379926 hit workplan-agent (unschedulable) and workplansvc (crash loop, otel errors reaching 169.254.169.254) at the same time.
None of the 12 deploys checked is this image. The argocd collector was 224 min stale, so the rollout may simply be missing from the bundle.
Next step: find who rolled suresh-eng8690-8981379926 into tilda on master. Compare its CPU/memory requests with free node capacity, then roll back to the previous image or add node capacity.

low confidence

claude-opus-5-5 · 3d ago · $0.06 — every link above is evidence it read; nothing else was claimed.

The facts

workloadvirtual-assistant
namespacetilda
restarts_30m0
restarts_1h17
restarts_24h17
restarts_this_run17
run_minutes15
since1,794,750,000
last_restart_at1,794,750,900
pods1
containers{"clamd": 7, "istio-proxy": 8, "virtual-assistant": 2}
reasons{"sched": 27, "probe": 2, "crash": 16}
dominant_reasonunschedulable
imageus-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926
last_event{"reason": "FailedScheduling", "message": "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", "at": 1794751016}
share_of_api_calls_24h0.06
discriminator
window2026-11-15 13:40:00 UTC → 2026-11-15 14:20:01 UTC

The story — 12 changes landed in the hour before this opened — adjacency, not proof.

History

  • 2026-11-15 13:51:32 UTCopened x-ray · severity=warning title=virtual-assistant on master has restarted 7 times in 5 min — unschedulable, image suresh-eng8690-8981379926 finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 7 times in 5 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 7, 'restarts_1h': 7, 'restarts_24h': 7, 'restarts_this_run': 7, 'run_minutes': 5, 'since': 1794750000, 'last_restart_at': 1794750300, 'pods': 1, 'containers': {'clamd': 2, 'istio-proxy': 5}, 'reasons': {'sched': 19, 'probe': 2, 'crash': 15}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container istio-proxy in pod virtual-assistant-58bd84fbdc-4fnfx_tilda(9ae12790-07ea-4fc2-9f63-849c6cfe347c)', 'at': 1794750426}, 'share_of_api_calls_24h': 0.0562, 'recent_deploys': [], 'window': [1794750000, 1794750692], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 33.1, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794750692, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 22, 'stale_s': 22}
  • 2026-11-15 13:59:09 UTCescalated x-ray · severity=critical was=warning title=virtual-assistant on master has restarted 15 times in 15 min — unschedulable, image suresh-eng8690-8981379926 level=low was_level=low finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 15 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 15, 'restarts_1h': 15, 'restarts_24h': 15, 'restarts_this_run': 15, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'clamd': 7, 'istio-proxy': 6, 'virtual-assistant': 2}, 'reasons': {'sched': 26, 'probe': 2, 'crash': 16}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'FailedScheduling', 'message': "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", 'at': 1794751016}, 'share_of_api_calls_24h': 0.0562, 'recent_deploys': [], 'window': [1794750000, 1794751149], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794750692, 'last_seen': 1794751149, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The virtual-assistant workload in the tilda namespace has restarted 7 times since 2026-11-15 13:40 UTC, including 7 restarts in the last 30 minutes and 7 in 24 hours, most recently at 2026-11-15 13:45 UTC. Kubernetes recorded 19 scheduling events, 15 crash events, and 2 probe events. The last event was a BackOff failure restarting the istio-proxy container in pod virtual-assistant-58bd84fbdc-4fnfx_tilda(9ae12790-07ea-4fc2-9f63-849c6cfe347c). The workload runs image suresh-eng8690-8981379926 with istio-proxy restarting 5 times and clamd 2 times. It handled 5.6% of cluster API calls in the last 24 hours. Restarts are counted from container restart counters only; crashes count, but rollouts and spot preemptions do not. Resolution occurs automatically after 30 quiet minutes.', 'narrative_kind': 'ai', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 527, 'stale_s': 70}
  • 2026-11-15 14:40:40 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17318, 'fingerprint': 'cf20847104b60ddf0e58', 'kind': 'pod.unstable', 'severity': 'critical', 'cluster': 'master', 'subject': 'tilda/virtual-assistant', 'title': 'virtual-assistant on master has restarted 17 times in 15 min — unschedulable, image suresh-eng8690-8981379926', 'facts': {'workload': 'virtual-assistant', 'namespace': 'tilda', 'restarts_30m': 0, 'restarts_1h': 17, 'restarts_24h': 17, 'restarts_this_run': 17, 'run_minutes': 15, 'since': 1794750000, 'last_restart_at': 1794750900, 'pods': 1, 'containers': {'clamd': 7, 'istio-proxy': 8, 'virtual-assistant': 2}, 'reasons': {'sched': 27, 'probe': 2, 'crash': 16}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/virtual-assistant:suresh-eng8690-8981379926', 'last_event': {'reason': 'FailedScheduling', 'message': "0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no new claims to deallocate, preemption: 0/7 nodes are available: 2 Preemption is not helpful for scheduling, 5 No preemption victims found for", 'at': 1794751016}, 'share_of_api_calls_24h': 0.0559, 'recent_deploys': [], 'window': [1794750000, 1794752401], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=master&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=master'}, {'label': 'virtual-assistant error logs', 'href': '/logs?cluster=master&service=virtual-assistant&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 58.5, 'status': 'resolved', 'first_seen': 1794750692, 'last_seen': 1794752401, 'resolved_at': 1794753640, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "virtual-assistant in tilda on master has restarted in place 17 times since 2026-11-15 13:40 UTC (0 in the last 30 minutes, 17 in 24 hours), most recently at 2026-11-15 13:55 UTC. Kubernetes' account of it: 27 sched events, 16 crash events, 2 probe events. Last event: FailedScheduling — “0/7 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 5 Insufficient memory. no ”. It is running image suresh-eng8690-8981379926. Containers restarting: istio-proxy ×8, clamd ×7, virtual-assistant ×2. The workload carried 5.6% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794750692, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 2968, 'stale_s': 1259}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
notepod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes
detectorpod_unstable