schema-migrate (tilda) image pull failing for at least 15 min — 1 pod
warning
Pod cannot start
delta
open
impact 43
opened 2d ago · last observed 1d ago · observed 122× · #17384 · a19335c7980481365e7c
What happened template
Pod cannot start: schema-migrate (tilda) image pull failing for at least 15 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 15, since 1794911087.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; schema-migrate pod schema-migrate-567b794477-528pq in tilda (cluster delta) has been in ImagePullBackOff since 2026-11-16 09:06 UTC, 65 min pending, 56 events.
Checked: 12 deploys 234 min before onset (config 6db9855a7d34 by [email protected], none touch schema-migrate), 1383 cluster error lines (cilium/metrics-server noise, nothing about image pulls), no code for this workload.
This recurs: same finding resolved on 10-31, 10-27 and 10-22 after roughly 2 h each, which points at a recurring migrate job referencing an image that is missing or not pullable.
Next step: read the Kubernetes events for schema-migrate to get the exact image reference and pull error (not found vs auth), then check that tag exists in the registry.
low confidence
claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| namespace | tilda |
| workload | schema-migrate |
| why | image pull failing |
| pods | schema-migrate-567b794477-fswhb |
| pending_minutes | 15 |
| since | 1,794,911,087 |
| events | 38 |
| last_message | Error: ImagePullBackOff |
| discriminator | image pull failing |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-16 08:44:24 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-hlwc (1× in 30 min)
- 2026-11-16 08:53:28 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-p8mj (1× in 30 min)
- 2026-11-16 09:06:49 UTCthis finding schema-migrate (tilda) image pull failing for at least 15 min — 1 pod
- 2026-11-16 12:06:52 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-8z2l (1× in 30 min)
- 2026-11-16 12:58:26 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-4cxt (1× in 30 min)
- 2026-11-16 14:26:30 UTCfinding · critical execsvc: a new error shape, 247× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
- 2026-11-16 14:26:30 UTCfinding · warning execsvc: a new error shape, 46× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — eva)
- 2026-11-16 14:26:30 UTCfinding · warning execsvc: a new error shape, 46× in the last hour, never seen in 90 days (evalResult is not a bool)
- 2026-11-16 14:38:42 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-tchw (1× in 30 min)
- 2026-11-16 16:55:00 UTClatency cluster p95 peaked at 7.14 s
- 2026-11-17 10:10:00 UTCerrors cluster 5xx peaked at 26 in one five-minute bucket (median 0 in this window)
History
- 2026-11-16 09:06:49 UTCopened x-ray · severity=warning title=schema-migrate (tilda) image pull failing for at least 22 min — 1 pod finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 22 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-528pq'], 'pending_minutes': 22, 'since': 1794818681, 'events': 38, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794816409&to=1794820009&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 38.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794820009, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 33, 'stale_s': 33}
- 2026-11-16 09:50:12 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 65 min — 1 pod level=high was_level=medium finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 65 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-528pq'], 'pending_minutes': 65, 'since': 1794818681, 'events': 56, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794819012&to=1794822612&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794822612, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The pod schema-migrate-567b794477-528pq in namespace tilda cannot start due to an image pull failure lasting at least 22 minutes. The workload schema-migrate has 1 pending pod since 1794818681, with 38 events recorded. Last error message: ImagePullBackOff.', 'narrative_kind': 'ai', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 2647, 'stale_s': 44}
- 2026-11-16 12:29:31 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 43 min — 1 pod level=high was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 43 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-5qrkv'], 'pending_minutes': 43, 'since': 1794829568, 'events': 47, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794828571&to=1794832171&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794832171, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 20 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 20, since 1794829568.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 12249, 'stale_s': 87}
- 2026-11-16 15:08:36 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 37 min — 1 pod level=high was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 37 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-4ffbz'], 'pending_minutes': 37, 'since': 1794839451, 'events': 45, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794838116&to=1794841716&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794841716, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 27 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 27, since 1794839451.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 21732, 'stale_s': 25}
- 2026-11-16 18:39:47 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 14 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-xkrbk'], 'pending_minutes': 14, 'since': 1794852619, 'events': 35, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794849867&to=1794853467&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 43.2, 'status': 'resolved', 'first_seen': 1794820009, 'last_seen': 1794853467, 'resolved_at': 1794854387, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 39, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 14 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 14, since 1794852619.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 34401, 'stale_s': 943}
- 2026-11-16 18:47:25 UTCreopened x-ray · severity=warning was=warning title=schema-migrate (tilda) image pull failing for at least 16 min — 1 pod level=medium was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 16 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-ztfj2'], 'pending_minutes': 16, 'since': 1794853856, 'events': 37, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794851245&to=1794854845&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 38.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794854845, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 40, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 14 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 14, since 1794852619.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794854845, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 34870, 'stale_s': 34}
- 2026-11-16 19:04:04 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 33 min — 1 pod level=high was_level=medium finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 33 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-ztfj2'], 'pending_minutes': 33, 'since': 1794853856, 'events': 43, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794852244&to=1794855844&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794855844, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 42, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 24 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 24, since 1794853856.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794854845, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 35902, 'stale_s': 67}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |