Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 21:07:15.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

schema-migrate (tilda) image pull failing for at least 15 min — 1 pod

warning Pod cannot start delta open impact 43 opened 2d ago · last observed 1d ago · observed 122× · #17384 · a19335c7980481365e7c

What happened template

Pod cannot start: schema-migrate (tilda) image pull failing for at least 15 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 15, since 1794911087.

A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; schema-migrate pod schema-migrate-567b794477-528pq in tilda (cluster delta) has been in ImagePullBackOff since 2026-11-16 09:06 UTC, 65 min pending, 56 events.
Checked: 12 deploys 234 min before onset (config 6db9855a7d34 by [email protected], none touch schema-migrate), 1383 cluster error lines (cilium/metrics-server noise, nothing about image pulls), no code for this workload.
This recurs: same finding resolved on 10-31, 10-27 and 10-22 after roughly 2 h each, which points at a recurring migrate job referencing an image that is missing or not pullable.
Next step: read the Kubernetes events for schema-migrate to get the exact image reference and pull error (not found vs auth), then check that tag exists in the registry.

low confidence

claude-opus-5-5 · 2d ago · $0.04 — every link above is evidence it read; nothing else was claimed.

The facts

namespacetilda
workloadschema-migrate
whyimage pull failing
podsschema-migrate-567b794477-fswhb
pending_minutes15
since1,794,911,087
events38
last_messageError: ImagePullBackOff
discriminatorimage pull failing

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-16 09:06:49 UTCopened x-ray · severity=warning title=schema-migrate (tilda) image pull failing for at least 22 min — 1 pod finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 22 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-528pq'], 'pending_minutes': 22, 'since': 1794818681, 'events': 38, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794816409&to=1794820009&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 38.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794820009, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 33, 'stale_s': 33}
  • 2026-11-16 09:50:12 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 65 min — 1 pod level=high was_level=medium finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 65 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-528pq'], 'pending_minutes': 65, 'since': 1794818681, 'events': 56, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794819012&to=1794822612&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794822612, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The pod schema-migrate-567b794477-528pq in namespace tilda cannot start due to an image pull failure lasting at least 22 minutes. The workload schema-migrate has 1 pending pod since 1794818681, with 38 events recorded. Last error message: ImagePullBackOff.', 'narrative_kind': 'ai', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 2647, 'stale_s': 44}
  • 2026-11-16 12:29:31 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 43 min — 1 pod level=high was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 43 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-5qrkv'], 'pending_minutes': 43, 'since': 1794829568, 'events': 47, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794828571&to=1794832171&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794832171, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 20 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 20, since 1794829568.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 12249, 'stale_s': 87}
  • 2026-11-16 15:08:36 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 37 min — 1 pod level=high was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 37 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-4ffbz'], 'pending_minutes': 37, 'since': 1794839451, 'events': 45, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794838116&to=1794841716&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794841716, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 27 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 27, since 1794839451.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 21732, 'stale_s': 25}
  • 2026-11-16 18:39:47 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 14 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-xkrbk'], 'pending_minutes': 14, 'since': 1794852619, 'events': 35, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794849867&to=1794853467&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 43.2, 'status': 'resolved', 'first_seen': 1794820009, 'last_seen': 1794853467, 'resolved_at': 1794854387, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 39, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 14 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 14, since 1794852619.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 34401, 'stale_s': 943}
  • 2026-11-16 18:47:25 UTCreopened x-ray · severity=warning was=warning title=schema-migrate (tilda) image pull failing for at least 16 min — 1 pod level=medium was_level=high finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 16 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-ztfj2'], 'pending_minutes': 16, 'since': 1794853856, 'events': 37, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794851245&to=1794854845&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 38.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794854845, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 40, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 14 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 14, since 1794852619.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794854845, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 34870, 'stale_s': 34}
  • 2026-11-16 19:04:04 UTCescalated x-ray · severity=critical was=warning title=schema-migrate (tilda) image pull failing for at least 33 min — 1 pod level=high was_level=medium finding={'id': 17384, 'fingerprint': 'a19335c7980481365e7c', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'delta', 'subject': 'tilda/schema-migrate', 'title': 'schema-migrate (tilda) image pull failing for at least 33 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'schema-migrate', 'why': 'image pull failing', 'pods': ['schema-migrate-567b794477-ztfj2'], 'pending_minutes': 33, 'since': 1794853856, 'events': 43, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794852244&to=1794855844&source=events&q=schema-migrate'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 65.2, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794855844, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 42, 'narrative': 'Pod cannot start: schema-migrate (tilda) image pull failing for at least 24 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 24, since 1794853856.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794854845, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 35902, 'stale_s': 67}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events