Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 13:57:34.
› →|
updated 2d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod

critical Pod cannot start production resolved impact 78 opened 11d ago · last observed 2d ago · observed 142× · #16883 · 9efadf270c28e8edc5d7

What happened template

Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794884707.

A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; checked error lines (none about this image), 12 deploys in the 24 h before (all config rev 6db9855a7d34 to other apps, none to x-ray), this finding's history and 2 findings at the same time.
Pod x-ray-weaviate-stats-29914720-w48vb (namespace x-ray) cannot pull us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea; pending 32 min with 53 events since 02:54 UTC.
This recurs daily: episodes open around 02:50 UTC and clear around 05:20 UTC (Nov 11, 12, 13, 14, 16); 134 occurrences since Nov 8. The schedule points to a CronJob run.
A node.unhealthy finding (NodeNotReady) opened at the same minute but is not linked to this pull.
Next step: read the Kubernetes events for the actual pull error (not found vs. permission denied), then confirm tag 3.7.1-5aab6ea exists in the tilda-dev-325721 registry and the production nodes can read it.

low confidence

claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

namespacex-ray
workloadx-ray-weaviate-stats
whyimage pull failing
podsx-ray-weaviate-stats-29914720-w48vb
pending_minutes115
since1,794,884,707
events2
last_messageError: ImagePullBackOff
discriminatorimage pull failing

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-16 02:51:52 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 11 min — 1 pod level=medium was_level=high finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 11 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 11, 'since': 1794796801, 'events': 20, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794793912&to=1794797512&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794797512, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 116, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 117 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 117, since 1794625510.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 690952, 'stale_s': 19}
  • 2026-11-16 03:12:35 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod level=high was_level=medium finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 32, 'since': 1794796801, 'events': 53, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794795155&to=1794798755&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794798755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 118, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 19 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 19, since 1794796801.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'medium', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 692214, 'stale_s': 38}
  • 2026-11-16 05:18:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913280-wnlv9'], 'pending_minutes': 119, 'since': 1794798006, 'events': 4, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794801556&to=1794805156&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 77.5, 'status': 'resolved', 'first_seen': 1794106579, 'last_seen': 1794805156, 'resolved_at': 1794806332, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 131, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794798006.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794797512, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 699790, 'stale_s': 1213}
  • 2026-11-17 02:54:44 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 14 min — 1 pod level=medium was_level=high finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 14 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 14, 'since': 1794883201, 'events': 43, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794880484&to=1794884084&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794884084, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 132, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794798006.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 777531, 'stale_s': 26}
  • 2026-11-17 03:12:58 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod level=high was_level=medium finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 32, 'since': 1794883201, 'events': 53, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5aab6ea"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794881578&to=1794885178&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794106579, 'last_seen': 1794885178, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 134, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 23 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 23, since 1794883201.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 778621, 'stale_s': 22}
  • 2026-11-17 05:17:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16883, 'fingerprint': '9efadf270c28e8edc5d7', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'production', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914720-w48vb'], 'pending_minutes': 115, 'since': 1794884707, 'events': 2, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794888047&to=1794891647&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 77.5, 'status': 'resolved', 'first_seen': 1794106579, 'last_seen': 1794891647, 'resolved_at': 1794892665, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 142, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794884707.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': 'high', 'episode_at': 1794884084, 'chat_thread': 'xray-alert-10-748fd5a928-9efadf270c28e8edc5d7-1794113635', 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 786105, 'stale_s': 1037}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events