Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:42:59.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod

critical Pod cannot start master resolved impact 63 opened 10d ago · last observed 1d ago · observed 176× · #16882 · c22a8ba76e38fb9cf885

What happened template

Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794883816.

A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; CronJob pod x-ray-weaviate-stats-29914690-fpppt (x-ray) has been in ImagePullBackOff for 34 min (49 events) since 2026-11-17 02:23 UTC.
The same failure has recurred every night since 2026-11-08, opening around 02:23–02:28 UTC and clearing by about 05:00–05:11, with 166 occurrences in total.
An earlier first look recorded the failing image as us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5e023f3.
None of the 12 deploys in the prior 24 h touched x-ray, and none of the cluster error lines mention this pull.
The feedsvc 403s at the same time look unrelated.
Next step: read the pull error in the Kubernetes events (not found or unauthorized), then check that this tag exists in the registry and the CronJob's pull access.

low confidence

claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

namespacex-ray
workloadx-ray-weaviate-stats
whyimage pull failing
podsx-ray-weaviate-stats-29914690-fpppt
pending_minutes115
since1,794,883,816
events2
last_messageError: ImagePullBackOff
discriminatorimage pull failing

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-15 02:26:17 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 16 min — 1 pod level=medium was_level=high finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 16 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29911810-r49q6'], 'pending_minutes': 16, 'since': 1794708602, 'events': 45, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794705977&to=1794709577&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794709577, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 130, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794624607.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794709577, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 604918, 'stale_s': 17}
  • 2026-11-15 02:49:08 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 39 min — 1 pod level=high was_level=medium finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 39 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29911810-r49q6'], 'pending_minutes': 39, 'since': 1794708602, 'events': 53, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794707348&to=1794710948&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794710948, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 132, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 27 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 27, since 1794708602.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794709577, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 606289, 'stale_s': 17}
  • 2026-11-15 05:09:36 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29911810-r49q6'], 'pending_minutes': 119, 'since': 1794711012, 'events': 2, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794714595&to=1794718195&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 63.0, 'status': 'resolved', 'first_seen': 1794104676, 'last_seen': 1794718195, 'resolved_at': 1794719376, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 146, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794711012.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794709577, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 614715, 'stale_s': 1196}
  • 2026-11-16 02:26:12 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 16 min — 1 pod level=medium was_level=high finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 16 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913250-s4t4m'], 'pending_minutes': 16, 'since': 1794795006, 'events': 47, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5e023f3"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794792372&to=1794795972&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794795972, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 147, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 119 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 119, since 1794711012.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794795972, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 691312, 'stale_s': 16}
  • 2026-11-16 02:42:11 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod level=high was_level=medium finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 32 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913250-s4t4m'], 'pending_minutes': 32, 'since': 1794795006, 'events': 53, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-5e023f3"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794793331&to=1794796931&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794796931, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 149, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 24 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 24, since 1794795006.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794795972, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 692329, 'stale_s': 74}
  • 2026-11-16 04:59:16 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 118 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29913250-s4t4m'], 'pending_minutes': 118, 'since': 1794797112, 'events': 4, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794800625&to=1794804225&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 63.0, 'status': 'resolved', 'first_seen': 1794104676, 'last_seen': 1794804225, 'resolved_at': 1794805156, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 163, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 118 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 118, since 1794797112.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794795972, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 700497, 'stale_s': 948}
  • 2026-11-17 02:23:20 UTCreopened x-ray · severity=warning was=critical title=x-ray-weaviate-stats (x-ray) image pull failing for at least 13 min — 1 pod level=medium was_level=high finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 13 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914690-fpppt'], 'pending_minutes': 13, 'since': 1794881420, 'events': 30, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794878600&to=1794882200&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794882200, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 164, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 118 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 118, since 1794797112.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794882200, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 777542, 'stale_s': 18}
  • 2026-11-17 02:44:54 UTCescalated x-ray · severity=critical was=warning title=x-ray-weaviate-stats (x-ray) image pull failing for at least 34 min — 1 pod level=high was_level=medium finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 34 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914690-fpppt'], 'pending_minutes': 34, 'since': 1794881420, 'events': 49, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794879894&to=1794883494&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794104676, 'last_seen': 1794883494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 166, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 25 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 25, since 1794881420.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794882200, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 778917, 'stale_s': 99}
  • 2026-11-17 05:09:52 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16882, 'fingerprint': 'c22a8ba76e38fb9cf885', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-weaviate-stats', 'title': 'x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-weaviate-stats', 'why': 'image pull failing', 'pods': ['x-ray-weaviate-stats-29914690-fpppt'], 'pending_minutes': 115, 'since': 1794883816, 'events': 2, 'last_message': 'Error: ImagePullBackOff', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794887157&to=1794890757&source=events&q=x-ray-weaviate-stats'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 63.0, 'status': 'resolved', 'first_seen': 1794104676, 'last_seen': 1794890757, 'resolved_at': 1794892192, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 176, 'narrative': 'Pod cannot start: x-ray-weaviate-stats (x-ray) image pull failing for at least 115 min — 1 pod. Facts: namespace x-ray, workload x-ray-weaviate-stats, why image pull failing, pending_minutes 115, since 1794883816.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794882200, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 787534, 'stale_s': 1453}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events