Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 48 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:26:38.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

critical Pod cannot start master open impact 62 opened 1d ago · last observed 1d ago · observed 14× · #17510 · a089fae4aa8b198aa1c4

What happened template

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.

First look — investigated before anyone looked

Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.

medium confidence config x-ray-services-master sync 556a243c8ac9 (autosync)

claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

namespacex-ray
workloadx-ray-argocd-collector
whyimage pull failing
podsx-ray-argocd-collector-29915107-rqnsj, x-ray-argocd-collector-29915167-zbmpl
pending_minutes93
since1,794,906,422
events104
last_messageError: ImagePullBackOff
discriminatorimage pull failing

The story — 2 changes landed in the hour before this opened — adjacency, not proof.

History

  • 2026-11-17 06:29:15 UTCopened x-ray · severity=warning title=x-ray-argocd-collector (x-ray) image pull failing for at least 22 min — 1 pod finding={'id': 17510, 'fingerprint': 'a089fae4aa8b198aa1c4', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'master', 'subject': 'x-ray/x-ray-argocd-collector', 'title': 'x-ray-argocd-collector (x-ray) image pull failing for at least 22 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-argocd-collector', 'why': 'image pull failing', 'pods': ['x-ray-argocd-collector-29914927-d557w'], 'pending_minutes': 22, 'since': 1794895625, 'events': 47, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794893355&to=1794896955&source=events&q=x-ray-argocd-collector'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.0, 'status': 'open', 'first_seen': 1794896955, 'last_seen': 1794896955, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 19, 'stale_s': 19}
  • 2026-11-17 06:42:35 UTCescalated x-ray · severity=critical was=warning title=x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod level=high was_level=medium finding={'id': 17510, 'fingerprint': 'a089fae4aa8b198aa1c4', 'kind': 'pod.cannot_start', 'severity': 'critical', 'cluster': 'master', 'subject': 'x-ray/x-ray-argocd-collector', 'title': 'x-ray-argocd-collector (x-ray) image pull failing for at least 35 min — 1 pod', 'facts': {'namespace': 'x-ray', 'workload': 'x-ray-argocd-collector', 'why': 'image pull failing', 'pods': ['x-ray-argocd-collector-29914927-d557w'], 'pending_minutes': 35, 'since': 1794895625, 'events': 51, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794894155&to=1794897755&source=events&q=x-ray-argocd-collector'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 58.0, 'status': 'open', 'first_seen': 1794896955, 'last_seen': 1794897755, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'The pod x-ray-argocd-collector in namespace x-ray cannot start due to image pull failures lasting at least 22 minutes. One pod (x-ray-argocd-collector-29914927-d557w) is affected. The system is attempting to pull image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc" and backing off. Since timestamp: 1794895625.', 'narrative_kind': 'ai', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 819, 'stale_s': 19}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events