← findings
rag-agent (tilda) image pull failing for at least 119 min — 1 pod
warning
Pod cannot start
test26
open
impact 38
opened 3d ago · last observed 23h ago · observed 316× · #17315 · 896b427715581fdddd00
What happened template
Pod cannot start: rag-agent (tilda) image pull failing for at least 119 min — 1 pod. Facts: namespace tilda, workload rag-agent, why image pull failing, pending_minutes 119, since 1794904894.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
The facts
| namespace | tilda |
| workload | rag-agent |
| why | image pull failing |
| pods | rag-agent-646f7f99f6-9tc2s |
| pending_minutes | 119 |
| since | 1,794,904,894 |
| events | 42 |
| last_message | Error: ImagePullBackOff |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | image pull failing |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-15 07:32:27 UTCcypher execsvc shape 00000000 p95 went 32 ms → 124 ms across the window (14,496 runs)
- 2026-11-15 07:32:27 UTCcypher virtual-assistant shape 00000000 p95 went 9 ms → 34 ms across the window (69,711 runs)
- 2026-11-15 08:21:26 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-rhuj (1× in 30 min)
- 2026-11-15 08:32:27 UTCthis finding rag-agent (tilda) image pull failing for at least 119 min — 1 pod
- 2026-11-15 09:24:04 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-8i4p (1× in 30 min)
- 2026-11-15 15:00:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:21:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:29:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:35:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 00:11:50 UTCfinding · warning virtual-assistant: a new error shape, 25× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)
- 2026-11-16 00:11:50 UTCfinding · warning virtual-assistant: a new error shape, 12× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e)
- 2026-11-16 00:50:21 UTCfinding · warning docproc: a new error shape, 8× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )
- 2026-11-16 01:09:28 UTCfinding · warning docproc: a new error shape, 5× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)
- 2026-11-16 02:09:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 02:26:12 UTCfinding · warning docsvc: a new error shape, 11× in the last hour, never seen in 90 days (Request completed — ? · document cannot be promoted to TMF: document m)
- 2026-11-16 02:26:12 UTCfinding · warning docsvc PUT /api/documents/promote: 16 of 16 calls failed (100 %) in the last 15 min
- 2026-11-16 05:02:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:05:00 UTCerrors cluster 5xx peaked at 98 in one five-minute bucket (median 0 in this window)
- 2026-11-16 13:26:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 13:27:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 14:00:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 16:05:00 UTClatency cluster p95 peaked at 8.51 s
- 2026-11-16 17:37:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:45:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 19:49:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:15:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:30:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:34:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:53:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 01:11:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 02:15:09 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 07:50:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-17 08:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 09:12:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
History
- 2026-11-15 08:32:27 UTCopened x-ray · severity=warning title=rag-agent (tilda) image pull failing for at least 14 min — 1 pod finding={'id': 17315, 'fingerprint': '896b427715581fdddd00', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/rag-agent', 'title': 'rag-agent (tilda) image pull failing for at least 14 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'rag-agent', 'why': 'image pull failing', 'pods': ['rag-agent-646f7f99f6-9tc2s'], 'pending_minutes': 14, 'since': 1794730649, 'events': 38, 'last_message': 'Back-off pulling image "us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/rag-agent:eng-test-run-tracing-71a404e"', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794727947&to=1794731547&source=events&q=rag-agent'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 21.2, 'status': 'open', 'first_seen': 1794731547, 'last_seen': 1794731547, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'image pull failing', 'level': 'medium', 'told_level': None, 'episode_at': 1794731547, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 21, 'stale_s': 21}
- 2026-11-15 10:37:20 UTCescalated x-ray · severity=warning was=warning title=rag-agent (tilda) image pull failing for at least 119 min — 1 pod level=high was_level=medium finding={'id': 17315, 'fingerprint': '896b427715581fdddd00', 'kind': 'pod.cannot_start', 'severity': 'warning', 'cluster': 'test26', 'subject': 'tilda/rag-agent', 'title': 'rag-agent (tilda) image pull failing for at least 119 min — 1 pod', 'facts': {'namespace': 'tilda', 'workload': 'rag-agent', 'why': 'image pull failing', 'pods': ['rag-agent-646f7f99f6-9tc2s'], 'pending_minutes': 119, 'since': 1794731859, 'events': 42, 'last_message': 'Error: ImagePullBackOff', 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': 'image pull failing'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=test26&from=1794735440&to=1794739040&source=events&q=rag-agent'}, {'label': 'Pods', 'href': '/pods?cluster=test26'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 32.5, 'status': 'open', 'first_seen': 1794731547, 'last_seen': 1794739040, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': 'Pod cannot start: rag-agent (tilda) image pull failing for at least 119 min — 1 pod. Facts: namespace tilda, workload rag-agent, why image pull failing, pending_minutes 119, since 1794731495.', 'narrative_kind': 'template', 'discriminator': 'image pull failing', 'level': 'high', 'told_level': None, 'episode_at': 1794731547, 'chat_thread': None, 'kind_label': 'Pod cannot start', 'group': 'Pods', 'age_s': 7511, 'stale_s': 18}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |