← findings
regsvc (tilda) image pull failing for at least 116 min — 1 pod
warning
Pod cannot start
test26
open
impact 38
opened 10d ago · last observed 23h ago · observed 985× · #16899 · fd546412409982bd0225
What happened template
Pod cannot start: regsvc (tilda) image pull failing for at least 116 min — 1 pod. Facts: namespace tilda, workload regsvc, why image pull failing, pending_minutes 116, since 1794905077.
A workload's pods have been unable to start for 10+ minutes — image pull failing, a container config error, or no node that can take them — and are still being reported.
The facts
| namespace | tilda |
| workload | regsvc |
| why | image pull failing |
| pods | regsvc-5657656d47-r9j2t |
| pending_minutes | 116 |
| since | 1,794,905,077 |
| events | 42 |
| last_message | Error: ImagePullBackOff |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | image pull failing |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-08 09:54:20 UTCthis finding regsvc (tilda) image pull failing for at least 116 min — 1 pod
- 2026-11-11 06:35:31 UTCfinding · warning studiosvc GET /api/studio/events: 4xx — 100 of 100 calls (100 %; other 4xx×100) in the last 15 min — it rejects nearly every call
- 2026-11-12 13:28:10 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-mw2n (1× in 30 min)
- 2026-11-14 10:16:00 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 10:51:22 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 11:00:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-14 11:14:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:16:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:52:45 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:16:14 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 14:25:45 UTCfinding · warning studiosvc: a new error shape, ≥4357× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 14:27:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:27:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:30:14 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:45:39 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 18:24:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:00:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:17:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 20:17:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 23:49:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:24:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 02:25:02 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 05:02:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 05:16:48 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:11:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:21:30 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-518ae407-p1sf (1× in 30 min)
- 2026-11-15 06:32:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 07:17:13 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-518ae407-zdm7 (1× in 30 min)
- 2026-11-15 07:28:58 UTCfinding · warning finsvc (tilda) image pull failing for at least 119 min — 1 pod
- 2026-11-15 08:21:26 UTCfinding · info ReadOnlyFileSystemDetected: gk3-test26-nap-1tqh1p5r-ab0f2c65-rhuj (1× in 30 min)
- 2026-11-15 08:32:27 UTCfinding · warning rag-agent (tilda) image pull failing for at least 119 min — 1 pod
- 2026-11-15 15:00:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:21:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:29:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:35:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 02:09:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:02:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:05:00 UTCerrors cluster 5xx peaked at 98 in one five-minute bucket (median 0 in this window)
- 2026-11-16 13:26:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 13:27:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-16 14:00:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:01:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:37:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 17:45:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 19:49:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:15:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:30:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:34:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 00:53:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 01:11:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 02:15:09 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 07:50:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-17 08:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-17 09:12:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
History
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |