ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min)
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
What X-ray is prepared to defend about master right now — each one a claim
with its numbers, the evidence to check it, and what normal looks like. Detectors last ran
never.
?A finding is opened by a detector — a statistical test over the stored rollups, or a
live check of a source — and stays ONE row while the condition persists: it updates,
escalates, and resolves itself fifteen minutes after the detector stops seeing it.
Acknowledge to say you have it; snooze to hide it for a while; nothing here is a
threshold somebody configured.
Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.
What this page will not tell you
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-8glb (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
Node problem: ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-f6208abb-bvdv (1× in 30 min). Facts: reason ReadOnlyFileSystemDetected, count_30m 1, last_message .
execsvc logged “Request completed — ? · could not save Careplan activity V2” 9 times in the last hour and never before in 90 days on delta (first seen 2026-11-17 09:46 UTC, raised at observability/middleware.go:281). Example trace: 9bff744b81af07042b1e3e7de4963df4.
First look low confidence
Cause not clear from logs and recent changes; execsvc returned 500 "could not save Careplan activity V2" 9× between 09:46 and 09:54 UTC on 2026-11-17 (new shape, logged at observability/middleware.go:281).
Checked: no execsvc deploys in the 24 h before onset (running build deployed 2026-11-16 05:11 UTC by [email protected]), no other findings at the same time.
Alongside it: "no result from branch evaluation" (branchV2.go:77, 2×), "Question object not found for question" (questionV2.go:1863, 3×, new at 09:53), and pubsub "transaction was never committed" (2×).
The code shown (careplan.go:1044) is the 403 branch; the 500 comes from a later error path after execution.SaveActivityV2, whose underlying error is not in the bundle.
Next step: open the example trace to read the wrapped error from SaveActivityV2. Then check whether it matches the branch-evaluation or missing-question errors for that protocol.
· its evidence →
execsvc logged “could not match participant questionnaire/question/table/doc-upload to activity graph” 787 times in the last hour and never before in 90 days on master (first seen 2026-11-17 04:05 UTC, raised at execution/activity_details_v2.go:2007). Example trace: b4f239ac36290842023cca51c51d0e33.
First look low confidence
Cause not clear from logs and recent changes; GET activity status in execsvc returned 500 about 700 times between 04:05 and 04:16 UTC on 2026-11-17, logged as 'could not match participant question to activity graph'.
One question appears in the samples, questionId="5571182f-6c21-491e-b8c6-51fce3166e1e", with empty refId, variableName and title, so it may be one participant record polled repeatedly.
The handler error (handlers/activity_details_v2.go:457) has existed since 2026-11-04. Only the log line at execution/activity_details_v2.go:2007 is new.
The 5 deploys in the prior 24 h were autosync config-only with no image commits. The latest, d8d21151cd28, landed 151 min before onset, and nothing links any of them to this error.
Next step: look up question 5571182f-6c21-491e-b8c6-51fce3166e1e and its activity graph, and check why it has no refId, variableName or title. Start from the example trace.
This is also driving the error budget burn on execsvc (18x over 1 h).
· its evidence →
execsvc logged "evalResult is not a bool" 55 times within a one-hour window ending at execlib/branchV2.go:96, with no prior occurrences in the preceding 90 days (first seen 2026-11-16 18:51 UTC). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.
First look low confidence
Cause not clear from logs and recent changes; execsvc logged 'evalResult is not a bool' 55× (18:51–19:21 UTC) at execlib/branchV2.go:96, alongside 'no result from branch evaluation' 150× at branchV2.go:91 since 18:42 (finding 17479).
The same two errors already ran from 14:20 UTC at branchV2.go:77/82, last at 18:31, via the participant auto-query path (activityV2.go:882).
The line numbers moved by 14 between 18:31 and 18:42, which suggests a new execsvc build started then, but no image deploy is recorded.
The only recorded deploy is config-only (6db9855a7d34, [email protected], 05:11 UTC, 820 min before), and nothing links it to these errors.
The error strings were not found in the default-branch code, so the running code is unknown.
Next step: confirm which execsvc build has run since about 18:40, then read execlib/branchV2.go:91-96 in it to see why branch evaluation returns no or non-bool results.
· its evidence →
execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.
First look low confidence
Cause not clear from logs and recent changes; this looks like an existing error that moved, not a new one.
execsvc logged 'no result from branch evaluation' 101× at execlib/branchV2.go:91 from 18:42 to 19:13 UTC, plus 'evalResult is not a bool' 38× at branchV2.go:96 from 18:51 UTC.
The same two messages came from branchV2.go:77 (106×, seen since Nov 7) and :82 (4×) until 18:31 UTC, then stopped. The activityV2.go:882 wrapper lines also stopped at 18:31.
That 14-line shift suggests a new execsvc build rolled out around 18:31–18:42, but the only recorded deploy is [email protected] at 05:11 UTC, 810 min earlier. Code and commits could not be read because the GitHub budget ran out.
Next step: find what execsvc version started running around 18:31–18:42 UTC, diff branchV2.go, and check why branch evaluation returns no result (failing since Nov 7).
· its evidence →
Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.
Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.
First look medium confidence
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
Suspect: x-ray-services-master sync 556a243c8ac9 (autosync)
· its evidence →
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 174, c4xx 166, share_pct 95.40229885057471, c401 0.
First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/news returned 403 on 44 of 47 calls (94 %) in 15 min, against a usual 0.01 per window. Onset was 2026-11-17 05:20 UTC, and the first failing request in the log is at 05:27. Every sampled 403 comes from one org, "Teammates Demo Sandbox", and returns fast (13–21 ms). That points to a permission check on that org or its users
· its evidence →
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: 4xx — 30 of 30 calls (100 %; other 4xx×30) in the last 15 min — 130× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 30, c4xx 30, share_pct 100.0, c401 0.
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.
First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see.
· its evidence →
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 33 of 52 calls (63 %; 403×33) in the last 15 min — 20× its usual 1.6. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 52, c4xx 33, share_pct 63.46153846153846, c401 0.
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 52 of 64 calls (81 %; 403×52) in the last 15 min — 22× its usual 2.3. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 64, c4xx 52, share_pct 81.25, c401 0.
4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.
First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies.
· its evidence →
The background job execsvc.auto-query-reeval-sweep on production failed 11 runs recently (425 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The background job execsvc.auto-query-reeval-sweep on delta failed 7 runs recently (99 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
Pod cannot start: schema-migrate (tilda) image pull failing for at least 15 min — 1 pod. Facts: namespace tilda, workload schema-migrate, why image pull failing, pending_minutes 15, since 1794911087.
Pod cannot start: rag-agent (tilda) image pull failing for at least 119 min — 1 pod. Facts: namespace tilda, workload rag-agent, why image pull failing, pending_minutes 119, since 1794904894.