the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz
warning
Out-of-memory kills
automation
open
impact 30
opened 1d ago · last observed 1d ago · observed 6× · #17512 · a53c09e28c4bb10af000
What happened template
Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.
The kernel killed a process for exceeding its memory cgroup — including a process inside a container that kept running, which restart counts never see.
The facts
| node | gk3-automation-nap-1p2bubxp-41aa9f63-gpdz |
| process | obs-serve |
| kills_30m | 2 |
| likely_workloads | matrix/chat-agent, matrix/dashql, matrix/dashrsvc, matrix/data-monitor, matrix/docsvc, matrix/dslsvc, matrix/enrollsvc, matrix/execsvc |
| last_message | Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985 |
| note | the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node |
| discriminator | obs-serve |
The story — 97 changes landed in the hour before this opened — adjacency, not proof.
- 2026-11-17 07:04:09 UTCcypher tusker shape 549A56E7 p95 went 2 ms → 18 ms across the window (837 runs)
- 2026-11-17 07:13:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:14:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:15:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:16:02 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:00 UTClatency cluster p95 peaked at 8.83 s
- 2026-11-17 07:20:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:32 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:33 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:37 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:47 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:51 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:24:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:24:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:13 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:13 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:40:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:40:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:48:41 UTChand change [email protected] changed apps/v1/namespaces/x-ray/deployments/x-ray by hand (Go-http-client/1.1)
- 2026-11-17 07:50:57 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:50:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:50:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:50:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:01 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:01 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:02 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:04 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:13 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:54:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:54:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:03:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:04:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:04:09 UTCthis finding the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz
- 2026-11-17 08:21:02 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:21:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:13 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:30:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:31:04 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:32 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:32 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:33 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:37 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:37 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:53:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:54:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:00:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:00:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:00:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:04 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:02 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/swaggersvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:02:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:32 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:04 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/swaggersvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:05:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:37 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:37 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:18:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:18:55 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:32:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:32:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/swaggersvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:51:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:57:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:57:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:58:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:58:34 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/chat-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashql by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dashrsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/data-monitor by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docproc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/docsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/dslsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:12 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/enrollsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/exportsvc-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:16 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/feedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/notifsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:20 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/ontologysvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/patientsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/rag-agent-worker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/regsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/schedsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studioaisvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tmfsync by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/tusker by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplan-agent by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/workplansvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:06:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-17 08:04:09 UTCopened x-ray · severity=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2545947 (obs-serve) total-vm:2989708kB, anon-rss:1042888kB, file-rss:8192kB, shmem-rss:0kB, UID:1000 pgtables:2652kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794899049&to=1794902649&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794902649, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 23, 'stale_s': 23}
- 2026-11-17 09:22:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794902154&to=1794905754&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 28.5, 'status': 'resolved', 'first_seen': 1794902649, 'last_seen': 1794905754, 'resolved_at': 1794907325, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 4700, 'stale_s': 1595}
- 2026-11-17 09:39:50 UTCreopened x-ray · severity=warning was=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz level=medium was_level=medium finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/docsvc', 'matrix/enrollsvc', 'matrix/exportsvc', 'matrix/feedsvc', 'matrix/finsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794904790&to=1794908390&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794908390, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 5819, 'stale_s': 78}
- 2026-11-17 10:21:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/docsvc', 'matrix/exportsvc', 'matrix/feedsvc', 'matrix/finsvc', 'matrix/onboardingsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794905977&to=1794909577&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.5, 'status': 'resolved', 'first_seen': 1794902649, 'last_seen': 1794909577, 'resolved_at': 1794910905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 8285, 'stale_s': 1357}
- 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz level=medium was_level=medium finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794908441&to=1794912041&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 6, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794912041, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 9497, 'stale_s': 105}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |