Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:34:23.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz

warning Out-of-memory kills automation open impact 30 opened 1d ago · last observed 1d ago · observed 6× · #17512 · a53c09e28c4bb10af000

What happened template

Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.

The kernel killed a process for exceeding its memory cgroup — including a process inside a container that kept running, which restart counts never see.

The facts

nodegk3-automation-nap-1p2bubxp-41aa9f63-gpdz
processobs-serve
kills_30m2
likely_workloadsmatrix/chat-agent, matrix/dashql, matrix/dashrsvc, matrix/data-monitor, matrix/docsvc, matrix/dslsvc, matrix/enrollsvc, matrix/execsvc
last_messageMemory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985
notethe kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node
discriminatorobs-serve

The story — 97 changes landed in the hour before this opened — adjacency, not proof.

History

  • 2026-11-17 08:04:09 UTCopened x-ray · severity=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2545947 (obs-serve) total-vm:2989708kB, anon-rss:1042888kB, file-rss:8192kB, shmem-rss:0kB, UID:1000 pgtables:2652kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794899049&to=1794902649&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794902649, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 23, 'stale_s': 23}
  • 2026-11-17 09:22:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794902154&to=1794905754&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 28.5, 'status': 'resolved', 'first_seen': 1794902649, 'last_seen': 1794905754, 'resolved_at': 1794907325, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 4700, 'stale_s': 1595}
  • 2026-11-17 09:39:50 UTCreopened x-ray · severity=warning was=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz level=medium was_level=medium finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/docsvc', 'matrix/enrollsvc', 'matrix/exportsvc', 'matrix/feedsvc', 'matrix/finsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794904790&to=1794908390&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794908390, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 4, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2558663 (obs-serve) total-vm:2706980kB, anon-rss:1037636kB, file-rss:8088kB, shmem-rss:0kB, UID:1000 pgtables:2600kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 5819, 'stale_s': 78}
  • 2026-11-17 10:21:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/docsvc', 'matrix/exportsvc', 'matrix/feedsvc', 'matrix/finsvc', 'matrix/onboardingsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794905977&to=1794909577&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.5, 'status': 'resolved', 'first_seen': 1794902649, 'last_seen': 1794909577, 'resolved_at': 1794910905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 8285, 'stale_s': 1357}
  • 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz level=medium was_level=medium finding={'id': 17512, 'fingerprint': 'a53c09e28c4bb10af000', 'kind': 'pod.oom', 'severity': 'warning', 'cluster': 'automation', 'subject': 'node/gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'title': 'the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'facts': {'node': 'gk3-automation-nap-1p2bubxp-41aa9f63-gpdz', 'process': 'obs-serve', 'kills_30m': 2, 'likely_workloads': ['matrix/chat-agent', 'matrix/dashql', 'matrix/dashrsvc', 'matrix/data-monitor', 'matrix/docsvc', 'matrix/dslsvc', 'matrix/enrollsvc', 'matrix/execsvc'], 'last_message': 'Memory cgroup out of memory: Killed process 2610801 (obs-serve) total-vm:2657908kB, anon-rss:1037496kB, file-rss:8984kB, shmem-rss:0kB, UID:1000 pgtables:2696kB oom_score_adj:985', 'note': 'the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node', 'discriminator': 'obs-serve'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=automation&from=1794908441&to=1794912041&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=automation'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 27.6, 'status': 'open', 'first_seen': 1794902649, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 6, 'narrative': 'Out-of-memory kills: the kernel OOM-killed “obs-serve” 2× in 30 min on gk3-automation-nap-1p2bubxp-41aa9f63-gpdz. Facts: node gk3-automation-nap-1p2bubxp-41aa9f63-gpdz, process obs-serve, kills_30m 2, last_message Memory cgroup out of memory: Killed process 2573743 (obs-serve) total-vm:2852324kB, anon-rss:1043692kB, file-rss:7824kB, shmem-rss:0kB, UID:1000 pgtables:2640kB oom_score_adj:985, note the kernel names the process and node, not the pod; likely_workloads are the ones recently placed on that node.', 'narrative_kind': 'template', 'discriminator': 'obs-serve', 'level': 'medium', 'told_level': None, 'episode_at': 1794912041, 'chat_thread': None, 'kind_label': 'Out-of-memory kills', 'group': 'Pods', 'age_s': 9497, 'stale_s': 105}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events