← findings
pods on an unready node: 12 affected in 30 min
warning
Node problem
delta
open
impact 32
opened 27d ago · last observed 2d ago · observed 2891× · #5460 · 7a705118693529fb3d24
What happened template
Node problem: pods on an unready node: 12 affected in 30 min. Facts: reason NodeNotReady, count_30m 12, last_message Node is not ready.
A node reporting a condition (not ready, read-only filesystem, disk pressure, kubelet/containerd restarting).
The facts
| reason | NodeNotReady |
| count_30m | 12 |
| objects | collector-2x8xz, collector-4m47k, collector-4v5pl, collector-54sx6, collector-5h4j2, collector-bnt8f, collector-gw68k, collector-kr9hj, collector-r8ppl, collector-vl2ft |
| last_message | Node is not ready |
| discriminator | NodeNotReady |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-23 13:03:15 UTCthis finding pods on an unready node: 12 affected in 30 min
- 2026-10-28 13:48:03 UTCfinding · warning Neo4j delta, database neo4j: 7 Cypher queries failed in the last 15 min
- 2026-10-30 18:48:53 UTCfinding · critical onboardingsvc GET /api/myprofile/v2: permission denied — 37 of 37 calls (100 %; 401×37) in the last 15 min — 167× its usual 0.2
- 2026-11-02 15:21:10 UTCfinding · warning HorizontalPodAutoscaler/tmfsync-hpa (tilda) cannot read its scaling metric — 28× in 2 h
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 38 of 38 calls (100 %; 401×38) in the last 15 min — 122× its usual 0.3
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/userprefs: permission denied — 35 of 35 calls (100 %; 401×35) in the last 15 min — 162× its usual 0.2
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/onboardingsvc/version: permission denied — 31 of 31 calls (100 %; 401×31) in the last 15 min — 151× its usual 0.2
- 2026-11-04 10:48:40 UTCfinding · info tmfsync request rate 20.0 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (4 of the last 4 checks)
- 2026-11-05 00:54:34 UTCfinding · warning delta AI spend projects to $295 this month, 238.7× last month's $1
- 2026-11-14 15:10:15 UTCdeploy automated deployed studiosvc-infra-delta
- 2026-11-15 06:25:00 UTCerrors cluster 5xx peaked at 35 in one five-minute bucket (median 0 in this window)
- 2026-11-16 05:11:39 UTCdeploy [email protected] deployed asm
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:11:45 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed litellm
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:11:48 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:11:49 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:11:50 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:11:51 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:11:54 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:11:55 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:11:57 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 16:55:00 UTClatency cluster p95 peaked at 7.14 s
History
- 2026-11-16 12:58:26 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 2 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 2, 'objects': ['collector-6bw4k', 'collector-wnwkp'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794828571&to=1794832171&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794832171, 'resolved_at': 1794833906, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2834, 'narrative': 'Node problem: pods on an unready node: 2 affected in 30 min. Facts: reason NodeNotReady, count_30m 2, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'low', 'told_level': None, 'episode_at': 1794611390, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2073336, 'stale_s': 1760}
- 2026-11-16 13:48:59 UTCreopened x-ray · severity=warning was=warning title=pods on an unready node: 1 affected in 30 min level=medium was_level=low finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-svvbr'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794833339&to=1794836939&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794836939, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2835, 'narrative': 'Node problem: pods on an unready node: 2 affected in 30 min. Facts: reason NodeNotReady, count_30m 2, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794836939, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2076369, 'stale_s': 25}
- 2026-11-16 18:16:26 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-rpjh7'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794848313&to=1794851913&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794851913, 'resolved_at': 1794852986, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2860, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794836939, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2092422, 'stale_s': 1104}
- 2026-11-16 20:29:07 UTCreopened x-ray · severity=warning was=warning title=pods on an unready node: 1 affected in 30 min level=medium was_level=medium finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-prmwb'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794857347&to=1794860947&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794860947, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2861, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794860947, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2100389, 'stale_s': 37}
- 2026-11-16 21:17:10 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-65px7'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794859282&to=1794862882&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794862882, 'resolved_at': 1794863830, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2865, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794860947, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2103304, 'stale_s': 1017}
- 2026-11-16 21:27:22 UTCreopened x-ray · severity=warning was=warning title=pods on an unready node: 1 affected in 30 min level=medium was_level=medium finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-znxtm'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794860842&to=1794864442&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794864442, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2866, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794864442, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2103870, 'stale_s': 23}
- 2026-11-16 23:37:31 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 2 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 2, 'objects': ['collector-7598n', 'collector-b4qzb'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794867727&to=1794871327&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794871327, 'resolved_at': 1794872251, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2879, 'narrative': 'Node problem: pods on an unready node: 2 affected in 30 min. Facts: reason NodeNotReady, count_30m 2, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794864442, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2111677, 'stale_s': 945}
- 2026-11-17 06:29:15 UTCreopened x-ray · severity=warning was=warning title=pods on an unready node: 2 affected in 30 min level=medium was_level=medium finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 2 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 2, 'objects': ['collector-5pkpd', 'collector-lntnf'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794893355&to=1794896955&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794896955, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2880, 'narrative': 'Node problem: pods on an unready node: 2 affected in 30 min. Facts: reason NodeNotReady, count_30m 2, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2136385, 'stale_s': 25}
- 2026-11-17 07:24:09 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 1 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 1, 'objects': ['collector-lntnf'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794894937&to=1794898537&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794898537, 'resolved_at': 1794900249, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2882, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794896955, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2139727, 'stale_s': 1785}
- 2026-11-17 07:42:20 UTCreopened x-ray · severity=warning was=warning title=pods on an unready node: 5 affected in 30 min level=medium was_level=medium finding={'id': 5460, 'fingerprint': '7a705118693529fb3d24', 'kind': 'node.unhealthy', 'severity': 'warning', 'cluster': 'delta', 'subject': 'node/NodeNotReady', 'title': 'pods on an unready node: 5 affected in 30 min', 'facts': {'reason': 'NodeNotReady', 'count_30m': 5, 'objects': ['collector-498rj', 'collector-g28xk', 'collector-j5nwg', 'collector-jmghr', 'collector-jzxbh'], 'last_message': 'Node is not ready', 'discriminator': 'NodeNotReady'}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794897740&to=1794901340&source=events'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794901340, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2883, 'narrative': 'Node problem: pods on an unready node: 1 affected in 30 min. Facts: reason NodeNotReady, count_30m 1, last_message Node is not ready.', 'narrative_kind': 'template', 'discriminator': 'NodeNotReady', 'level': 'medium', 'told_level': None, 'episode_at': 1794901340, 'chat_thread': None, 'kind_label': 'Node problem', 'group': 'Platform', 'age_s': 2140773, 'stale_s': 28}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |