← findings
HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h
warning
Autoscaler blind
delta
open
impact 32
opened 49d ago · last observed 1d ago · observed 953× · #2936 · 1e98a87121ca36522ddc
What happened template
Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message unable to get metric memory: no metrics returned from resource metrics API.
A KEDA scaler or HPA that cannot read its metric — it will not scale when load changes.
The facts
| namespace | tilda |
| object | HorizontalPodAutoscaler/docproc-hpa |
| failures_2h | 25 |
| last_message | unable to get metric memory: no metrics returned from resource metrics API |
| discriminator |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-29 22:11:08 UTCthis finding HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h
- 2026-09-30 11:38:19 UTCfinding · warning HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 28× in 2 h
- 2026-10-23 13:03:15 UTCfinding · warning pods on an unready node: 12 affected in 30 min
- 2026-10-28 13:48:03 UTCfinding · warning Neo4j delta, database neo4j: 7 Cypher queries failed in the last 15 min
- 2026-10-30 18:48:53 UTCfinding · critical onboardingsvc GET /api/myprofile/v2: permission denied — 37 of 37 calls (100 %; 401×37) in the last 15 min — 167× its usual 0.2
- 2026-11-02 15:21:10 UTCfinding · warning HorizontalPodAutoscaler/tmfsync-hpa (tilda) cannot read its scaling metric — 28× in 2 h
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 38 of 38 calls (100 %; 401×38) in the last 15 min — 122× its usual 0.3
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/userprefs: permission denied — 35 of 35 calls (100 %; 401×35) in the last 15 min — 162× its usual 0.2
- 2026-11-03 17:37:41 UTCfinding · critical onboardingsvc GET /api/onboardingsvc/version: permission denied — 31 of 31 calls (100 %; 401×31) in the last 15 min — 151× its usual 0.2
- 2026-11-14 15:10:15 UTCdeploy automated deployed studiosvc-infra-delta
- 2026-11-15 06:25:00 UTCerrors cluster 5xx peaked at 35 in one five-minute bucket (median 0 in this window)
- 2026-11-16 05:11:39 UTCdeploy [email protected] deployed asm
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:11:45 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed litellm
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:11:48 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:11:49 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:11:50 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:11:51 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:11:54 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:11:55 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:11:57 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 16:55:00 UTClatency cluster p95 peaked at 7.14 s
History
- 2026-11-14 11:46:06 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794651862&to=1794655462&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794655462, 'resolved_at': 1794656766, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 852, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794638260, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3936916, 'stale_s': 1322}
- 2026-11-14 19:44:51 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 2, 'last_message': 'unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: googleapi: Error 500: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs., backendError', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794681891&to=1794685491&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794685491, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 853, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794685491, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3965641, 'stale_s': 18}
- 2026-11-14 21:58:37 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 2, 'last_message': 'unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: googleapi: Error 500: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs., backendError', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794688671&to=1794692271&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794692271, 'resolved_at': 1794693517, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 866, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 2, last_message unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: googleapi: Error 500: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs., backendError.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794685491, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3973667, 'stale_s': 1264}
- 2026-11-15 04:29:28 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 7× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 7× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 7, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: unable to fetch metrics from custom metrics API: no known available metric versions found', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794713368&to=1794716968&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794716968, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 867, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 2, last_message unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: googleapi: Error 500: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs., backendError.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794716968, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3997149, 'stale_s': 49}
- 2026-11-15 08:13:26 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 11, 'last_message': 'did not receive metrics for targeted pods (pods might be unready)', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794725718&to=1794729318&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794729318, 'resolved_at': 1794730406, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 894, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 11, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794716968, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4010557, 'stale_s': 1107}
- 2026-11-16 05:18:52 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 5× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 5× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 5, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794802732&to=1794806332&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794806332, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 895, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 11× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 11, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794806332, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4086503, 'stale_s': 39}
- 2026-11-16 07:26:53 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 5× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 5, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794809323&to=1794812923&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794812923, 'resolved_at': 1794814013, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 905, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 5× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 5, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794806332, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4094187, 'stale_s': 1132}
- 2026-11-16 08:53:28 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'did not receive metrics for targeted pods (pods might be unready)', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794815608&to=1794819208&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794819208, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 906, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 5× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 5, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4099390, 'stale_s': 50}
- 2026-11-16 10:49:21 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'did not receive metrics for targeted pods (pods might be unready)', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794820778&to=1794824378&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794824378, 'resolved_at': 1794826161, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 909, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794819208, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4106322, 'stale_s': 1812}
- 2026-11-16 12:06:52 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794827212&to=1794830812&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794830812, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 910, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4110976, 'stale_s': 32}
- 2026-11-16 13:58:18 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794832797&to=1794836397&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794836397, 'resolved_at': 1794837498, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 915, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794830812, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4117659, 'stale_s': 1130}
- 2026-11-16 14:38:42 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'did not receive metrics for targeted pods (pods might be unready)', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794836322&to=1794839922&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794839922, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 916, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794839922, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4120078, 'stale_s': 24}
- 2026-11-16 18:24:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 18× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 18, 'last_message': 'did not receive metrics for targeted pods (pods might be unready)', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794848823&to=1794852423&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794852423, 'resolved_at': 1794853467, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 939, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 18× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 18, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794839922, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4133625, 'stale_s': 1070}
- 2026-11-16 18:32:09 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 24× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 24× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 24, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794850329&to=1794853929&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794853929, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 940, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 18× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 18, last_message did not receive metrics for targeted pods (pods might be unready).', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4134084, 'stale_s': 23}
- 2026-11-16 20:46:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794857347&to=1794860947&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 31.8, 'status': 'resolved', 'first_seen': 1790719868, 'last_seen': 1794860947, 'resolved_at': 1794861972, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 951, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794853929, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4142126, 'stale_s': 1047}
- 2026-11-17 10:21:45 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h level=medium was_level=medium finding={'id': 2936, 'fingerprint': '1e98a87121ca36522ddc', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tilda/HorizontalPodAutoscaler/docproc-hpa', 'title': 'HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/docproc-hpa', 'failures_2h': 25, 'last_message': 'unable to get metric memory: no metrics returned from resource metrics API', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=delta&from=1794907305&to=1794910905&source=events&q=HorizontalPodAutoscaler%2Fdocproc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 26.8, 'status': 'open', 'first_seen': 1790719868, 'last_seen': 1794910905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 952, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 25× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 25, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794910905, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4191072, 'stale_s': 35}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |