← findings
HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
warning
Autoscaler blind
production
resolved
impact 35
opened 46d ago · last observed 2d ago · observed 188× · #3393 · 982cba7dcdaa0c42eac7
What happened template
Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 4, last_message No recommendation.
A KEDA scaler or HPA that cannot read its metric — it will not scale when load changes.
The facts
| namespace | tilda |
| object | HorizontalPodAutoscaler/ontologysvc-hpa |
| failures_2h | 4 |
| last_message | No recommendation |
| discriminator |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-03 04:31:35 UTCthis finding HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
- 2026-10-13 15:54:07 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
- 2026-10-18 18:16:26 UTCfinding · critical docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
- 2026-10-19 06:46:26 UTCfinding · critical notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
- 2026-10-23 13:03:15 UTCfinding · info virtual-assistant: Neo4j queries per request 83.8 — 14.0× the fleet's usual 6.0
- 2026-10-23 13:42:32 UTCfinding · info dashql: p95 latency (ms) 3.77 s — 5.2× the fleet's usual 726 ms
- 2026-10-23 13:49:17 UTCfinding · info virtual-assistant Neo4j load 413 q/min vs a normal 4 q/min ± 4 q/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-10-23 14:04:30 UTCfinding · info docsvc: p99 latency (ms) 3.88 s — 4.7× the fleet's usual 825 ms
- 2026-10-23 14:04:30 UTCfinding · info docsvc runs 34 foreground Neo4j queries per request (23,246 queries / 693 calls, last hour)
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
History
- 2026-11-15 05:48:18 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h level=medium was_level=medium finding={'id': 3393, 'fingerprint': '982cba7dcdaa0c42eac7', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/HorizontalPodAutoscaler/ontologysvc-hpa', 'title': 'HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/ontologysvc-hpa', 'failures_2h': 16, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794718098&to=1794721698&source=events&q=HorizontalPodAutoscaler%2Fontologysvc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.7, 'status': 'open', 'first_seen': 1791001895, 'last_seen': 1794721698, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 163, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 3× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 3, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721698, 'chat_thread': 'xray-alert-11-748fd5a928-g-1792156093', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3719820, 'stale_s': 17}
- 2026-11-15 05:54:29 UTCnotified x-ray · alert_id=11 channel=in-app status=delivered send_id=50613
- 2026-11-15 05:54:29 UTCnotified x-ray · alert_id=11 channel=prod-notifications status=sent send_id=50613
- 2026-11-15 07:55:18 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 3393, 'fingerprint': '982cba7dcdaa0c42eac7', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/HorizontalPodAutoscaler/ontologysvc-hpa', 'title': 'HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/ontologysvc-hpa', 'failures_2h': 16, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794724738&to=1794728338&source=events&q=HorizontalPodAutoscaler%2Fontologysvc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.7, 'status': 'resolved', 'first_seen': 1791001895, 'last_seen': 1794728338, 'resolved_at': 1794729318, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 177, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 16, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721698, 'chat_thread': 'xray-alert-11-748fd5a928-982cba7dcdaa0c42eac7-1794722069', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3727443, 'stale_s': 1000}
- 2026-11-15 13:54:55 UTCnotified x-ray · alert_id=11 channel=in-app status=delivered send_id=50820
- 2026-11-15 13:54:55 UTCnotified x-ray · alert_id=11 channel=prod-notifications status=sent send_id=50820
- 2026-11-16 05:18:52 UTCreopened x-ray · severity=warning was=warning title=HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h level=medium was_level=medium finding={'id': 3393, 'fingerprint': '982cba7dcdaa0c42eac7', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/HorizontalPodAutoscaler/ontologysvc-hpa', 'title': 'HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/ontologysvc-hpa', 'failures_2h': 4, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794802732&to=1794806332&source=events&q=HorizontalPodAutoscaler%2Fontologysvc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.7, 'status': 'open', 'first_seen': 1791001895, 'last_seen': 1794806332, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 178, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 16, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794806332, 'chat_thread': 'xray-alert-11-748fd5a928-982cba7dcdaa0c42eac7-1794722069', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3804474, 'stale_s': 37}
- 2026-11-16 07:26:53 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 3393, 'fingerprint': '982cba7dcdaa0c42eac7', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/HorizontalPodAutoscaler/ontologysvc-hpa', 'title': 'HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'HorizontalPodAutoscaler/ontologysvc-hpa', 'failures_2h': 4, 'last_message': 'No recommendation', 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794809323&to=1794812923&source=events&q=HorizontalPodAutoscaler%2Fontologysvc-hpa'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.7, 'status': 'resolved', 'first_seen': 1791001895, 'last_seen': 1794812923, 'resolved_at': 1794814013, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 188, 'narrative': 'Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 4, last_message No recommendation.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794806332, 'chat_thread': 'xray-alert-11-748fd5a928-982cba7dcdaa0c42eac7-1794722069', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 3812158, 'stale_s': 1130}
Notifications
- 2026-11-15 13:54:55 UTCresolvedin-app · delivered
- 2026-11-15 13:54:55 UTCresolvedprod-notifications · sent
- 2026-11-15 05:54:29 UTCreopenedin-app · delivered
- 2026-11-15 05:54:29 UTCreopenedprod-notifications · sent
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |