ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h
warning
Autoscaler blind
production
open
impact 35
opened 49d ago · last observed 1d ago · observed 539× · #2939 · 6bafa4a6c7f1ac2f22bb
What happened template
Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s.
A KEDA scaler or HPA that cannot read its metric — it will not scale when load changes.
The facts
| namespace | tilda |
| object | ScaledJob/rag-agent-worker |
| failures_2h | 2 |
| last_message | could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s |
| discriminator |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-29 23:04:32 UTCthis finding ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h
- 2026-10-03 04:31:35 UTCfinding · warning HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
- 2026-10-13 15:54:07 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
- 2026-10-18 18:16:26 UTCfinding · critical docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
- 2026-10-19 06:46:26 UTCfinding · critical notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
- 2026-10-23 13:03:15 UTCfinding · info virtual-assistant: Neo4j queries per request 83.8 — 14.0× the fleet's usual 6.0
- 2026-10-23 13:42:32 UTCfinding · info dashql: p95 latency (ms) 3.77 s — 5.2× the fleet's usual 726 ms
- 2026-10-23 13:49:17 UTCfinding · info virtual-assistant Neo4j load 413 q/min vs a normal 4 q/min ± 4 q/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-10-23 14:04:30 UTCfinding · info docsvc: p99 latency (ms) 3.88 s — 4.7× the fleet's usual 825 ms
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
- 2026-11-17 00:20:00 UTCerrors cluster 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
History
- 2026-11-16 16:32:01 UTCreopened x-ray · severity=warning was=warning title=ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h level=medium was_level=medium finding={'id': 2939, 'fingerprint': '6bafa4a6c7f1ac2f22bb', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794843121&to=1794846721&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.7, 'status': 'open', 'first_seen': 1790723072, 'last_seen': 1794846721, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 526, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794846721, 'chat_thread': 'xray-alert-11-748fd5a928-6bafa4a6c7f1ac2f22bb-1794465595', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4123672, 'stale_s': 23}
- 2026-11-16 18:39:47 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2939, 'fingerprint': '6bafa4a6c7f1ac2f22bb', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794849867&to=1794853467&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 34.7, 'status': 'resolved', 'first_seen': 1790723072, 'last_seen': 1794853467, 'resolved_at': 1794854387, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 538, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794846721, 'chat_thread': 'xray-alert-11-748fd5a928-6bafa4a6c7f1ac2f22bb-1794465595', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4131337, 'stale_s': 942}
- 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h level=medium was_level=medium finding={'id': 2939, 'fingerprint': '6bafa4a6c7f1ac2f22bb', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=production&from=1794908441&to=1794912041&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 29.7, 'status': 'open', 'first_seen': 1790723072, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 539, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-production' && resource.subscription_id == 'tilda-document-production-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794912041, 'chat_thread': 'xray-alert-11-748fd5a928-6bafa4a6c7f1ac2f22bb-1794465595', 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 4189078, 'stale_s': 109}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | Kubernetes events (logs/events in BigQuery), last 45 min |
| detector | k8s_events |