Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 11:39:34.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h

warning Autoscaler blind master open impact 29 opened 25d ago · last observed 1d ago · observed 3259× · #5452 · 8824a0da9cf312cf0ed2

What happened template

Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 35, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.

A KEDA scaler or HPA that cannot read its metric — it will not scale when load changes.

The facts

namespacetilda
objectScaledJob/rag-agent-worker
failures_2h35
last_messagecould not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s
discriminator

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-15 05:41:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5452, 'fingerprint': '8824a0da9cf312cf0ed2', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794716548&to=1794720148&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 28.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794720148, 'resolved_at': 1794721319, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2938, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794635047, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 1960735, 'stale_s': 1182}
  • 2026-11-15 06:59:37 UTCreopened x-ray · severity=warning was=warning title=ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h level=medium was_level=medium finding={'id': 5452, 'fingerprint': '8824a0da9cf312cf0ed2', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794722377&to=1794725977&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 23.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794725977, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2939, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794725977, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 1965416, 'stale_s': 34}
  • 2026-11-16 13:13:39 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5452, 'fingerprint': '8824a0da9cf312cf0ed2', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 2, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794830306&to=1794833906&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 28.8, 'status': 'resolved', 'first_seen': 1792760595, 'last_seen': 1794833906, 'resolved_at': 1794834819, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3146, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794725977, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2074245, 'stale_s': 934}
  • 2026-11-16 14:10:27 UTCreopened x-ray · severity=warning was=warning title=ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 4× in 2 h level=medium was_level=medium finding={'id': 5452, 'fingerprint': '8824a0da9cf312cf0ed2', 'kind': 'scaler.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'tilda/ScaledJob/rag-agent-worker', 'title': 'ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 4× in 2 h', 'facts': {'namespace': 'tilda', 'object': 'ScaledJob/rag-agent-worker', 'failures_2h': 4, 'last_message': "could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s", 'discriminator': ''}, 'evidence': [{'label': 'Kubernetes events', 'href': '/logs?cluster=master&from=1794834627&to=1794838227&source=events&q=ScaledJob%2Frag-agent-worker'}, {'label': 'Pods', 'href': '/pods?cluster=master'}], 'provenance': {'source': 'x-ray', 'note': 'Kubernetes events (logs/events in BigQuery), last 45 min'}, 'impact': 23.8, 'status': 'open', 'first_seen': 1792760595, 'last_seen': 1794838227, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3147, 'narrative': "Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 2, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794838227, 'chat_thread': None, 'kind_label': 'Autoscaler blind', 'group': 'Platform', 'age_s': 2077652, 'stale_s': 20}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
noteKubernetes events (logs/events in BigQuery), last 45 min
detectork8s_events