pubsub.dashboards-production-dashsub: 14 failed runs in the last 6 h — message type retired, no longer ingested
warning
Background job failing
production
open
impact 45
opened 72d ago · last observed 1d ago · observed 572× · #1003 · 7b583610b88f0d8fc9df
What happened template
The background job pubsub.dashboards-production-dashsub on production failed 14 runs recently (123 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | pubsub.dashboards-production-dashsub |
| service | dashrsvc |
| failed_runs | 14 |
| error_lines | 123 |
| sample | message type retired, no longer ingested |
| error_signature | message type retired, no longer ingested |
| failing_at | queue/pubsub.go:63 |
| trace | projects/tilda-trial-production/traces/b451e82b8f2806d41c09fcf4d3db371d |
| detected_via | structured error/exception field (below ERROR severity) |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-07 08:20:32 UTCthis finding pubsub.dashboards-production-dashsub: 14 failed runs in the last 6 h — message type retired, no longer ingested
- 2026-09-08 06:59:31 UTCfinding · info dashrsvc Neo4j load 61 q/min vs a normal 0 q/min ± 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-09-08 21:04:38 UTCfinding · info onboardingsvc request rate 7.8 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 23:00 UTC (3 of the last 4 checks)
- 2026-09-09 06:20:40 UTCfinding · info docproc request rate 5.6 req/min vs a normal 0.4 req/min ± 0.6 req/min for Mon 18:00 UTC (3 of the last 4 checks)
- 2026-09-09 06:47:08 UTCfinding · info onboardingsvc Neo4j load 34 q/min vs a normal 2 q/min ± 3 q/min for Mon 12:00 UTC (3 of the last 4 checks)
- 2026-09-09 12:26:51 UTCfinding · info execsvc p95 4.15 s vs a normal 462 ms ± 398 ms for Mon 20:00 UTC (3 of the last 4 checks)
- 2026-09-09 14:37:46 UTCfinding · info docsvc p95 2.47 s vs a normal 490 ms ± 174 ms for Mon 15:00 UTC (3 of the last 4 checks)
- 2026-09-10 13:53:27 UTCfinding · info studiosvc Neo4j load 46 q/min vs a normal 0 q/min ± 1 q/min for Mon 16:00 UTC (3 of the last 4 checks)
- 2026-09-11 21:51:42 UTCfinding · warning onboardingsvc is burning its error budget at 0.0× (1 h) / 13.3× (6 h) against a 99.50% objective
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
- 2026-11-17 00:20:00 UTCerrors cluster 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
History
- 2026-11-16 07:26:53 UTCreopened x-ray · severity=warning was=warning title=pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ? level=medium was_level=medium finding={'id': 1003, 'fingerprint': '7b583610b88f0d8fc9df', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'pubsub.dashboards-production-dashsub', 'title': 'pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ?', 'facts': {'job': 'pubsub.dashboards-production-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 27, 'sample': 'no data found for label RecommendationByDocument, id 0d2b5f93-69f7-4725-a10b-69b19e08e475', 'error_signature': 'no data found for label RecommendationByDocument, id ?', 'failing_at': 'db/neo4j.go:83', 'trace': 'projects/tilda-trial-production/traces/33642b699b8bf687a2f32147f15c6f64', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794792413, 1794814013], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-production-dashsub?cluster=production&from=1794792413&to=1794814013'}, {'label': 'Background jobs', 'href': '/jobs?cluster=production&from=1794792413&to=1794814013'}, {'label': 'Its error logs', 'href': '/logs?cluster=production&from=1794792413&to=1794814013&severity=ERROR&q=pubsub.dashboards-production-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-production/traces/33642b699b8bf687a2f32147f15c6f64?cluster=production'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 40.0, 'status': 'open', 'first_seen': 1788769232, 'last_seen': 1794814013, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 553, 'narrative': "The background job pubsub.dashboards-production-dashsub on production failed 5 runs recently (58 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794814013, 'chat_thread': 'xray-alert-10-748fd5a928-7b583610b88f0d8fc9df-1794315719', 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6044831, 'stale_s': 50}
- 2026-11-16 07:33:30 UTCnotified x-ray · alert_id=11 channel=in-app status=delivered send_id=51500
- 2026-11-16 07:33:30 UTCnotified x-ray · alert_id=11 channel=prod-notifications status=sent send_id=51500
- 2026-11-16 17:27:13 UTCescalated x-ray · severity=warning was=warning title=pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested level=high was_level=medium finding={'id': 1003, 'fingerprint': '7b583610b88f0d8fc9df', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'pubsub.dashboards-production-dashsub', 'title': 'pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested', 'facts': {'job': 'pubsub.dashboards-production-dashsub', 'service': 'dashrsvc', 'failed_runs': 22, 'error_lines': 326, 'sample': 'message type retired, no longer ingested', 'error_signature': 'message type retired, no longer ingested', 'failing_at': 'queue/pubsub.go:63', 'trace': 'projects/tilda-trial-production/traces/79f7473c854f50c10a0ef1de6a29ef98', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794828433, 1794850033], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-production-dashsub?cluster=production&from=1794828433&to=1794850033'}, {'label': 'Background jobs', 'href': '/jobs?cluster=production&from=1794828433&to=1794850033'}, {'label': 'Its error logs', 'href': '/logs?cluster=production&from=1794828433&to=1794850033&severity=ERROR&q=pubsub.dashboards-production-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-production/traces/79f7473c854f50c10a0ef1de6a29ef98?cluster=production'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 40.0, 'status': 'open', 'first_seen': 1788769232, 'last_seen': 1794850033, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 561, 'narrative': "The background job pubsub.dashboards-production-dashsub on production failed 18 runs recently (248 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'high', 'told_level': 'medium', 'episode_at': 1794814013, 'chat_thread': 'xray-alert-11-748fd5a928-7b583610b88f0d8fc9df-1794814409', 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6080832, 'stale_s': 31}
- 2026-11-16 17:28:30 UTCnotified x-ray · alert_id=10 channel=in-app status=delivered send_id=51845
- 2026-11-16 17:28:30 UTCnotified x-ray · alert_id=10 channel=prod-notifications status=sent send_id=51845
- 2026-11-17 04:00:15 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1003, 'fingerprint': '7b583610b88f0d8fc9df', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'pubsub.dashboards-production-dashsub', 'title': 'pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested', 'facts': {'job': 'pubsub.dashboards-production-dashsub', 'service': 'dashrsvc', 'failed_runs': 7, 'error_lines': 39, 'sample': 'message type retired, no longer ingested', 'error_signature': 'message type retired, no longer ingested', 'failing_at': 'queue/pubsub.go:63', 'trace': 'projects/tilda-trial-production/traces/042f8bdf0b5fcf684baf03cda023aec6', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794862484, 1794884084], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-production-dashsub?cluster=production&from=1794862484&to=1794884084'}, {'label': 'Background jobs', 'href': '/jobs?cluster=production&from=1794862484&to=1794884084'}, {'label': 'Its error logs', 'href': '/logs?cluster=production&from=1794862484&to=1794884084&severity=ERROR&q=pubsub.dashboards-production-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-production/traces/042f8bdf0b5fcf684baf03cda023aec6?cluster=production'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 45.0, 'status': 'resolved', 'first_seen': 1788769232, 'last_seen': 1794884084, 'resolved_at': 1794888015, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 570, 'narrative': "The background job pubsub.dashboards-production-dashsub on production failed 7 runs recently (39 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794814013, 'chat_thread': 'xray-alert-10-748fd5a928-g-1794850110', 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6118828, 'stale_s': 3976}
- 2026-11-17 05:30:00 UTCnotified x-ray · alert_id=10 channel=in-app status=delivered send_id=52016
- 2026-11-17 05:30:00 UTCnotified x-ray · alert_id=10 channel=prod-notifications status=sent send_id=52016
- 2026-11-17 07:35:31 UTCnotified x-ray · alert_id=11 channel=in-app status=delivered send_id=52045
- 2026-11-17 07:35:31 UTCnotified x-ray · alert_id=11 channel=prod-notifications status=sent send_id=52045
- 2026-11-17 09:39:50 UTCreopened x-ray · severity=warning was=warning title=pubsub.dashboards-production-dashsub: 8 failed runs in the last 6 h — message type retired, no longer ingested level=medium was_level=medium finding={'id': 1003, 'fingerprint': '7b583610b88f0d8fc9df', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'production', 'subject': 'pubsub.dashboards-production-dashsub', 'title': 'pubsub.dashboards-production-dashsub: 8 failed runs in the last 6 h — message type retired, no longer ingested', 'facts': {'job': 'pubsub.dashboards-production-dashsub', 'service': 'dashrsvc', 'failed_runs': 8, 'error_lines': 112, 'sample': 'message type retired, no longer ingested', 'error_signature': 'message type retired, no longer ingested', 'failing_at': 'queue/pubsub.go:63', 'trace': 'projects/tilda-trial-production/traces/31965328bfd6a801837587490a93ba76', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794886790, 1794908390], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-production-dashsub?cluster=production&from=1794886790&to=1794908390'}, {'label': 'Background jobs', 'href': '/jobs?cluster=production&from=1794886790&to=1794908390'}, {'label': 'Its error logs', 'href': '/logs?cluster=production&from=1794886790&to=1794908390&severity=ERROR&q=pubsub.dashboards-production-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-production/traces/31965328bfd6a801837587490a93ba76?cluster=production'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 40.0, 'status': 'open', 'first_seen': 1788769232, 'last_seen': 1794908390, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 571, 'narrative': "The background job pubsub.dashboards-production-dashsub on production failed 7 runs recently (39 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': 'high', 'episode_at': 1794908390, 'chat_thread': 'xray-alert-10-748fd5a928-g-1794850110', 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6139264, 'stale_s': 106}
Notifications
- 2026-11-17 07:35:31 UTCresolvedin-app · delivered
- 2026-11-17 07:35:31 UTCresolvedprod-notifications · sent
- 2026-11-17 05:30:00 UTCresolvedin-app · delivered
- 2026-11-17 05:30:00 UTCresolvedprod-notifications · sent
- 2026-11-16 07:33:30 UTCreopenedin-app · delivered
- 2026-11-16 07:33:29 UTCreopenedprod-notifications · sent
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |