pubsub.dashboards-test26-dashsub: 6 failed runs in the last 6 h — message type retired, no longer ingested
warning
Background job failing
test26
resolved
impact 25
opened 63d ago · last observed 2d ago · observed 39× · #1698 · 2b88c0f5f279f0bc3ba8
What happened template
The background job pubsub.dashboards-test26-dashsub on test26 failed 6 runs recently (206 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | pubsub.dashboards-test26-dashsub |
| service | dashrsvc |
| failed_runs | 6 |
| error_lines | 206 |
| sample | message type retired, no longer ingested |
| error_signature | message type retired, no longer ingested |
| failing_at | queue/pubsub.go:63 |
| trace | projects/tilda-trial-dev/traces/778d33c9f2017298ad41523edab771a2 |
| detected_via | structured error/exception field (below ERROR severity) |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-16 01:26:53 UTC → 2026-11-16 07:26:53 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-16 08:24:08 UTCcypher schedsvc shape 00000000 p95 went 47 ms → 135 ms across the window (291 runs)
- 2026-09-16 09:24:08 UTCthis finding pubsub.dashboards-test26-dashsub: 6 failed runs in the last 6 h — message type retired, no longer ingested
- 2026-09-24 21:07:06 UTCfinding · info onboardingsvc request rate 8.0 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 13:00 UTC (3 of the last 4 checks)
- 2026-09-29 12:56:10 UTCfinding · warning HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
- 2026-10-05 17:52:16 UTCfinding · info studiosvc Neo4j load 511 q/min vs a normal 0 q/min ± 20 q/min for Sat 11:00 UTC (3 of the last 4 checks)
- 2026-10-09 13:53:22 UTCfinding · warning pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch
- 2026-10-15 20:18:38 UTCfinding · warning dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)
- 2026-10-26 07:19:27 UTCfinding · info execsvc runs 37 foreground Neo4j queries per request (12,049 queries / 327 calls, last hour)
- 2026-10-26 07:28:31 UTCfinding · info enrollsvc Neo4j load 43 q/min vs a normal 3 q/min ± 4 q/min for Tue 06:00 UTC (3 of the last 4 checks)
- 2026-10-27 19:38:41 UTCfinding · info feedsvc Neo4j load 93 q/min vs a normal 0 q/min ± 20 q/min for Tue 00:00 UTC (3 of the last 4 checks)
- 2026-11-14 10:16:00 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 10:51:22 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 11:00:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-14 11:14:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:16:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:52:45 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:16:14 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 14:27:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:27:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:30:14 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:45:39 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 18:24:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:00:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:17:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 20:17:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 23:49:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:24:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 02:25:02 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 05:02:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 05:16:48 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:11:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:32:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:00:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:21:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:29:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 15:35:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 02:09:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:02:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/ontologysvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-16 05:05:00 UTCerrors cluster 5xx peaked at 98 in one five-minute bucket (median 0 in this window)
History
- 2026-11-16 02:09:59 UTCreopened x-ray · severity=warning was=warning title=pubsub.dashboards-test26-dashsub: 5 failed runs in the last 6 h — message type retired, no longer ingested level=medium was_level=medium finding={'id': 1698, 'fingerprint': '2b88c0f5f279f0bc3ba8', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test26', 'subject': 'pubsub.dashboards-test26-dashsub', 'title': 'pubsub.dashboards-test26-dashsub: 5 failed runs in the last 6 h — message type retired, no longer ingested', 'facts': {'job': 'pubsub.dashboards-test26-dashsub', 'service': 'dashrsvc', 'failed_runs': 5, 'error_lines': 7, 'sample': 'message type retired, no longer ingested', 'error_signature': 'message type retired, no longer ingested', 'failing_at': 'queue/pubsub.go:63', 'trace': 'projects/tilda-trial-dev/traces/c4facbeefc81cae5413fdeae29794899', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794773399, 1794794999], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test26-dashsub?cluster=test26&from=1794773399&to=1794794999'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test26&from=1794773399&to=1794794999'}, {'label': 'Its error logs', 'href': '/logs?cluster=test26&from=1794773399&to=1794794999&severity=ERROR&q=pubsub.dashboards-test26-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/c4facbeefc81cae5413fdeae29794899?cluster=test26'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 20.0, 'status': 'open', 'first_seen': 1789550648, 'last_seen': 1794794999, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 34, 'narrative': "The background job pubsub.dashboards-test26-dashsub on test26 failed 5 runs recently (63 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794794999, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 5244406, 'stale_s': 55}
- 2026-11-16 08:28:23 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1698, 'fingerprint': '2b88c0f5f279f0bc3ba8', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test26', 'subject': 'pubsub.dashboards-test26-dashsub', 'title': 'pubsub.dashboards-test26-dashsub: 6 failed runs in the last 6 h — message type retired, no longer ingested', 'facts': {'job': 'pubsub.dashboards-test26-dashsub', 'service': 'dashrsvc', 'failed_runs': 6, 'error_lines': 206, 'sample': 'message type retired, no longer ingested', 'error_signature': 'message type retired, no longer ingested', 'failing_at': 'queue/pubsub.go:63', 'trace': 'projects/tilda-trial-dev/traces/778d33c9f2017298ad41523edab771a2', 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794792413, 1794814013], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.dashboards-test26-dashsub?cluster=test26&from=1794792413&to=1794814013'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test26&from=1794792413&to=1794814013'}, {'label': 'Its error logs', 'href': '/logs?cluster=test26&from=1794792413&to=1794814013&severity=ERROR&q=pubsub.dashboards-test26-dashsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/778d33c9f2017298ad41523edab771a2?cluster=test26'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 25.0, 'status': 'resolved', 'first_seen': 1789550648, 'last_seen': 1794814013, 'resolved_at': 1794817703, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 39, 'narrative': "The background job pubsub.dashboards-test26-dashsub on test26 failed 6 runs recently (206 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794794999, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 5267110, 'stale_s': 3745}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |