pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found
warning
Background job failing
automation
resolved
impact 31
opened 76d ago · last observed 3d ago · observed 919× · #862 · 39d515061bdabdfe70f3
What happened template
The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (23 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | pubsub.careplan-visit-automation-finsub |
| service | finsvc |
| failed_runs | 5 |
| error_lines | 23 |
| sample | no budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found |
| error_signature | no budget found for siteID ? and studyID ?: not found |
| failing_at | repo/report.go:579 |
| trace | projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540 |
| detected_via | ERROR-severity log lines |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-17 03:39:50 UTC → 2026-11-17 09:39:50 UTC |
What is failing in finsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from finsvc. In this finding's window (2026-11-17 03:39:50 UTC → 2026-11-17 09:39:50 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 49× | Failed to fetch budget for site and study — no budget found for siteID e3958ec8-6ace-404a-958f-37d5137b9683 and studyID … since 2026-10-06 00:51:39 UTC |
| 6× | error in pubsub handler — failed to get budget (siteId=97bd332d-afff-4201-adcd-d8600db9d4f7, studyId=79b1b0cc-6e52-41f1-… since 2026-11-17 03:18:36 UTC |
The story — No finsvc deploy or hand change is recorded in the hour before this opened. Everything below is finsvc only.
- 2026-09-04 21:46:31 UTCthis finding pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found
- 2026-11-14 18:05:15 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:47 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 00:00:00 UTCerrors finsvc 5xx peaked at 6 in one five-minute bucket (median 0 in this window)
- 2026-11-15 07:17:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:28:09 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:43:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:59:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:46:02 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:56:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 06:49:50 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:02:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:18:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:05:57 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:21:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:32:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:41:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:45:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:48:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:18:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:39:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:49:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 13:44:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 14:03:24 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:53:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 04:16:30 UTCfinding · warning finsvc: a new error shape, 6× in the last hour, never seen in 90 days (error in pubsub handler — failed to get budget (siteId=?, studyId=?): )
- 2026-11-17 04:56:31 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 05:32:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 05:54:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 06:47:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:20:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:37:13 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 07:51:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:26:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 08:50:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:01:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:04:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 09:14:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-17 10:03:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/finsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-14 23:23:36 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 17, 'sample': 'no budget found for siteID 3fb51b18-50bc-4cae-a24d-02a612dbd141 and studyID 3fca87c5-ae39-41a5-a9c2-88d99b2d5d75: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e57f77b69156ccd2b36faea98a36', 'detected_via': 'ERROR-severity log lines', 'window': [1794677016, 1794698616], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794677016&to=1794698616'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794677016&to=1794698616'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794677016&to=1794698616&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e57f77b69156ccd2b36faea98a36?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794698616, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 886, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 10 runs recently (16 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794698616, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6140267, 'stale_s': 42}
- 2026-11-15 09:08:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 7 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 7, 'error_lines': 24, 'sample': 'no budget found for siteID dc4eb424-c050-4f9d-9299-19db30e29cc8 and studyID 8fe80bec-56a8-4ae0-a9c0-ec549412929c: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e57d78c558540ee8fd66cf1e8398', 'detected_via': 'ERROR-severity log lines', 'window': [1794708258, 1794729858], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794708258&to=1794729858'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794708258&to=1794729858'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794708258&to=1794729858&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e57d78c558540ee8fd66cf1e8398?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794729858, 'resolved_at': 1794733739, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 894, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 7 runs recently (24 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794698616, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6175394, 'stale_s': 3927}
- 2026-11-15 23:02:23 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 6, 'error_lines': 13, 'sample': 'no budget found for siteID 41bdd8c7-9e27-4964-b06c-091a58a78d5b and studyID 3cb0af22-cf72-4437-9e22-27528ea9bfe9: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e575c5686203f7e9ea6b17a2898c', 'detected_via': 'ERROR-severity log lines', 'window': [1794762143, 1794783743], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794762143&to=1794783743'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794762143&to=1794783743'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794762143&to=1794783743&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e575c5686203f7e9ea6b17a2898c?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794783743, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 895, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 7 runs recently (24 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794783743, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6225402, 'stale_s': 50}
- 2026-11-16 20:37:48 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 13, 'sample': 'no budget found for siteID 81b6a358-9392-4b3c-8cfc-f384a45b4716 and studyID 4b62c7b1-3ea8-4b9c-9a64-6cbd3f2e7bad: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5789a5370190c333ee3e79686ce', 'detected_via': 'ERROR-severity log lines', 'window': [1794828433, 1794850033], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794828433&to=1794850033'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794828433&to=1794850033'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794828433&to=1794850033&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5789a5370190c333ee3e79686ce?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794850033, 'resolved_at': 1794861468, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 910, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (13 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794783743, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6303105, 'stale_s': 11463}
- 2026-11-16 23:44:07 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 10 failed runs in the last 6 h — failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 10 failed runs in the last 6 h — failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 10, 'error_lines': 20, 'sample': 'failed to get budget (siteId=ebb982f4-902b-49b4-ae11-66ee92998b41, studyId=3a8e29a3-efdf-4714-8cff-e74088db5982): no budget found for siteID ebb982f4-902b-49b4-ae11-66ee92998b41 and studyID 3a8e29a3-efdf-4714-8cff-e74088db5982: not found', 'error_signature': 'failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e573b27ea4587fd7f8b91611ddc5', 'detected_via': 'ERROR-severity log lines', 'window': [1794851047, 1794872647], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794851047&to=1794872647'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794851047&to=1794872647'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794851047&to=1794872647&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e573b27ea4587fd7f8b91611ddc5?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794872647, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 911, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (13 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6314286, 'stale_s': 30}
- 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 6, 'error_lines': 28, 'sample': 'no budget found for siteID 97bd332d-afff-4201-adcd-d8600db9d4f7 and studyID 79b1b0cc-6e52-41f1-a547-15ba47b624e6: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e579e75dda840ee8fd65b18397e7', 'detected_via': 'ERROR-severity log lines', 'window': [1794878649, 1794900249], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794878649&to=1794900249'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794878649&to=1794900249'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794878649&to=1794900249&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e579e75dda840ee8fd65b18397e7?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794900249, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 918, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 6 runs recently (28 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6345707, 'stale_s': 3849}
- 2026-11-17 09:39:50 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 23, 'sample': 'no budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540', 'detected_via': 'ERROR-severity log lines', 'window': [1794886790, 1794908390], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794886790&to=1794908390&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.0, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794908390, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 919, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 6 runs recently (28 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6350104, 'stale_s': 105}
- 2026-11-17 10:40:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 23, 'sample': 'no budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540', 'detected_via': 'ERROR-severity log lines', 'window': [1794886790, 1794908390], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794886790&to=1794908390&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794908390, 'resolved_at': 1794912041, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 919, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (23 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6353775, 'stale_s': 3776}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |