Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-21 04:38:29.
› →|
updated 2d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found

warning Background job failing automation resolved impact 31 opened 76d ago · last observed 2d ago · observed 919× · #862 · 39d515061bdabdfe70f3

What happened template

The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (23 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.

The facts

jobpubsub.careplan-visit-automation-finsub
servicefinsvc
failed_runs5
error_lines23
sampleno budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found
error_signatureno budget found for siteID ? and studyID ?: not found
failing_atrepo/report.go:579
traceprojects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540
detected_viaERROR-severity log lines
notea run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.
discriminator
window2026-11-17 03:39:50 UTC → 2026-11-17 09:39:50 UTC

What is failing in finsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from finsvc. In this finding's window (2026-11-17 03:39:50 UTC → 2026-11-17 09:39:50 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

49× Failed to fetch budget for site and study — no budget found for siteID e3958ec8-6ace-404a-958f-37d5137b9683 and studyID … since 2026-10-06 00:51:39 UTC
6× error in pubsub handler — failed to get budget (siteId=97bd332d-afff-4201-adcd-d8600db9d4f7, studyId=79b1b0cc-6e52-41f1-… since 2026-11-17 03:18:36 UTC

All finsvc errors, grouped → · service page · its logs

The story — No finsvc deploy or hand change is recorded in the hour before this opened. Everything below is finsvc only.

History

  • 2026-11-14 23:23:36 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 17, 'sample': 'no budget found for siteID 3fb51b18-50bc-4cae-a24d-02a612dbd141 and studyID 3fca87c5-ae39-41a5-a9c2-88d99b2d5d75: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e57f77b69156ccd2b36faea98a36', 'detected_via': 'ERROR-severity log lines', 'window': [1794677016, 1794698616], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794677016&to=1794698616'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794677016&to=1794698616'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794677016&to=1794698616&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e57f77b69156ccd2b36faea98a36?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794698616, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 886, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 10 runs recently (16 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794698616, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6140267, 'stale_s': 42}
  • 2026-11-15 09:08:59 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 7 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 7, 'error_lines': 24, 'sample': 'no budget found for siteID dc4eb424-c050-4f9d-9299-19db30e29cc8 and studyID 8fe80bec-56a8-4ae0-a9c0-ec549412929c: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e57d78c558540ee8fd66cf1e8398', 'detected_via': 'ERROR-severity log lines', 'window': [1794708258, 1794729858], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794708258&to=1794729858'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794708258&to=1794729858'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794708258&to=1794729858&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e57d78c558540ee8fd66cf1e8398?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794729858, 'resolved_at': 1794733739, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 894, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 7 runs recently (24 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794698616, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6175394, 'stale_s': 3927}
  • 2026-11-15 23:02:23 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 6, 'error_lines': 13, 'sample': 'no budget found for siteID 41bdd8c7-9e27-4964-b06c-091a58a78d5b and studyID 3cb0af22-cf72-4437-9e22-27528ea9bfe9: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e575c5686203f7e9ea6b17a2898c', 'detected_via': 'ERROR-severity log lines', 'window': [1794762143, 1794783743], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794762143&to=1794783743'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794762143&to=1794783743'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794762143&to=1794783743&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e575c5686203f7e9ea6b17a2898c?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794783743, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 895, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 7 runs recently (24 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794783743, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6225402, 'stale_s': 50}
  • 2026-11-16 20:37:48 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 13, 'sample': 'no budget found for siteID 81b6a358-9392-4b3c-8cfc-f384a45b4716 and studyID 4b62c7b1-3ea8-4b9c-9a64-6cbd3f2e7bad: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5789a5370190c333ee3e79686ce', 'detected_via': 'ERROR-severity log lines', 'window': [1794828433, 1794850033], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794828433&to=1794850033'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794828433&to=1794850033'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794828433&to=1794850033&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5789a5370190c333ee3e79686ce?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794850033, 'resolved_at': 1794861468, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 910, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (13 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794783743, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6303105, 'stale_s': 11463}
  • 2026-11-16 23:44:07 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 10 failed runs in the last 6 h — failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 10 failed runs in the last 6 h — failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 10, 'error_lines': 20, 'sample': 'failed to get budget (siteId=ebb982f4-902b-49b4-ae11-66ee92998b41, studyId=3a8e29a3-efdf-4714-8cff-e74088db5982): no budget found for siteID ebb982f4-902b-49b4-ae11-66ee92998b41 and studyID 3a8e29a3-efdf-4714-8cff-e74088db5982: not found', 'error_signature': 'failed to get budget (siteId=?, studyId=?): no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e573b27ea4587fd7f8b91611ddc5', 'detected_via': 'ERROR-severity log lines', 'window': [1794851047, 1794872647], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794851047&to=1794872647'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794851047&to=1794872647'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794851047&to=1794872647&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e573b27ea4587fd7f8b91611ddc5?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794872647, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 911, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (13 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6314286, 'stale_s': 30}
  • 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 6 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 6, 'error_lines': 28, 'sample': 'no budget found for siteID 97bd332d-afff-4201-adcd-d8600db9d4f7 and studyID 79b1b0cc-6e52-41f1-a547-15ba47b624e6: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e579e75dda840ee8fd65b18397e7', 'detected_via': 'ERROR-severity log lines', 'window': [1794878649, 1794900249], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794878649&to=1794900249'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794878649&to=1794900249'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794878649&to=1794900249&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e579e75dda840ee8fd65b18397e7?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794900249, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 918, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 6 runs recently (28 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6345707, 'stale_s': 3849}
  • 2026-11-17 09:39:50 UTCreopened x-ray · severity=warning was=warning title=pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found level=medium was_level=medium finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 23, 'sample': 'no budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540', 'detected_via': 'ERROR-severity log lines', 'window': [1794886790, 1794908390], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794886790&to=1794908390&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 26.0, 'status': 'open', 'first_seen': 1788558391, 'last_seen': 1794908390, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 919, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 6 runs recently (28 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6350104, 'stale_s': 105}
  • 2026-11-17 10:40:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 862, 'fingerprint': '39d515061bdabdfe70f3', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'automation', 'subject': 'pubsub.careplan-visit-automation-finsub', 'title': 'pubsub.careplan-visit-automation-finsub: 5 failed runs in the last 6 h — no budget found for siteID ? and studyID ?: not found', 'facts': {'job': 'pubsub.careplan-visit-automation-finsub', 'service': 'finsvc', 'failed_runs': 5, 'error_lines': 23, 'sample': 'no budget found for siteID e0c746f3-a288-4dad-b1db-36f2ffb07c71 and studyID 1db71f88-a07d-4b35-a12e-2d28e003329e: not found', 'error_signature': 'no budget found for siteID ? and studyID ?: not found', 'failing_at': 'repo/report.go:579', 'trace': 'projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540', 'detected_via': 'ERROR-severity log lines', 'window': [1794886790, 1794908390], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.careplan-visit-automation-finsub?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Background jobs', 'href': '/jobs?cluster=automation&from=1794886790&to=1794908390'}, {'label': 'Its error logs', 'href': '/logs?cluster=automation&from=1794886790&to=1794908390&severity=ERROR&q=pubsub.careplan-visit-automation-finsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/e2e7e5728e284ef82fca432c2032a540?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 31.0, 'status': 'resolved', 'first_seen': 1788558391, 'last_seen': 1794908390, 'resolved_at': 1794912041, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 919, 'narrative': "The background job pubsub.careplan-visit-automation-finsub on automation failed 5 runs recently (23 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794908390, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 6353775, 'stale_s': 3776}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcelive
collected_atNone
age_secondsNone
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteerror-severity job lines from BigQuery, last hour, burst-grouped per job
detectorjob_failing