workplan.workplan_config: 5 failed runs in the last 6 h — A listed condition would be lost: extracted condition c3 (?) was not given to any step
warning
Background job failing
master
resolved
impact 37
opened 5d ago · last observed 3d ago · observed 20× · #17174 · 666309284c9412c1d74f
What happened template
The background job workplan.workplan_config on master failed 5 runs recently (46 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | workplan.workplan_config |
| service | workplan-agent |
| failed_runs | 5 |
| error_lines | 46 |
| sample | A listed condition would be lost: extracted condition c3 ("Approved sites proceed to Site Selection") was not given to any step |
| error_signature | A listed condition would be lost: extracted condition c3 (?) was not given to any step |
| failing_at | worker/pubsub_manager.py:325 |
| trace | projects/tilda-trial-automation/traces/de20953d11ab9ed9405ea60719fca029 |
| detected_via | ERROR-severity log lines |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-14 16:21:52 UTC → 2026-11-14 22:21:52 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-13 00:09:49 UTCthis finding workplan.workplan_config: 5 failed runs in the last 6 h — A listed condition would be lost: extracted condition c3 (?) was not given to any step
- 2026-11-13 19:37:11 UTCfinding · info feedsvc request rate 27.4 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-11-14 11:10:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-14 13:43:38 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 14:54:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 15:10:23 UTCdeploy automated deployed studiosvc-infra-master
- 2026-11-14 15:15:47 UTCdeploy automated deployed dashql
- 2026-11-14 15:15:49 UTCdeploy automated deployed schedsvc
- 2026-11-14 15:15:50 UTCdeploy automated deployed studiosvc
- 2026-11-14 15:15:58 UTCdeploy automated deployed onboardingsvc
- 2026-11-14 15:16:01 UTCdeploy automated deployed tusker
- 2026-11-14 15:16:06 UTCdeploy automated deployed notifsvc
- 2026-11-14 15:17:18 UTCdeploy automated deployed docsvc
- 2026-11-14 15:17:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 15:17:20 UTCdeploy automated deployed finsvc
- 2026-11-14 15:17:33 UTCdeploy automated deployed dashrsvc
- 2026-11-14 15:17:33 UTCdeploy automated deployed enrollsvc
- 2026-11-14 15:17:48 UTCdeploy automated deployed regsvc
- 2026-11-14 15:19:13 UTCdeploy automated deployed execsvc
- 2026-11-14 15:19:18 UTCdeploy automated deployed patientsvc
- 2026-11-14 16:03:10 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:05:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:08:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:08:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:10:47 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:13:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:13:28 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:15:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:31:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:31:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 16:54:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 17:50:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplan-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-14 19:35:47 UTCdeploy automated deployed docproc
- 2026-11-14 19:37:58 UTCdeploy automated deployed docproc
- 2026-11-14 19:38:05 UTCdeploy automated deployed phase-2-master
- 2026-11-14 20:27:41 UTCdeploy automated deployed docproc
History
- 2026-11-14 18:07:47 UTCreopened x-ray · severity=warning was=warning title=workplan.workplan_config: 8 failed runs in the last 6 h — Plan logic cannot be authored yet: condition c5: unknown reviewMilestoneId ? level=medium was_level=medium finding={'id': 17174, 'fingerprint': '666309284c9412c1d74f', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'workplan.workplan_config', 'title': 'workplan.workplan_config: 8 failed runs in the last 6 h — Plan logic cannot be authored yet: condition c5: unknown reviewMilestoneId ?', 'facts': {'job': 'workplan.workplan_config', 'service': 'workplan-agent', 'failed_runs': 8, 'error_lines': 102, 'sample': "Plan logic cannot be authored yet: condition c5: unknown reviewMilestoneId 'cd331368-9703-44de-4217-b6f7-f5fbd67dc1ac'", 'error_signature': 'Plan logic cannot be authored yet: condition c5: unknown reviewMilestoneId ?', 'failing_at': 'worker/pubsub_manager.py:325', 'trace': 'projects/tilda-trial-automation/traces/901ae2ac96afcc3b6cdf76d9f34de690', 'detected_via': 'ERROR-severity log lines', 'window': [1794658067, 1794679667], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/workplan.workplan_config?cluster=master&from=1794658067&to=1794679667'}, {'label': 'Background jobs', 'href': '/jobs?cluster=master&from=1794658067&to=1794679667'}, {'label': 'Its error logs', 'href': '/logs?cluster=master&from=1794658067&to=1794679667&severity=ERROR&q=workplan.workplan_config'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/901ae2ac96afcc3b6cdf76d9f34de690?cluster=master'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 32.0, 'status': 'open', 'first_seen': 1794528589, 'last_seen': 1794679667, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': "The background job workplan.workplan_config on master failed 6 runs recently (316 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794679667, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 151116, 'stale_s': 38}
- 2026-11-14 23:23:36 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17174, 'fingerprint': '666309284c9412c1d74f', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'master', 'subject': 'workplan.workplan_config', 'title': 'workplan.workplan_config: 5 failed runs in the last 6 h — A listed condition would be lost: extracted condition c3 (?) was not given to any step', 'facts': {'job': 'workplan.workplan_config', 'service': 'workplan-agent', 'failed_runs': 5, 'error_lines': 46, 'sample': 'A listed condition would be lost: extracted condition c3 ("Approved sites proceed to Site Selection") was not given to any step', 'error_signature': 'A listed condition would be lost: extracted condition c3 (?) was not given to any step', 'failing_at': 'worker/pubsub_manager.py:325', 'trace': 'projects/tilda-trial-automation/traces/de20953d11ab9ed9405ea60719fca029', 'detected_via': 'ERROR-severity log lines', 'window': [1794673312, 1794694912], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/workplan.workplan_config?cluster=master&from=1794673312&to=1794694912'}, {'label': 'Background jobs', 'href': '/jobs?cluster=master&from=1794673312&to=1794694912'}, {'label': 'Its error logs', 'href': '/logs?cluster=master&from=1794673312&to=1794694912&severity=ERROR&q=workplan.workplan_config'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-automation/traces/de20953d11ab9ed9405ea60719fca029?cluster=master'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 37.0, 'status': 'resolved', 'first_seen': 1794528589, 'last_seen': 1794694912, 'resolved_at': 1794698616, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 20, 'narrative': "The background job workplan.workplan_config on master failed 5 runs recently (46 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794679667, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 170069, 'stale_s': 3746}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |