workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?
critical
Background job failing
master
open
impact 61
opened 25d ago · last observed 1d ago · observed 545× · #5483 · c8c4cddd279a421a0b6c
What happened template
The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
First look — investigated before anyone looked
workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?.
Cause not clear from logs and recent changes; checked: error lines (none stored), deploys in the 24 h before (12), commits in the deploy's range (0), this finding's history, other findings at the same time (1).
low confidence
claude-opus-5-5 · 1d ago · $0.04 — every link above is evidence it read; nothing else was claimed. the model named pubsub/checkpoint_scoring_sweep.go:115, which the evidence does not hold — its answer was withheld.
The facts
| job | workplan.summarization-sweep |
| service | workplansvc |
| failed_runs | 30 |
| error_lines | 30 |
| sample | checkpoint scoring: unknown checkpoint source type "clinical_trial_data" |
| error_signature | checkpoint scoring: unknown checkpoint source type ? |
| failing_at | pubsub/checkpoint_scoring_sweep.go:115 |
| trace | — |
| detected_via | structured error/exception field (below ERROR severity) |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC |
The story — No workplansvc deploy or hand change is recorded in the hour before this opened. Everything below is workplansvc only.
- 2026-10-23 13:03:15 UTCfinding · info workplansvc: Neo4j queries per request 622.4 — 140.7× the fleet's usual 4.4
- 2026-10-23 13:33:46 UTCthis finding workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?
- 2026-11-14 00:00:00 UTClatency workplansvc p95 peaked at 41.29 s
- 2026-11-15 06:09:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 07:11:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 07:37:36 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:45:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (otel resource detection failed — error detecting resource: partial res)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability: failed to initialize tracing — create otel resource: er)
- 2026-11-15 13:51:32 UTCfinding · warning workplansvc: a new error shape, 7× in the last hour, never seen in 90 days (observability init failed — observability: init tracing: create otel r)
- 2026-11-15 13:51:32 UTCfinding · critical workplansvc on master has restarted 15 times in 10 min — crash loop, image suresh-eng8690-8981379926
- 2026-11-15 13:56:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 15:08:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:06:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 01:27:57 UTCdeploy automated deployed workplansvc
- 2026-11-16 05:19:50 UTCdeploy automated deployed workplansvc
- 2026-11-16 08:03:30 UTCdeploy automated deployed workplansvc
- 2026-11-16 15:37:46 UTCdeploy automated deployed workplansvc
- 2026-11-16 18:21:05 UTCdeploy automated deployed workplansvc
- 2026-11-16 19:33:28 UTCfinding · warning workplansvc shape 252AE86D: p95 184 ms → 313 ms across automated's deploy at 2026-11-16 18:21:05 UTC (p = 1.3e-11, 201/259 runs)
- 2026-11-17 09:07:57 UTCdeploy automated deployed workplansvc
- 2026-11-17 09:23:51 UTCdeploy automated deployed workplansvc
History
- 2026-11-16 17:27:13 UTCescalated x-ray · severity=critical was=warning title=workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ? level=high was_level=high finding={'id': 5483, 'fingerprint': 'c8c4cddd279a421a0b6c', 'kind': 'job.failing', 'severity': 'critical', 'cluster': 'master', 'subject': 'workplan.summarization-sweep', 'title': 'workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?', 'facts': {'job': 'workplan.summarization-sweep', 'service': 'workplansvc', 'failed_runs': 28, 'error_lines': 28, 'sample': 'checkpoint scoring: unknown checkpoint source type "clinical_trial_data"', 'error_signature': 'checkpoint scoring: unknown checkpoint source type ?', 'failing_at': 'pubsub/checkpoint_scoring_sweep.go:115', 'trace': None, 'detected_via': 'structured error/exception field (below ERROR severity)', 'window': [1794828433, 1794850033], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/workplan.summarization-sweep?cluster=master&from=1794828433&to=1794850033'}, {'label': 'Background jobs', 'href': '/jobs?cluster=master&from=1794828433&to=1794850033'}, {'label': 'Its error logs', 'href': '/logs?cluster=master&from=1794828433&to=1794850033&severity=ERROR&q=workplan.summarization-sweep'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 56.0, 'status': 'open', 'first_seen': 1792762426, 'last_seen': 1794850033, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 529, 'narrative': "The background job workplan.summarization-sweep on master failed 23 runs recently (23 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'high', 'told_level': None, 'episode_at': 1792762426, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 2087635, 'stale_s': 28}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |