pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch
warning
Background job failing
test26
resolved
impact 25
opened 40d ago · last observed 1d ago · observed 99× · #4291 · 48844bdd842df0e41b88
What happened template
The background job pubsub.study-smo-binding-test26-enrollsub on test26 failed 5 runs recently (4,845 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | pubsub.study-smo-binding-test26-enrollsub |
| service | enrollsvc |
| failed_runs | 5 |
| error_lines | 4,845 |
| sample | Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exists with label `ChartReviewStudy` and properties `orgid` = '1ac69569-e12c-4606-909e-8c099b6b47bd', `studyid` = 'd4ccf900-31a7-47fc-83bf-37afc98f7217', `deltimestamp` = 0) |
| error_signature | Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `ChartReviewStudy` and |
| failing_at | tildadb/util.go:993 |
| trace | projects/tilda-trial-dev/traces/9d57d9b2341bd1acb13ff8f20fd7cc2c |
| detected_via | ERROR-severity log lines |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-17 01:24:09 UTC → 2026-11-17 07:24:09 UTC |
What is failing in enrollsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from enrollsvc. In this finding's window (2026-11-17 01:24:09 UTC → 2026-11-17 07:24:09 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 1566× | could not add chart review study node while binding study to org — Neo4jError: Neo.ClientError.Schema.ConstraintValidati… since 2026-10-09 11:44:30 UTC |
| 19× | failed to create node — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12181421) already exists wit… since 2026-10-30 19:23:45 UTC |
| 19× | failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12181421) already exis… since 2026-10-30 19:23:45 UTC |
| 18× | failed to create node — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exists wit… since 2026-11-02 07:19:50 UTC |
| 18× | failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exis… since 2026-11-02 07:19:50 UTC |
The story — No enrollsvc deploy or hand change is recorded in the hour before this opened. Everything below is enrollsvc only.
- 2026-10-09 13:53:22 UTCthis finding pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch
- 2026-10-26 07:28:31 UTCfinding · info enrollsvc Neo4j load 43 q/min vs a normal 3 q/min ± 4 q/min for Tue 06:00 UTC (3 of the last 4 checks)
- 2026-11-16 06:22:27 UTCfinding · warning enrollsvc: a new error shape, 1081× in the last hour, never seen in 90 days (could not add chart review study node while binding study to org — Neo)
- 2026-11-16 06:22:27 UTCfinding · warning enrollsvc: a new error shape, 18× in the last hour, never seen in 90 days (failed to create node — Neo4jError: Neo.ClientError.Schema.ConstraintV)
- 2026-11-16 06:22:27 UTCfinding · warning enrollsvc: a new error shape, 18× in the last hour, never seen in 90 days (could not process study smo org binding change — Neo4jError: Neo.Clien)
- 2026-11-16 06:22:27 UTCfinding · warning enrollsvc: a new error shape, 18× in the last hour, never seen in 90 days (failed to process db result — Neo4jError: Neo.ClientError.Schema.Const)
- 2026-11-16 14:01:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-16 14:48:32 UTCfinding · warning enrollsvc: a new error shape, 8× in the last hour, never seen in 90 days (could not process study smo org binding change — Neo4jError: Neo.Clien)
- 2026-11-16 14:48:32 UTCfinding · warning enrollsvc: a new error shape, 8× in the last hour, never seen in 90 days (could not process study smo org binding change — Neo4jError: Neo.Clien)
- 2026-11-16 14:48:32 UTCfinding · warning enrollsvc: a new error shape, 8× in the last hour, never seen in 90 days (could not process study smo org binding change — Neo4jError: Neo.Clien)
History
- 2026-11-16 17:27:13 UTCreopened x-ray · severity=warning was=warning title=pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch level=medium was_level=medium finding={'id': 4291, 'fingerprint': '48844bdd842df0e41b88', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test26', 'subject': 'pubsub.study-smo-binding-test26-enrollsub', 'title': 'pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch', 'facts': {'job': 'pubsub.study-smo-binding-test26-enrollsub', 'service': 'enrollsvc', 'failed_runs': 5, 'error_lines': 5525, 'sample': "Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619347) already exists with label `ChartReviewStudy` and properties `orgid` = '4f9df213-aa1d-40dd-8fe3-7ec3239d7af5', `studyid` = 'd4ccf900-31a7-47fc-83bf-37afc98f7217', `deltimestamp` = 0)", 'error_signature': 'Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `ChartReviewStudy` and', 'failing_at': 'tildadb/util.go:993', 'trace': 'projects/tilda-trial-dev/traces/9f3df23f82582d60dfc222e734952007', 'detected_via': 'ERROR-severity log lines', 'window': [1794828433, 1794850033], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.study-smo-binding-test26-enrollsub?cluster=test26&from=1794828433&to=1794850033'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test26&from=1794828433&to=1794850033'}, {'label': 'Its error logs', 'href': '/logs?cluster=test26&from=1794828433&to=1794850033&severity=ERROR&q=pubsub.study-smo-binding-test26-enrollsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/9f3df23f82582d60dfc222e734952007?cluster=test26'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 20.2, 'status': 'open', 'first_seen': 1791554002, 'last_seen': 1794850033, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 86, 'narrative': "The background job pubsub.study-smo-binding-test26-enrollsub on test26 failed 5 runs recently (8,200 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794850033, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 3296065, 'stale_s': 34}
- 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4291, 'fingerprint': '48844bdd842df0e41b88', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test26', 'subject': 'pubsub.study-smo-binding-test26-enrollsub', 'title': 'pubsub.study-smo-binding-test26-enrollsub: 5 failed runs in the last 6 h — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `Ch', 'facts': {'job': 'pubsub.study-smo-binding-test26-enrollsub', 'service': 'enrollsvc', 'failed_runs': 5, 'error_lines': 4845, 'sample': "Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exists with label `ChartReviewStudy` and properties `orgid` = '1ac69569-e12c-4606-909e-8c099b6b47bd', `studyid` = 'd4ccf900-31a7-47fc-83bf-37afc98f7217', `deltimestamp` = 0)", 'error_signature': 'Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(?) already exists with label `ChartReviewStudy` and', 'failing_at': 'tildadb/util.go:993', 'trace': 'projects/tilda-trial-dev/traces/9d57d9b2341bd1acb13ff8f20fd7cc2c', 'detected_via': 'ERROR-severity log lines', 'window': [1794878649, 1794900249], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/pubsub.study-smo-binding-test26-enrollsub?cluster=test26&from=1794878649&to=1794900249'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test26&from=1794878649&to=1794900249'}, {'label': 'Its error logs', 'href': '/logs?cluster=test26&from=1794878649&to=1794900249&severity=ERROR&q=pubsub.study-smo-binding-test26-enrollsub'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/9d57d9b2341bd1acb13ff8f20fd7cc2c?cluster=test26'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 25.4, 'status': 'resolved', 'first_seen': 1791554002, 'last_seen': 1794900249, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 99, 'narrative': "The background job pubsub.study-smo-binding-test26-enrollsub on test26 failed 5 runs recently (4,845 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794850033, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 3350098, 'stale_s': 3851}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |