execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)
critical
Error never seen before
master
open
impact 58
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 2d ago · last observed 2d ago · observed 3× · #17470 · e3b60cb9058d48282622
What happened template
execsvc logged “failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[tran” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 16:49 UTC, raised at doclib/context.go:335). Example trace: f91346437409440b0cc533d78b9297f2.
An error shape appearing for the first time in 90 days, five or more times in an hour.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; execsvc hit a burst of Neo4j DeadlockDetected errors at 16:49 UTC while copying documents.
This finding is 5× 'failed to add document to scope binders' (doclib/context.go:335). Same burst: 14× tildadb/util.go:993, 12× bulk-write (doclib/createDocument.go:1015), 12× + 5× copy failures (doclib/copyDocument.go:74).
All lines are from 16:49 UTC only. The util.go:993 deadlock shape was already seen on 2026-11-15, so deadlocks are not new.
Checked: the 5 deploys in the prior 24 h were config-only autosyncs (latest 15:39, 69 min before) with no new commits. Nothing links them to this burst.
Next step: open the example trace to see which concurrent document copies collided on the same nodes. Check whether the copy path retries Neo4j TransientErrors.
low confidence
claude-opus-5-5 · 2d ago · $0.05 — every link above is evidence it read; nothing else was claimed.
The facts
| service | execsvc |
| caller | doclib/context.go:335 |
| shape | failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't ac |
| count | 5 |
| error_class | neo4j |
| first_seen | 2026-11-16 16:49:02 UTC (2d ago) |
| sample | failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=2457665, clientId=4383] can't ac |
| trace | f91346437409440b0cc533d78b9297f2 |
| shape_hash | abd4b9fac947d33f |
| count_is_floor | no |
| discriminator | abd4b9fac947d33f |
| window | 2026-11-16 16:34:53 UTC → 2026-11-16 17:34:53 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 16:34:53 UTC → 2026-11-16 17:34:53 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 7× | error anchor visit index has not been set. Does this study have an anchor visit? continuing without setting day/date ran… since 2026-11-12 15:42:02 UTC |
| 6× | could not list DutyPlans for Study 85333258-21d0-4c47-92b2-da3a9fa8428f: passed Study was not found since 2026-10-12 17:57:06 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-11-16 16:00:00 UTClatency execsvc p95 peaked at 1.51 s
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to create document copy — could not write document node and cor)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 12× in the last hour, never seen in 90 days (failed to bulk-write document nodes — Neo4jError: Neo.TransientError.T)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to create document copy — could not add document context: could)
- 2026-11-16 16:55:04 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to run cypher in transaction — Neo4jError: Neo.TransientError.T)
- 2026-11-16 16:55:04 UTCthis finding execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)
History
- 2026-11-16 16:55:04 UTCopened x-ray · severity=critical title=execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop) finding={'id': 17470, 'fingerprint': 'e3b60cb9058d48282622', 'kind': 'error.novel', 'severity': 'critical', 'cluster': 'master', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 5× in the last hour, never seen in 90 days (failed to add document to scope binders — could not write context scop)', 'facts': {'service': 'execsvc', 'caller': 'doclib/context.go:335', 'shape': "failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=?, clientId=?] can't ac", 'count': 5, 'error_class': 'neo4j', 'first_seen': 1794847742, 'sample': "failed to add document to scope binders — could not write context scope binders: Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=2457665, clientId=4383] can't ac", 'trace': 'f91346437409440b0cc533d78b9297f2', 'shape_hash': 'abd4b9fac947d33f', 'window': [1794844504, 1794848104], 'count_is_floor': False, 'discriminator': 'abd4b9fac947d33f'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=master&from=1794844504&to=1794848104&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=master&from=1794844504&to=1794848104&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/f91346437409440b0cc533d78b9297f2?cluster=master'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 60.0, 'status': 'open', 'first_seen': 1794848104, 'last_seen': 1794848104, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'abd4b9fac947d33f', 'level': 'medium', 'told_level': None, 'episode_at': 1794848104, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 134, 'stale_s': 134}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |