execsvc: a new error shape, 150× in the last hour, never seen in 90 days (no result from branch evaluation)
critical
Error never seen before
delta
resolved
impact 66
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 1d ago · last observed 1d ago · observed 3× · #17479 · aa35934b3ff55d52fb41
What happened template
execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.
An error shape appearing for the first time in 90 days, five or more times in an hour.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; this looks like an existing error that moved, not a new one.
execsvc logged 'no result from branch evaluation' 101× at execlib/branchV2.go:91 from 18:42 to 19:13 UTC, plus 'evalResult is not a bool' 38× at branchV2.go:96 from 18:51 UTC.
The same two messages came from branchV2.go:77 (106×, seen since Nov 7) and :82 (4×) until 18:31 UTC, then stopped. The activityV2.go:882 wrapper lines also stopped at 18:31.
That 14-line shift suggests a new execsvc build rolled out around 18:31–18:42, but the only recorded deploy is [email protected] at 05:11 UTC, 810 min earlier. Code and commits could not be read because the GitHub budget ran out.
Next step: find what execsvc version started running around 18:31–18:42 UTC, diff branchV2.go, and check why branch evaluation returns no result (failing since Nov 7).
low confidence
claude-opus-5-5 · 1d ago · $0.04 — every link above is evidence it read; nothing else was claimed.
The facts
| service | execsvc |
| caller | execlib/branchV2.go:91 |
| shape | no result from branch evaluation |
| count | 150 |
| error_class | — |
| first_seen | 2026-11-16 18:42:43 UTC (1d ago) |
| sample | no result from branch evaluation |
| trace | 76e7e01f9377ba990c4164bcd594d9c8 |
| shape_hash | c6bc871f49723776 |
| count_is_floor | no |
| discriminator | c6bc871f49723776 |
| window | 2026-11-16 18:33:28 UTC → 2026-11-16 19:33:28 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 18:33:28 UTC → 2026-11-16 19:33:28 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 421× | no result from branch evaluation since 2026-11-16 18:42:43 UTC |
| 247× | no result from branch evaluation since 2026-11-07 14:16:43 UTC |
| 247× | failed to evaluate and raise auto query for participant question — no result from branch evaluation for branch, protocol… since 2026-11-16 14:20:18 UTC |
| 117× | evalResult is not a bool since 2026-11-16 18:51:49 UTC |
| 50× | evalResult is not a bool since 2026-11-16 14:20:12 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-11-16 17:54:59 UTCcypher execsvc shape B0E839E0 p95 went 18 ms → 74 ms across the window (260 runs)
- 2026-11-16 18:39:47 UTCfinding · info execsvc: p95 latency (ms) 16.90 s — 37.9× the fleet's usual 446 ms (also p99 latency (ms) 1.6 min; Neo4j queries per request 76.9)
- 2026-11-16 18:54:59 UTCthis finding execsvc: a new error shape, 150× in the last hour, never seen in 90 days (no result from branch evaluation)
- 2026-11-16 19:00:00 UTClatency execsvc p95 peaked at 32.10 s
- 2026-11-16 19:14:41 UTCfinding · critical execsvc: a new error shape, 55× in the last hour, never seen in 90 days (evalResult is not a bool)
History
- 2026-11-16 18:54:59 UTCopened x-ray · severity=warning title=execsvc: a new error shape, 41× in the last hour, never seen in 90 days (no result from branch evaluation) finding={'id': 17479, 'fingerprint': 'aa35934b3ff55d52fb41', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'delta', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 41× in the last hour, never seen in 90 days (no result from branch evaluation)', 'facts': {'service': 'execsvc', 'caller': 'execlib/branchV2.go:91', 'shape': 'no result from branch evaluation', 'count': 41, 'error_class': None, 'first_seen': 1794854563, 'sample': 'no result from branch evaluation', 'trace': '76e7e01f9377ba990c4164bcd594d9c8', 'shape_hash': 'c6bc871f49723776', 'window': [1794851699, 1794855299], 'count_is_floor': False, 'discriminator': 'c6bc871f49723776'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=delta&from=1794851699&to=1794855299&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=delta&from=1794851699&to=1794855299&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/76e7e01f9377ba990c4164bcd594d9c8?cluster=delta'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 37.2, 'status': 'open', 'first_seen': 1794855299, 'last_seen': 1794855299, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'c6bc871f49723776', 'level': 'medium', 'told_level': None, 'episode_at': 1794855299, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 32, 'stale_s': 32}
- 2026-11-16 19:14:41 UTCescalated x-ray · severity=critical was=warning title=execsvc: a new error shape, 101× in the last hour, never seen in 90 days (no result from branch evaluation) level=high was_level=medium finding={'id': 17479, 'fingerprint': 'aa35934b3ff55d52fb41', 'kind': 'error.novel', 'severity': 'critical', 'cluster': 'delta', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 101× in the last hour, never seen in 90 days (no result from branch evaluation)', 'facts': {'service': 'execsvc', 'caller': 'execlib/branchV2.go:91', 'shape': 'no result from branch evaluation', 'count': 101, 'error_class': None, 'first_seen': 1794854563, 'sample': 'no result from branch evaluation', 'trace': '76e7e01f9377ba990c4164bcd594d9c8', 'shape_hash': 'c6bc871f49723776', 'window': [1794852881, 1794856481], 'count_is_floor': False, 'discriminator': 'c6bc871f49723776'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=delta&from=1794852881&to=1794856481&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=delta&from=1794852881&to=1794856481&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/76e7e01f9377ba990c4164bcd594d9c8?cluster=delta'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 64.7, 'status': 'open', 'first_seen': 1794855299, 'last_seen': 1794856481, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'execsvc logged "no result from branch evaluation" 41 times in the last hour, with no prior occurrences in 90 days on delta. First seen 2026-11-16 18:42 UTC at execlib/branchV2.go:91. Example trace: 76e7e01f9377ba990c4164bcd594d9c8.', 'narrative_kind': 'ai', 'discriminator': 'c6bc871f49723776', 'level': 'high', 'told_level': None, 'episode_at': 1794855299, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 1216, 'stale_s': 34}
- 2026-11-16 19:57:55 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17479, 'fingerprint': 'aa35934b3ff55d52fb41', 'kind': 'error.novel', 'severity': 'critical', 'cluster': 'delta', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 150× in the last hour, never seen in 90 days (no result from branch evaluation)', 'facts': {'service': 'execsvc', 'caller': 'execlib/branchV2.go:91', 'shape': 'no result from branch evaluation', 'count': 150, 'error_class': None, 'first_seen': 1794854563, 'sample': 'no result from branch evaluation', 'trace': '76e7e01f9377ba990c4164bcd594d9c8', 'shape_hash': 'c6bc871f49723776', 'window': [1794854008, 1794857608], 'count_is_floor': False, 'discriminator': 'c6bc871f49723776'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=delta&from=1794854008&to=1794857608&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=delta&from=1794854008&to=1794857608&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/76e7e01f9377ba990c4164bcd594d9c8?cluster=delta'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 65.6, 'status': 'resolved', 'first_seen': 1794855299, 'last_seen': 1794857608, 'resolved_at': 1794859075, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 3, 'narrative': 'execsvc logged “no result from branch evaluation” 150 times in the last hour and never before in 90 days on delta (first seen 2026-11-16 18:42 UTC, raised at execlib/branchV2.go:91). Example trace: 76e7e01f9377ba990c4164bcd594d9c8.', 'narrative_kind': 'template', 'discriminator': 'c6bc871f49723776', 'level': 'high', 'told_level': None, 'episode_at': 1794855299, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 3813, 'stale_s': 1504}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |