execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
critical
Error never seen before
automation
open
impact 47
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 2d ago · last observed 2d ago · observed 3× · #17379 · bb4d70526bc27ac1bd1f
What happened template
execsvc logged “Request completed — ? · {?:" failed to create Actor for Study ? Actor {? [{{ 5c4ac420-2d4d-4efe-b649-985c” 5 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:11 UTC, raised at observability/middleware.go:281). Example trace: e2e7e57fcb03b44000000000de379caf.
An error shape appearing for the first time in 90 days, five or more times in an hour.
The facts
| service | execsvc |
| caller | observability/middleware.go:281 |
| shape | Request completed — ? · {?:" failed to create Actor for Study ? Actor {? [{{ 5c4ac420-2d4d-4efe-b649-985c |
| count | 5 |
| error_class | server_error |
| first_seen | 2026-11-16 08:11:39 UTC (2d ago) |
| sample | Request completed — 500 · {"detail":" failed to create Actor for Study 6305ace2-5a30-413d-a77c-0dc79a9984da Actor {49d30832-2209-40dc-84d0-be390a95d87c [{{ 5c4ac420-2d4d-4efe-b649-985c |
| trace | e2e7e57fcb03b44000000000de379caf |
| shape_hash | b2eb9cf0c476f506 |
| count_is_floor | no |
| discriminator | b2eb9cf0c476f506 |
| window | 2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC) there were 10; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/actor |
10 bad of 105 calls · 100% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
The story — 4 execsvc changes landed in the hour before this opened — adjacency, not proof. Everything below is execsvc only.
- 2026-11-16 07:18:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:00:00 UTCerrors execsvc 5xx peaked at 10 in one five-minute bucket (median 0 in this window)
- 2026-11-16 08:00:00 UTClatency execsvc p95 peaked at 1.99 s
- 2026-11-16 08:05:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCthis finding execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:21:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 08:15:41 UTCopened x-ray · severity=critical title=execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor ) finding={'id': 17379, 'fingerprint': 'bb4d70526bc27ac1bd1f', 'kind': 'error.novel', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )', 'facts': {'service': 'execsvc', 'caller': 'observability/middleware.go:281', 'shape': 'Request completed — ? · {?:" failed to create Actor for Study ? Actor {? [{{ 5c4ac420-2d4d-4efe-b649-985c', 'count': 5, 'error_class': 'server_error', 'first_seen': 1794816699, 'sample': 'Request completed — 500 · {"detail":" failed to create Actor for Study 6305ace2-5a30-413d-a77c-0dc79a9984da Actor {49d30832-2209-40dc-84d0-be390a95d87c [{{ 5c4ac420-2d4d-4efe-b649-985c', 'trace': 'e2e7e57fcb03b44000000000de379caf', 'shape_hash': 'b2eb9cf0c476f506', 'window': [1794813341, 1794816941], 'count_is_floor': False, 'discriminator': 'b2eb9cf0c476f506'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=automation&from=1794813341&to=1794816941&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=automation&from=1794813341&to=1794816941&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/e2e7e57fcb03b44000000000de379caf?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 46.0, 'status': 'open', 'first_seen': 1794816941, 'last_seen': 1794816941, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'b2eb9cf0c476f506', 'level': 'medium', 'told_level': None, 'episode_at': 1794816941, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 46, 'stale_s': 46}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |