execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)
warning
Error never seen before
automation
open
impact 27
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 2d ago · last observed 2d ago · observed 3× · #17374 · 8424f7d37caa76b475be
What happened template
execsvc logged “failed to create Actor for Study — user ? is not a member of organization e2EAg1339Cro” 10 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:11 UTC, raised at handlers/plan_definition.go:744). Example trace: e2e7e57fcb03b44000000009a54cd31f.
An error shape appearing for the first time in 90 days, five or more times in an hour.
The facts
| service | execsvc |
| caller | handlers/plan_definition.go:744 |
| shape | failed to create Actor for Study — user ? is not a member of organization e2EAg1339Cro |
| count | 10 |
| error_class | — |
| first_seen | 2026-11-16 08:11:39 UTC (2d ago) |
| sample | failed to create Actor for Study — user 49d30832-2209-40dc-84d0-be390a95d87c is not a member of organization e2EAg1339Cro |
| trace | e2e7e57fcb03b44000000009a54cd31f |
| shape_hash | 92d44488bd1aff6b |
| count_is_floor | no |
| discriminator | 92d44488bd1aff6b |
| window | 2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC) there were 10; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/actor |
10 bad of 105 calls · 100% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
The story — 4 execsvc changes landed in the hour before this opened — adjacency, not proof. Everything below is execsvc only.
- 2026-11-16 07:18:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:00:00 UTCerrors execsvc 5xx peaked at 10 in one five-minute bucket (median 0 in this window)
- 2026-11-16 08:00:00 UTClatency execsvc p95 peaked at 1.99 s
- 2026-11-16 08:05:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCthis finding execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)
- 2026-11-16 08:21:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 08:15:41 UTCopened x-ray · severity=warning title=execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat) finding={'id': 17374, 'fingerprint': '8424f7d37caa76b475be', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)', 'facts': {'service': 'execsvc', 'caller': 'handlers/plan_definition.go:744', 'shape': 'failed to create Actor for Study — user ? is not a member of organization e2EAg1339Cro', 'count': 10, 'error_class': None, 'first_seen': 1794816699, 'sample': 'failed to create Actor for Study — user 49d30832-2209-40dc-84d0-be390a95d87c is not a member of organization e2EAg1339Cro', 'trace': 'e2e7e57fcb03b44000000009a54cd31f', 'shape_hash': '92d44488bd1aff6b', 'window': [1794813341, 1794816941], 'count_is_floor': False, 'discriminator': '92d44488bd1aff6b'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=automation&from=1794813341&to=1794816941&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=automation&from=1794813341&to=1794816941&service=execsvc&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/e2e7e57fcb03b44000000009a54cd31f?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 26.5, 'status': 'open', 'first_seen': 1794816941, 'last_seen': 1794816941, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '92d44488bd1aff6b', 'level': 'medium', 'told_level': None, 'episode_at': 1794816941, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 46, 'stale_s': 46}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |