execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)
warning
Error never seen before
automation
open
impact 27
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 3d ago · last observed 3d ago · observed 3× · #17375 · f175c57e018b85b32374
What happened template
execsvc logged “failed to create Actor for Study error: user ? is not a member of organization e2EAg1339Cro” 10 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 08:11 UTC, raised at utils/utils.go:133). Example trace: none carried.
An error shape appearing for the first time in 90 days, five or more times in an hour.
The facts
| service | execsvc |
| caller | utils/utils.go:133 |
| shape | failed to create Actor for Study error: user ? is not a member of organization e2EAg1339Cro |
| count | 10 |
| error_class | — |
| first_seen | 2026-11-16 08:11:39 UTC (3d ago) |
| sample | failed to create Actor for Study error: user 49d30832-2209-40dc-84d0-be390a95d87c is not a member of organization e2EAg1339Cro |
| trace | — |
| shape_hash | 2651e6aa35fff145 |
| count_is_floor | no |
| discriminator | 2651e6aa35fff145 |
| window | 2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 08:06:49 UTC → 2026-11-16 09:06:49 UTC) there were 10; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/actor |
10 bad of 105 calls · 100% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
The story — 4 execsvc changes landed in the hour before this opened — adjacency, not proof. Everything below is execsvc only.
- 2026-11-16 07:18:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:00:00 UTCerrors execsvc 5xx peaked at 10 in one five-minute bucket (median 0 in this window)
- 2026-11-16 08:00:00 UTClatency execsvc p95 peaked at 1.99 s
- 2026-11-16 08:05:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCthis finding execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)
- 2026-11-16 08:21:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 08:15:41 UTCopened x-ray · severity=warning title=execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga) finding={'id': 17375, 'fingerprint': 'f175c57e018b85b32374', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)', 'facts': {'service': 'execsvc', 'caller': 'utils/utils.go:133', 'shape': 'failed to create Actor for Study error: user ? is not a member of organization e2EAg1339Cro', 'count': 10, 'error_class': None, 'first_seen': 1794816699, 'sample': 'failed to create Actor for Study error: user 49d30832-2209-40dc-84d0-be390a95d87c is not a member of organization e2EAg1339Cro', 'trace': None, 'shape_hash': '2651e6aa35fff145', 'window': [1794813341, 1794816941], 'count_is_floor': False, 'discriminator': '2651e6aa35fff145'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=automation&from=1794813341&to=1794816941&service=execsvc'}, {'label': 'Logs', 'href': '/logs?cluster=automation&from=1794813341&to=1794816941&service=execsvc&severity=ERROR'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 26.5, 'status': 'open', 'first_seen': 1794816941, 'last_seen': 1794816941, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '2651e6aa35fff145', 'level': 'medium', 'told_level': None, 'episode_at': 1794816941, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 46, 'stale_s': 46}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |