execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min
critical
Endpoint failing
automation
resolved
impact 51
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 69d ago · last observed 3d ago · observed 8× · #1511 · 60916432cdcbfa7ef958
What happened template
execsvc's POST /api/trialexec/v2/study/:id/actor on automation failed 10 of 17 calls (59 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.
One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.
The facts
| endpoint | POST /api/trialexec/v2/study/:id/actor |
| service | execsvc |
| calls | 17 |
| failed | 10 |
| c4xx | 0 |
| fail_pct | 58.82 |
| window_min | 15 |
| discriminator | POST /api/trialexec/v2/study/:id/actor |
| window | 2026-11-16 08:00:00 UTC → 2026-11-16 08:15:00 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 08:00:00 UTC → 2026-11-16 08:15:00 UTC) there were 10; every line links to the grouped errors behind it.
POST /api/trialexec/v2/study/:id/actor |
10 bad of 17 calls · 100% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
The story — No execsvc deploy or hand change is recorded in the hour before this opened. Everything below is execsvc only.
- 2026-09-11 21:11:41 UTCthis finding execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min
- 2026-10-31 04:53:55 UTCfinding · warning execsvc GET /api/trialexec/study/:id/actor: 4xx — 21 of 60 calls (35 %; other 4xx×21) in the last 15 min — 39× its usual 0.5
- 2026-11-10 14:33:32 UTCfinding · info execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)
- 2026-11-14 14:16:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:17:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:28:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:29:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:35:42 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:36:14 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:41:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 14:41:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:04:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:35 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:08:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:09:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:18:25 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:18:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:46:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:47:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:52:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:53:28 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 00:00:00 UTCerrors execsvc 5xx peaked at 32 in one five-minute bucket (median 0 in this window)
- 2026-11-15 07:16:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:28:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:43:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:59:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:45:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:56:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 06:49:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:02:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:13:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:13:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:18:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:40 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:03 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:05:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study — user ? is not a member of organizat)
- 2026-11-16 08:15:41 UTCfinding · warning execsvc: a new error shape, 10× in the last hour, never seen in 90 days (failed to create Actor for Study error: user ? is not a member of orga)
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:15:41 UTCfinding · critical execsvc: a new error shape, 5× in the last hour, never seen in 90 days (Request completed — ? · {?:" failed to create Actor for Study ? Actor )
- 2026-11-16 08:21:39 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/execsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 08:28:23 UTCreopened x-ray · severity=critical was=critical title=execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min level=high was_level=high finding={'id': 1511, 'fingerprint': '60916432cdcbfa7ef958', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/actor', 'service': 'execsvc', 'calls': 17, 'failed': 10, 'c4xx': 0, 'fail_pct': 58.8235294117647, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/actor'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=automation&from=1794816000&to=1794817703'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794816000&to=1794817703'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817380, 'age_seconds': 330, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 46.1, 'status': 'open', 'first_seen': 1789161101, 'last_seen': 1794817703, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/actor on automation failed 8 of 12 calls (67 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/actor', 'level': 'high', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 5656609, 'stale_s': 7}
- 2026-11-16 08:44:24 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1511, 'fingerprint': '60916432cdcbfa7ef958', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc POST /api/trialexec/v2/study/:id/actor: 10 of 17 calls failed (59 %) in the last 15 min', 'facts': {'endpoint': 'POST /api/trialexec/v2/study/:id/actor', 'service': 'execsvc', 'calls': 17, 'failed': 10, 'c4xx': 0, 'fail_pct': 58.8235294117647, 'window': [1794816000, 1794816900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'POST /api/trialexec/v2/study/:id/actor'}, 'evidence': [{'label': 'execsvc service page', 'href': '/service/execsvc?cluster=automation&from=1794816000&to=1794817703'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794816000&to=1794817703'}], 'provenance': {'source': 'rollup', 'collected_at': 1794817380, 'age_seconds': 330, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 51.1, 'status': 'resolved', 'first_seen': 1789161101, 'last_seen': 1794817703, 'resolved_at': 1794818664, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': "execsvc's POST /api/trialexec/v2/study/:id/actor on automation failed 10 of 17 calls (59 %) in the last 15 min. One endpoint, not the service: the rest of execsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'POST /api/trialexec/v2/study/:id/actor', 'level': 'high', 'told_level': None, 'episode_at': 1794817703, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 5657569, 'stale_s': 967}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794817380 |
| age_seconds | 330 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_failing |