execsvc: 5xx rate (%) 0.31% — 6.6× the fleet's usual 0.05%
info
Service outlier
automation
open
impact 13
Owner: Madhava Mukkamala (backup: Shashank Agrawal)
opened 85d ago · last observed 2d ago · observed 1668× · #150 · 076a14f155e32f214ab6
What happened template
execsvc's 5xx rate (%) is 0.31% against the 14 other services' median of 0.05% (3.8 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.
A service unlike the rest of the fleet on one measure (robust z against its peers).
The facts
| measures | {'measure': '5xx rate (%)', 'value': 0.3116073747078681, 'value_fmt': '0.31%', 'median': 0.047319865703853846, 'median_fmt': '0.05%', 'mad': 0.0701564328925337, 'z': 3.7671172565009456, 'direction': 'above', 'peers': 14} |
| measure | 5xx rate (%) |
| value | 0.31 |
| value_fmt | 0.31% |
| median | 0.05 |
| median_fmt | 0.05% |
| z | 3.77 |
| direction | above |
| peers | 14 |
| rule | {"min_score": 3.5, "min_relative_pct": 35, "warn_on": []} |
| discriminator | |
| window | 2026-11-16 16:06:42 UTC → 2026-11-16 22:06:42 UTC |
What is failing in execsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 16:06:42 UTC → 2026-11-16 22:06:42 UTC) there were 12; every line links to the grouped errors behind it.
POST /api/trialexec/careplan/:id/task |
6 bad of 8 calls · 50% of the service's 5xx |
POST /api/trialexec/careplan/:id/visit/:id/task |
3 bad of 3 calls · 25% of the service's 5xx |
POST /api/trialexec/v2/study/:id/actor |
2 bad of 100 calls · 17% of the service's 5xx |
POST /api/trialexec/careplan/:id/visit/:id/activity/task |
1 bad of 3 calls · 8% of the service's 5xx |
The error shapes behind them
| 388× | failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC |
| 373× | failed to get Study c262fdf4-38c8-4f68-9b57-83f1586e8ee2. Error: object Study {tildaid:$a_tildaid} not found since 2026-10-24 00:46:35 UTC |
| 78× | failed to get visit schedule for careplan — careplan is not in preview mode: 0 since 2026-10-31 04:21:27 UTC |
| 78× | Request completed — 500 · Could not get studyConfig for Careplan f4495ea5-a88a-42de-8410-e9a863a8a256: careplan is not i… since 2026-10-31 04:21:27 UTC |
| 76× | Could not read from db — Neo4jError: Neo.ClientError.Database.DatabaseNotFound (Unable to get a routing table for databa… since 2026-10-29 12:43:26 UTC |
History
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794866663 |
| age_seconds | 140 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | outliers |