docsvc: p99 latency (ms) 3.88 s — 4.7× the fleet's usual 825 ms
info
Service outlier
production
open
impact 25
Owner: Prasad Limaye (backup: Harish Kumar)
opened 26d ago · last observed 1d ago · observed 561× · #5501 · 0b08e5e374d567506b13
What happened template
docsvc's p99 latency (ms) is 3.88 s against the 5 other services' median of 825 ms (4.6 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.
A service unlike the rest of the fleet on one measure (robust z against its peers).
The facts
| measures | {'measure': 'p99 latency (ms)', 'value': 3878.947368421049, 'value_fmt': '3.88 s', 'median': 824.9999999999993, 'median_fmt': '825 ms', 'mad': 665.1932000000007, 'z': 4.591068231637134, 'direction': 'above', 'peers': 5} |
| measure | p99 latency (ms) |
| value | 3,879 |
| value_fmt | 3.88 s |
| median | 825.00 |
| median_fmt | 825 ms |
| z | 4.59 |
| direction | above |
| peers | 5 |
| rule | {"min_score": 3.5, "min_relative_pct": 35, "warn_on": []} |
| discriminator | |
| window | 2026-11-16 22:45:57 UTC → 2026-11-17 04:45:57 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 22:45:57 UTC → 2026-11-17 04:45:57 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 30× | failed to fetch document, err: VersionId 10b81676-b9d2-4a9d-89ea-320ebf61c3c7 not found in document 693f6fbc-cec0-448c-9… since 2026-11-09 14:40:22 UTC |
| 3× | access denied for this study since 2026-11-11 13:51:29 UTC |
| 3× | failed to get document: document fc8c6283-c350-44c9-87b0-a9f91e79761c, user [email protected], document not found: no… since 2026-11-09 17:11:09 UTC |
History
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794890419 |
| age_seconds | 339 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | outliers |