studiosvc: p99 latency (ms) 24.50 s — 12.6× the fleet's usual 1.94 s
info
Service outlier
test26
resolved
impact 12
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 84d ago · last observed 3d ago · observed 1051× · #126 · b4559bf8c1f02ebb9945
What happened template
studiosvc's p99 latency (ms) is 24.50 s against the 7 other services' median of 1.94 s (8.7 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.
A service unlike the rest of the fleet on one measure (robust z against its peers).
The facts
| measures | {'measure': 'p99 latency (ms)', 'value': 24500.000000000455, 'value_fmt': '24.50 s', 'median': 1938.1249999999995, 'median_fmt': '1.94 s', 'mad': 2604.0801977272718, 'z': 8.664047681669512, 'direction': 'above', 'peers': 7} |
| measure | p99 latency (ms) |
| value | 24,500 |
| value_fmt | 24.50 s |
| median | 1,938 |
| median_fmt | 1.94 s |
| z | 8.66 |
| direction | above |
| peers | 7 |
| rule | {"min_score": 3.5, "min_relative_pct": 35, "warn_on": []} |
| discriminator | |
| window | 2026-11-15 00:59:37 UTC → 2026-11-15 06:59:37 UTC |
What is failing in studiosvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-15 00:59:37 UTC → 2026-11-15 06:59:37 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 6× | http error response returned to client since 2026-10-07 00:39:15 UTC |
| 1× | Request completed — 500 · graph action pre failed for FormV2:protocol edit applicable to completed visits, cannot modify… since 2026-11-16 13:33:41 UTC |
| 1× | failed to handle graph node action — graph action pre failed for FormV2:protocol edit applicable to completed visits, ca… since 2026-11-16 13:33:41 UTC |
| 1× | failed to handle graph node action — graph action pre failed for FormV2:protocol edit applicable to completed visits, ca… since 2026-11-16 13:33:41 UTC |
History
- 2026-11-16 17:34:53 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 126, 'fingerprint': 'b4559bf8c1f02ebb9945', 'kind': 'outlier.service', 'severity': 'info', 'cluster': 'test26', 'subject': 'studiosvc', 'title': "studiosvc: p99 latency (ms) 24.50 s — 12.6× the fleet's usual 1.94 s", 'facts': {'measures': [{'measure': 'p99 latency (ms)', 'value': 24500.000000000455, 'value_fmt': '24.50 s', 'median': 1938.1249999999995, 'median_fmt': '1.94 s', 'mad': 2604.0801977272718, 'z': 8.664047681669512, 'direction': 'above', 'peers': 7}], 'measure': 'p99 latency (ms)', 'value': 24500.000000000455, 'value_fmt': '24.50 s', 'median': 1938.1249999999995, 'median_fmt': '1.94 s', 'z': 8.664047681669512, 'direction': 'above', 'peers': 7, 'window': [1794704377, 1794725977], 'recent_deploys': [], 'rule': {'min_score': 3.5, 'min_relative_pct': 35, 'warn_on': []}, 'discriminator': ''}, 'evidence': [{'label': 'Outliers', 'href': '/outliers?cluster=test26&from=1794704377&to=1794725977'}, {'label': 'studiosvc service page', 'href': '/service/studiosvc?cluster=test26&from=1794704377&to=1794725977'}, {'label': 'Tune this rule', 'href': '/alerts/system/outliers'}], 'provenance': {'source': 'rollup', 'collected_at': 1794725977, 'age_seconds': 1, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.8, 'status': 'resolved', 'first_seen': 1787737993, 'last_seen': 1794725977, 'resolved_at': 1794850493, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1051, 'narrative': "studiosvc's p99 latency (ms) is 24.50 s against the 7 other services' median of 1.94 s (8.7 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1793363960, 'chat_thread': None, 'kind_label': 'Service outlier', 'group': 'Anomalies', 'age_s': 7112501, 'stale_s': 124517}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794725977 |
| age_seconds | 1 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | outliers |