Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:16:54.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)

info Latency anomaly test27 resolved impact 12 Owner: Suresh Boddapati (backup: Madhava Mukkamala) opened 12d ago · last observed 1d ago · observed 59× · #16316 · 1ed1fd72aeacc504a8b8

What happened template

studiosvc's p95 is 7.4 s on test27, above a normal of 199 ms ± 157 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 45.6, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

p95 far above its hour-of-week normal — and at least 1 s; under a second is noise.

The facts

measureapi_p95_ms
unitms
value7,353
expected_median199.26
expected_mad156.81
z45.62
rel35.90
buckets_hit4
n21
weeks3
slotTue 07:00 UTC
bucket2026-11-17 07:40:00 UTC (1d ago)
floor_ms1,000
discriminatorapi_p95_ms
window2026-11-17 07:25:00 UTC → 2026-11-17 07:45:00 UTC

What is failing in studiosvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-17 07:25:00 UTC → 2026-11-17 07:45:00 UTC) there were 1; every line links to the grouped errors behind it.

POST /api/studio/forms/:id/activationRuleChains 1 bad of 47 calls · 100% of the service's 5xx

The error shapes behind them

17× http error response returned to client since 2026-10-06 01:20:44 UTC
6× Request completed — 500 · could not create activation rule chain:activation rule \ since 2026-10-02 11:58:33 UTC

All studiosvc errors, grouped → · service page · its logs

History

  • 2026-11-16 14:48:32 UTCreopened x-ray · severity=info was=info title=studiosvc p95 1.7 min vs a normal 375 ms ± 141 ms for Mon 14:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 1.7 min vs a normal 375 ms ± 141 ms for Mon 14:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 101250.0, 'expected_median': 375.0, 'expected_mad': 140.847, 'z': 716.202688023174, 'rel': 269.0, 'buckets_hit': 3, 'n': 23, 'weeks': 3, 'slot': 'Mon 14:00 UTC', 'bucket': 1794839700, 'window': [1794838800, 1794840000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794817200&to=1794840000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794817200&to=1794840000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794840510, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.1, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794840512, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 51, 'narrative': "studiosvc's p95 is 120.0 s on test27, above a normal of 432 ms ± 278 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 430.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794840512, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 922891, 'stale_s': 18}
  • 2026-11-16 15:08:36 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 1.7 min vs a normal 375 ms ± 141 ms for Mon 14:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 101250.0, 'expected_median': 375.0, 'expected_mad': 140.847, 'z': 716.202688023174, 'rel': 269.0, 'buckets_hit': 3, 'n': 23, 'weeks': 3, 'slot': 'Mon 14:00 UTC', 'bucket': 1794839700, 'window': [1794838800, 1794840000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794817200&to=1794840000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794817200&to=1794840000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794840510, 'age_seconds': 20, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.1, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794840512, 'resolved_at': 1794841716, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 51, 'narrative': "studiosvc's p95 is 101.2 s on test27, above a normal of 375 ms ± 141 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 716.2, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794840512, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 924094, 'stale_s': 1221}
  • 2026-11-16 15:18:05 UTCreopened x-ray · severity=info was=info title=studiosvc p95 1.10 s vs a normal 461 ms ± 113 ms for Mon 15:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 1.10 s vs a normal 461 ms ± 113 ms for Mon 15:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1099.9999999999943, 'expected_median': 461.0435, 'expected_mad': 112.94572, 'z': 5.657199759317966, 'rel': 1.3858920036829374, 'buckets_hit': 3, 'n': 30, 'weeks': 4, 'slot': 'Mon 15:00 UTC', 'bucket': 1794841500, 'window': [1794840600, 1794841800], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794819000&to=1794841800'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794819000&to=1794841800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794842283, 'age_seconds': 18, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.1, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794842285, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 52, 'narrative': "studiosvc's p95 is 101.2 s on test27, above a normal of 375 ms ± 141 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 716.2, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794842285, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 924662, 'stale_s': 16}
  • 2026-11-16 15:51:04 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 26.50 s vs a normal 461 ms ± 113 ms for Mon 15:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 26499.999999999985, 'expected_median': 461.0435, 'expected_mad': 112.94572, 'z': 230.54398608464302, 'rel': 56.47830736145285, 'buckets_hit': 4, 'n': 30, 'weeks': 4, 'slot': 'Mon 15:00 UTC', 'bucket': 1794842100, 'window': [1794841200, 1794842400], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794819600&to=1794842400'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794819600&to=1794842400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794842900, 'age_seconds': 19, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.1, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794842902, 'resolved_at': 1794844264, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 53, 'narrative': "studiosvc's p95 is 26.5 s on test27, above a normal of 461 ms ± 113 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 230.5, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794842285, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 926638, 'stale_s': 1375}
  • 2026-11-16 17:43:06 UTCreopened x-ray · severity=info was=info title=studiosvc p95 29.00 s vs a normal 452 ms ± 52 ms for Mon 17:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 29.00 s vs a normal 452 ms ± 52 ms for Mon 17:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 28999.999999999913, 'expected_median': 451.51785, 'expected_mad': 51.917477, 'z': 549.8819241543636, 'rel': 63.22780406134533, 'buckets_hit': 3, 'n': 20, 'weeks': 3, 'slot': 'Mon 17:00 UTC', 'bucket': 1794850200, 'window': [1794849300, 1794850500], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794827700&to=1794850500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794827700&to=1794850500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794850983, 'age_seconds': 19, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.8, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794850986, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 54, 'narrative': "studiosvc's p95 is 26.5 s on test27, above a normal of 461 ms ± 113 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 230.5, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794850986, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 933363, 'stale_s': 16}
  • 2026-11-16 18:07:03 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 29.00 s vs a normal 452 ms ± 52 ms for Mon 17:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 28999.999999999913, 'expected_median': 451.51785, 'expected_mad': 51.917477, 'z': 549.8819241543636, 'rel': 63.22780406134533, 'buckets_hit': 3, 'n': 20, 'weeks': 3, 'slot': 'Mon 17:00 UTC', 'bucket': 1794850200, 'window': [1794849600, 1794850800], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794828000&to=1794850800'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794828000&to=1794850800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794851431, 'age_seconds': 62, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.8, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794851475, 'resolved_at': 1794852423, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 55, 'narrative': "studiosvc's p95 is 29.0 s on test27, above a normal of 452 ms ± 52 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 549.9, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794850986, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 934800, 'stale_s': 964}
  • 2026-11-16 20:37:48 UTCreopened x-ray · severity=info was=info title=studiosvc p95 2.0 min vs a normal 462 ms ± 101 ms for Mon 20:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 2.0 min vs a normal 462 ms ± 101 ms for Mon 20:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 120000.0, 'expected_median': 461.82895, 'expected_mad': 100.53881, 'z': 1188.9753921893446, 'rel': 258.83646109668956, 'buckets_hit': 3, 'n': 26, 'weeks': 5, 'slot': 'Mon 20:00 UTC', 'bucket': 1794860400, 'window': [1794859800, 1794861000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794838200&to=1794861000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794838200&to=1794861000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794861466, 'age_seconds': 17, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.6, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794861468, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "studiosvc's p95 is 29.0 s on test27, above a normal of 452 ms ± 52 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 549.9, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794861468, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 943844, 'stale_s': 15}
  • 2026-11-16 20:53:56 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 2.0 min vs a normal 462 ms ± 101 ms for Mon 20:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 120000.0, 'expected_median': 461.82895, 'expected_mad': 100.53881, 'z': 1188.9753921893446, 'rel': 258.83646109668956, 'buckets_hit': 3, 'n': 26, 'weeks': 5, 'slot': 'Mon 20:00 UTC', 'bucket': 1794860400, 'window': [1794859800, 1794861000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794838200&to=1794861000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794838200&to=1794861000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794861466, 'age_seconds': 17, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.6, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794861468, 'resolved_at': 1794862436, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "studiosvc's p95 is 120.0 s on test27, above a normal of 462 ms ± 101 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 1189.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794861468, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 944811, 'stale_s': 982}
  • 2026-11-16 23:44:07 UTCreopened x-ray · severity=info was=info title=studiosvc p95 55.50 s vs a normal 455 ms ± 63 ms for Mon 23:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 55.50 s vs a normal 455 ms ± 63 ms for Mon 23:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 55500.000000000044, 'expected_median': 455.0, 'expected_mad': 63.0105, 'z': 873.5845613032756, 'rel': 120.97802197802207, 'buckets_hit': 3, 'n': 21, 'weeks': 4, 'slot': 'Mon 23:00 UTC', 'bucket': 1794871800, 'window': [1794870900, 1794872100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794849300&to=1794872100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794849300&to=1794872100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794872645, 'age_seconds': 16, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.8, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794872647, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "studiosvc's p95 is 120.0 s on test27, above a normal of 462 ms ± 101 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 1189.0, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 955022, 'stale_s': 14}
  • 2026-11-17 00:04:07 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 55.50 s vs a normal 455 ms ± 63 ms for Mon 23:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 55500.000000000044, 'expected_median': 455.0, 'expected_mad': 63.0105, 'z': 873.5845613032756, 'rel': 120.97802197802207, 'buckets_hit': 3, 'n': 21, 'weeks': 4, 'slot': 'Mon 23:00 UTC', 'bucket': 1794871800, 'window': [1794870900, 1794872100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794849300&to=1794872100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794849300&to=1794872100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794872645, 'age_seconds': 16, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.8, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794872647, 'resolved_at': 1794873847, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "studiosvc's p95 is 55.5 s on test27, above a normal of 455 ms ± 63 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 873.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794872647, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 956220, 'stale_s': 1212}
  • 2026-11-17 01:43:17 UTCreopened x-ray · severity=info was=info title=studiosvc p95 49.50 s vs a normal 440 ms ± 149 ms for Tue 01:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 49.50 s vs a normal 440 ms ± 149 ms for Tue 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 49499.999999999956, 'expected_median': 440.0, 'expected_mad': 149.319, 'z': 328.55832144603136, 'rel': 111.4999999999999, 'buckets_hit': 3, 'n': 21, 'weeks': 4, 'slot': 'Tue 01:00 UTC', 'bucket': 1794878700, 'window': [1794877800, 1794879000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794856200&to=1794879000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794856200&to=1794879000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794879632, 'age_seconds': 179, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.2, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794879797, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 58, 'narrative': "studiosvc's p95 is 55.5 s on test27, above a normal of 455 ms ± 63 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 873.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794879797, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 962172, 'stale_s': 14}
  • 2026-11-17 02:06:20 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 49.50 s vs a normal 440 ms ± 149 ms for Tue 01:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 49499.999999999956, 'expected_median': 440.0, 'expected_mad': 149.319, 'z': 328.55832144603136, 'rel': 111.4999999999999, 'buckets_hit': 3, 'n': 21, 'weeks': 4, 'slot': 'Tue 01:00 UTC', 'bucket': 1794878700, 'window': [1794877800, 1794879000], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794856200&to=1794879000'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794856200&to=1794879000'}], 'provenance': {'source': 'rollup', 'collected_at': 1794879632, 'age_seconds': 179, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.2, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794879797, 'resolved_at': 1794881180, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 58, 'narrative': "studiosvc's p95 is 49.5 s on test27, above a normal of 440 ms ± 149 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 328.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794879797, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 963554, 'stale_s': 1396}
  • 2026-11-17 08:04:09 UTCreopened x-ray · severity=info was=info title=studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks) level=low was_level=low finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 7352.941176470588, 'expected_median': 199.26471, 'expected_mad': 156.80675, 'z': 45.62097273536113, 'rel': 35.9003682411732, 'buckets_hit': 4, 'n': 21, 'weeks': 3, 'slot': 'Tue 07:00 UTC', 'bucket': 1794901200, 'window': [1794900300, 1794901500], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794878700&to=1794901500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794878700&to=1794901500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794902361, 'age_seconds': 306, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 7.5, 'status': 'open', 'first_seen': 1793917639, 'last_seen': 1794902649, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': "studiosvc's p95 is 49.5 s on test27, above a normal of 440 ms ± 149 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 328.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 985028, 'stale_s': 18}
  • 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16316, 'fingerprint': '1ed1fd72aeacc504a8b8', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 7352.941176470588, 'expected_median': 199.26471, 'expected_mad': 156.80675, 'z': 45.62097273536113, 'rel': 35.9003682411732, 'buckets_hit': 4, 'n': 21, 'weeks': 3, 'slot': 'Tue 07:00 UTC', 'bucket': 1794901200, 'window': [1794900300, 1794901500], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/studiosvc?cluster=test27&from=1794878700&to=1794901500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=test27&from=1794878700&to=1794901500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794902361, 'age_seconds': 306, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 12.5, 'status': 'resolved', 'first_seen': 1793917639, 'last_seen': 1794902649, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 59, 'narrative': "studiosvc's p95 is 7.4 s on test27, above a normal of 199 ms ± 157 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 45.6, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794902649, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 986382, 'stale_s': 1372}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794902361
age_seconds306
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorseasonal