Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 21:01:45.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)

info Latency anomaly automation resolved impact 14 Owner: Madhava Mukkamala (backup: Shashank Agrawal) opened 8d ago · last observed 1d ago · observed 9× · #16997 · e9268f5abb3c0bb1c6b7

What happened template

execsvc's p95 is 1.5 s on automation, above a normal of 472 ms ± 178 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 5.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

p95 far above its hour-of-week normal — and at least 1 s; under a second is noise.

The facts

measureapi_p95_ms
unitms
value1,525
expected_median471.83
expected_mad178.06
z5.91
rel2.23
buckets_hit4
n29
weeks5
slotTue 05:00 UTC
bucket2026-11-17 05:20:00 UTC (1d ago)
floor_ms1,000
discriminatorapi_p95_ms
window2026-11-17 05:05:00 UTC → 2026-11-17 05:25:00 UTC

What is failing in execsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-17 05:05:00 UTC → 2026-11-17 05:25:00 UTC) there were 1; every line links to the grouped errors behind it.

POST /api/trialexec/v2/study/:id/activity/v2/complete 1 bad of 9 calls · 100% of the service's 5xx

The error shapes behind them

388× failed to get Study 0478051e-6f85-452a-b527-549d63fff940. Error: object Study {tildaid:$a_tildaid} is marked deleted since 2026-10-24 00:37:37 UTC
20× could not list DutyPlans for Study d9b557d0-7319-45b6-b4fb-57e2ec1298b1: passed Study was not found since 2026-10-08 12:44:23 UTC
4× failed to create Actor for Study error: cannot modify actors for site 78adecaa-0c77-4b00-9595-1a27ec13cd01: it is owned … since 2026-08-26 01:01:46 UTC
4× failed to create Actor for Study — cannot modify actors for site d8c7a8b4-daaf-44c9-b09b-a76b9cac399d: it is owned by a … since 2026-10-30 00:45:57 UTC
4× failed to get Adopted ProtocolVersion for site and plan definition since 2026-11-17 05:57:21 UTC

All execsvc errors, grouped → · service page · its logs

History

  • 2026-11-17 05:26:24 UTCreopened x-ray · severity=info was=info title=execsvc p95 1.94 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16997, 'fingerprint': 'e9268f5abb3c0bb1c6b7', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc p95 1.94 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1940.9090909090903, 'expected_median': 471.82925, 'expected_mad': 178.05803, 'z': 8.250567755405866, 'rel': 3.1135836553352516, 'buckets_hit': 3, 'n': 29, 'weeks': 5, 'slot': 'Tue 05:00 UTC', 'bucket': 1794892200, 'window': [1794891300, 1794892500], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=automation&from=1794869700&to=1794892500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794869700&to=1794892500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794893090, 'age_seconds': 104, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 9.4, 'status': 'open', 'first_seen': 1794321212, 'last_seen': 1794893184, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': "execsvc's p95 is 1.7 s on automation, above a normal of 263 ms ± 348 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 4.0, 3 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794893184, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 571983, 'stale_s': 11}
  • 2026-11-17 06:00:02 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16997, 'fingerprint': 'e9268f5abb3c0bb1c6b7', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'execsvc', 'title': 'execsvc p95 1.52 s vs a normal 472 ms ± 178 ms for Tue 05:00 UTC (4 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1524.9999999999986, 'expected_median': 471.82925, 'expected_mad': 178.05803, 'z': 5.9147613280906155, 'rel': 2.232101443477696, 'buckets_hit': 4, 'n': 29, 'weeks': 5, 'slot': 'Tue 05:00 UTC', 'bucket': 1794892800, 'window': [1794891900, 1794893100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/execsvc?cluster=automation&from=1794870300&to=1794893100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794870300&to=1794893100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794893590, 'age_seconds': 102, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 14.3, 'status': 'resolved', 'first_seen': 1794321212, 'last_seen': 1794893682, 'resolved_at': 1794895202, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "execsvc's p95 is 1.5 s on automation, above a normal of 472 ms ± 178 ms for this hour of the week (4 of the last 4 five-minute buckets, z = 5.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794893184, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 574002, 'stale_s': 1532}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794893590
age_seconds102
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorseasonal