Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 48 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:43:35.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

master (whole cluster) server-error rate 57.64% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)

critical Error-rate anomaly master resolved impact 81 opened 55d ago · last observed 1d ago · observed 9× · #2318 · d966574561e79de03668

What happened template

the whole master cluster's error rate is 57.6% on master, above a normal of 0.0% ± 1.0% for this hour of the week (3 of the last 4 five-minute buckets, z = 57.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

Error percentage far above its hour-of-week normal.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; execsvc POST /api/trialexec/v2/study/:id/activity/status is failing 787 of 787 calls (cluster 5xx 57.64% vs normal 0.00%) since 2026-11-17 04:05 UTC.
All 700 error lines in the window say "could not match participant question to activity graph" (handlers/activity_details_v2.go:457, execution/activity_details_v2.go:2007), for questionId="5571182f-6c21-491e-b8c6-51fce3166e1e" with an empty refId, variableName and title.
The only execsvc deploy was an autosync config change d8d21151cd28 at 01:33 UTC, 151 min before onset, with no new image; nothing ties it to this error.
This message was already seen on 2026-10-16 and 2026-11-04, so bad study data is more likely than new code.
Next step: check which study and activity graph reference question 5571182f-6c21-491e-b8c6-51fce3166e1e, and why the graph has no matching refId or variableName.

low confidence

claude-opus-5-5 · 1d ago · $0.05 — every link above is evidence it read; nothing else was claimed.

The facts

measureerr5_pct
unit%
value57.64
expected_median0.00
expected_mad1.00
z57.64
relinf
buckets_hit3
n40
weeks4
slotTue 04:00 UTC
bucket2026-11-17 04:15:00 UTC (1d ago)
failing_endpoints{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}
discriminatorerr5_pct
window2026-11-17 04:05:00 UTC → 2026-11-17 04:25:00 UTC

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-17 04:45:57 UTCreopened x-ray · severity=critical was=critical title=master (whole cluster) server-error rate 57.64% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx) level=high was_level=high finding={'id': 2318, 'fingerprint': 'd966574561e79de03668', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': '*', 'title': 'master (whole cluster) server-error rate 57.64% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 57.64192139737992, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 57.64192139737992, 'rel': inf, 'buckets_hit': 3, 'n': 40, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Cluster charts', 'href': '/charts?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 651, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.0, 'status': 'open', 'first_seen': 1790229390, 'last_seen': 1794890757, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "the whole master cluster's error rate is 12.6% on master, above a normal of 0.0% ± 1.0% for this hour of the week (3 of the last 4 five-minute buckets, z = 12.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 4661377, 'stale_s': 10}
  • 2026-11-17 05:17:45 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 2318, 'fingerprint': 'd966574561e79de03668', 'kind': 'anomaly.errors', 'severity': 'critical', 'cluster': 'master', 'subject': '*', 'title': 'master (whole cluster) server-error rate 57.64% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)', 'facts': {'measure': 'err5_pct', 'unit': '%', 'value': 57.64192139737992, 'expected_median': 0.0, 'expected_mad': 1.0, 'z': 57.64192139737992, 'rel': inf, 'buckets_hit': 3, 'n': 40, 'weeks': 4, 'slot': 'Tue 04:00 UTC', 'bucket': 1794888900, 'window': [1794888300, 1794889500], 'recent_deploys': [], 'failing_endpoints': [{'service': 'execsvc', 'endpoint': 'POST /api/trialexec/v2/study/:id/activity/status', 'c5xx': 787, 'calls': 787}], 'discriminator': 'err5_pct'}, 'evidence': [{'label': 'Cluster charts', 'href': '/charts?cluster=master&from=1794866700&to=1794889500'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=master&from=1794866700&to=1794889500'}], 'provenance': {'source': 'rollup', 'collected_at': 1794890117, 'age_seconds': 651, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 81.0, 'status': 'resolved', 'first_seen': 1790229390, 'last_seen': 1794890757, 'resolved_at': 1794892665, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "the whole master cluster's error rate is 57.6% on master, above a normal of 0.0% ± 1.0% for this hour of the week (3 of the last 4 five-minute buckets, z = 57.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.", 'narrative_kind': 'template', 'discriminator': 'err5_pct', 'level': 'high', 'told_level': None, 'episode_at': 1794890757, 'chat_thread': None, 'kind_label': 'Error-rate anomaly', 'group': 'Anomalies', 'age_s': 4663283, 'stale_s': 1916}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794890117
age_seconds651
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorseasonal