Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 19:36:41.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min

critical Endpoint failing production resolved impact 82 Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) opened 36d ago · last observed 1d ago · observed 57× · #4570 · c7eff385694e63dc81ee

What happened template

onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 15 of 15 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; GET /api/myprofile/visibility/study/:id/blindedScopes failed 8 of 8 calls (100 %, all 500, 0 4xx) in 15 min, first failure 17:19 UTC.
All 8 calls come from org ophthalmicConsultantsOfBoston and fail fast (9–11 ms) at health/actor.go:49: "could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId…".
This error shape has existed since 2026-10-13, and this finding has opened and closed 5 times since Nov 6, mostly around 17:00 UTC, so it is not new.
The only deploy in the prior 24 h was config-only (ash-nazg 6db9855a7d34, 05:10 UTC, 719 min before onset), with no code change and nothing linking it.
Code search for the endpoint and the error text found nothing in tildabio/main/onboardingsvc.
Next step: open a failing trace, see which :id the client sends, and find why the Study lookup at health/actor.go:49 gets an empty tildaId. Add a 4xx guard for an empty or unresolved id.

low confidence

claude-opus-5-5 · 2d ago · $0.06 — every link above is evidence it read; nothing else was claimed.

The facts

endpointGET /api/myprofile/visibility/study/:id/blindedScopes
serviceonboardingsvc
calls15
failed15
c4xx0
fail_pct100.00
window_min15
discriminatorGET /api/myprofile/visibility/study/:id/blindedScopes
window2026-11-17 00:10:00 UTC → 2026-11-17 00:25:00 UTC

What is failing in onboardingsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from onboardingsvc. In this finding's window (2026-11-17 00:10:00 UTC → 2026-11-17 00:25:00 UTC) there were 15; every line links to the grouped errors behind it.

GET /api/myprofile/visibility/study/:id/blindedScopes 15 bad of 15 calls · 100% of the service's 5xx

The error shapes behind them

145× request failed — 500 since 2026-10-13 11:01:59 UTC
145× Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-10-13 11:32:37 UTC
145× could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-10-13 11:32:37 UTC
21× Error getting folder manager — FolderManager(StudyInstance) not found: <nil> since 2026-11-09 15:36:19 UTC

All onboardingsvc errors, grouped → · service page · its logs

The story — No onboardingsvc deploy or hand change is recorded in the hour before this opened. Everything below is onboardingsvc only.

History

  • 2026-11-16 17:34:53 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794849000, 1794849900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794849000&to=1794850493'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794849000&to=1794850493'}], 'provenance': {'source': 'rollup', 'collected_at': 1794850446, 'age_seconds': 56, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 78.1, 'status': 'open', 'first_seen': 1791906847, 'last_seen': 1794850493, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794850493, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2943655, 'stale_s': 9}
  • 2026-11-16 19:45:22 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794849000, 1794849900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794849000&to=1794850493'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794849000&to=1794850493'}], 'provenance': {'source': 'rollup', 'collected_at': 1794850446, 'age_seconds': 56, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 83.1, 'status': 'resolved', 'first_seen': 1791906847, 'last_seen': 1794850493, 'resolved_at': 1794858322, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794850493, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2951483, 'stale_s': 7837}
  • 2026-11-17 00:37:22 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794874200, 1794875100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794874200&to=1794875842'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794874200&to=1794875842'}], 'provenance': {'source': 'rollup', 'collected_at': 1794875679, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.8, 'status': 'open', 'first_seen': 1791906847, 'last_seen': 1794875842, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794875842, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2969004, 'stale_s': 9}
  • 2026-11-17 02:44:54 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794874200, 1794875100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794874200&to=1794875842'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794874200&to=1794875842'}], 'provenance': {'source': 'rollup', 'collected_at': 1794875679, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 81.8, 'status': 'resolved', 'first_seen': 1791906847, 'last_seen': 1794875842, 'resolved_at': 1794883494, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 15 of 15 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794875842, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2976654, 'stale_s': 7659}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794875679
age_seconds173
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorendpoint_failing