onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
critical
Endpoint failing
production
resolved
impact 82
Owner: Prasad Limaye (backup: Sambasiva Bandarupalli)
opened 36d ago · last observed 1d ago · observed 57× · #4570 · c7eff385694e63dc81ee
What happened template
onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 15 of 15 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.
One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; GET /api/myprofile/visibility/study/:id/blindedScopes failed 8 of 8 calls (100 %, all 500, 0 4xx) in 15 min, first failure 17:19 UTC.
All 8 calls come from org ophthalmicConsultantsOfBoston and fail fast (9–11 ms) at health/actor.go:49: "could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId…".
This error shape has existed since 2026-10-13, and this finding has opened and closed 5 times since Nov 6, mostly around 17:00 UTC, so it is not new.
The only deploy in the prior 24 h was config-only (ash-nazg 6db9855a7d34, 05:10 UTC, 719 min before onset), with no code change and nothing linking it.
Code search for the endpoint and the error text found nothing in tildabio/main/onboardingsvc.
Next step: open a failing trace, see which :id the client sends, and find why the Study lookup at health/actor.go:49 gets an empty tildaId. Add a 4xx guard for an empty or unresolved id.
low confidence
claude-opus-5-5 · 2d ago · $0.06 — every link above is evidence it read; nothing else was claimed.
The facts
| endpoint | GET /api/myprofile/visibility/study/:id/blindedScopes |
| service | onboardingsvc |
| calls | 15 |
| failed | 15 |
| c4xx | 0 |
| fail_pct | 100.00 |
| window_min | 15 |
| discriminator | GET /api/myprofile/visibility/study/:id/blindedScopes |
| window | 2026-11-17 00:10:00 UTC → 2026-11-17 00:25:00 UTC |
What is failing in onboardingsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from onboardingsvc. In this finding's window (2026-11-17 00:10:00 UTC → 2026-11-17 00:25:00 UTC) there were 15; every line links to the grouped errors behind it.
GET /api/myprofile/visibility/study/:id/blindedScopes |
15 bad of 15 calls · 100% of the service's 5xx |
The error shapes behind them
| 145× | request failed — 500 since 2026-10-13 11:01:59 UTC |
| 145× | Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-10-13 11:32:37 UTC |
| 145× | could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-10-13 11:32:37 UTC |
| 21× | Error getting folder manager — FolderManager(StudyInstance) not found: <nil> since 2026-11-09 15:36:19 UTC |
All onboardingsvc errors, grouped → · service page · its logs
The story — No onboardingsvc deploy or hand change is recorded in the hour before this opened. Everything below is onboardingsvc only.
- 2026-10-13 15:54:07 UTCthis finding onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-17 00:00:00 UTCerrors onboardingsvc 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
History
- 2026-11-16 17:34:53 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794849000, 1794849900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794849000&to=1794850493'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794849000&to=1794850493'}], 'provenance': {'source': 'rollup', 'collected_at': 1794850446, 'age_seconds': 56, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 78.1, 'status': 'open', 'first_seen': 1791906847, 'last_seen': 1794850493, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794850493, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2943655, 'stale_s': 9}
- 2026-11-16 19:45:22 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794849000, 1794849900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794849000&to=1794850493'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794849000&to=1794850493'}], 'provenance': {'source': 'rollup', 'collected_at': 1794850446, 'age_seconds': 56, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 83.1, 'status': 'resolved', 'first_seen': 1791906847, 'last_seen': 1794850493, 'resolved_at': 1794858322, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 56, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794850493, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2951483, 'stale_s': 7837}
- 2026-11-17 00:37:22 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794874200, 1794875100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794874200&to=1794875842'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794874200&to=1794875842'}], 'provenance': {'source': 'rollup', 'collected_at': 1794875679, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 76.8, 'status': 'open', 'first_seen': 1791906847, 'last_seen': 1794875842, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794875842, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2969004, 'stale_s': 9}
- 2026-11-17 02:44:54 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4570, 'fingerprint': 'c7eff385694e63dc81ee', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'production', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794874200, 1794875100], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=production&from=1794874200&to=1794875842'}, {'label': 'Error budgets', 'href': '/budgets?cluster=production&from=1794874200&to=1794875842'}], 'provenance': {'source': 'rollup', 'collected_at': 1794875679, 'age_seconds': 173, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 81.8, 'status': 'resolved', 'first_seen': 1791906847, 'last_seen': 1794875842, 'resolved_at': 1794883494, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 57, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on production failed 15 of 15 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': 'high', 'episode_at': 1794875842, 'chat_thread': 'xray-alert-10-748fd5a928-c7eff385694e63dc81ee-1794604567', 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2976654, 'stale_s': 7659}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794875679 |
| age_seconds | 173 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_failing |