onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min
critical
Endpoint failing
automation
resolved
impact 56
Owner: Prasad Limaye (backup: Sambasiva Bandarupalli)
opened 27d ago · last observed 2d ago · observed 147× · #5476 · 635f7e6df4b6146018f4
What happened template
onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.
One endpoint of a service failing most of its calls (5xx) in the last 15 minutes; 4xx are shown beside it, not counted as failing.
The facts
| endpoint | GET /api/myprofile/visibility/study/:id/blindedScopes |
| service | onboardingsvc |
| calls | 8 |
| failed | 8 |
| c4xx | 0 |
| fail_pct | 100.00 |
| window_min | 15 |
| discriminator | GET /api/myprofile/visibility/study/:id/blindedScopes |
| window | 2026-11-16 22:25:00 UTC → 2026-11-16 22:40:00 UTC |
What is failing in onboardingsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from onboardingsvc. In this finding's window (2026-11-16 22:25:00 UTC → 2026-11-16 22:40:00 UTC) there were 8; every line links to the grouped errors behind it.
GET /api/myprofile/visibility/study/:id/blindedScopes |
8 bad of 8 calls · 100% of the service's 5xx |
The error shapes behind them
| 840× | request failed — 500 since 2026-09-14 00:17:50 UTC |
| 170× | Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-09-26 01:48:22 UTC |
| 170× | could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-09-15 02:42:25 UTC |
| 165× | Error getting folder manager — FolderManager(StudyInstance) not found: <nil> since 2026-11-07 05:46:31 UTC |
| 88× | Request completed — 500 · Could not initialize an organization: organization name cannot be empty since 2026-09-25 09:01:42 UTC |
All onboardingsvc errors, grouped → · service page · its logs
The story — No onboardingsvc deploy or hand change is recorded in the hour before this opened. Everything below is onboardingsvc only.
- 2026-10-23 13:23:01 UTCthis finding onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min
- 2026-10-23 13:33:46 UTCfinding · info onboardingsvc request rate 78.2 req/min vs a normal 10.6 req/min ± 15.7 req/min for Tue 08:00 UTC (4 of the last 4 checks)
- 2026-10-31 11:10:58 UTCfinding · warning onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4
- 2026-11-14 15:59:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 16:17:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 16:28:38 UTCfinding · critical onboardingsvc: a new error shape, ≥24000× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 17:39:13 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 17:54:13 UTCfinding · critical onboardingsvc: a new error shape, 115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us)
- 2026-11-14 18:05:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:09:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:55 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:46:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:46:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:00:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:01:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:11:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:11:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:20:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:21:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:27:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:40:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:40:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:48:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:48:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:53:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:54:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:18:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:19:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:55 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:17:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:28:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:43:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:59:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 00:00:00 UTCerrors onboardingsvc 5xx peaked at 110 in one five-minute bucket (median 0 in this window)
- 2026-11-16 05:46:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:56:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 06:49:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:02:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:18:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:05:59 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:21:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 08:44:24 UTCfinding · critical onboardingsvc GET /api/organizations: permission denied — 36 of 39 calls (92 %; 403×36) in the last 15 min — it usually has none
- 2026-11-16 11:32:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:41:27 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:45:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 11:48:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:18:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:40:01 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 12:49:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 13:44:10 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 14:03:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:37:38 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:48:36 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:48:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:53:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:14:41 UTCfinding · critical onboardingsvc: a new error shape, 5× in the last hour, never seen in 90 days (Could not begin transaction — Neo4jError: Neo.ClientError.Database.Dat)
History
- 2026-11-14 21:03:50 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794688800, 1794689700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794688800&to=1794690230'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794688800&to=1794690230'}], 'provenance': {'source': 'rollup', 'collected_at': 1794690153, 'age_seconds': 81, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.0, 'status': 'open', 'first_seen': 1792761781, 'last_seen': 1794690230, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 132, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 9 of 9 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794690230, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 1928453, 'stale_s': 4}
- 2026-11-15 01:08:50 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 9 of 9 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 9, 'failed': 9, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794695700, 1794696600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794695700&to=1794697288'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794695700&to=1794697288'}], 'provenance': {'source': 'rollup', 'collected_at': 1794697188, 'age_seconds': 105, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 55.8, 'status': 'resolved', 'first_seen': 1792761781, 'last_seen': 1794697288, 'resolved_at': 1794704930, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 136, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 9 of 9 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794690230, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 1943156, 'stale_s': 7649}
- 2026-11-15 21:08:16 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 18 of 18 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 18 of 18 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 18, 'failed': 18, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794775500, 1794776400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794775500&to=1794776896'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794775500&to=1794776896'}], 'provenance': {'source': 'rollup', 'collected_at': 1794776833, 'age_seconds': 70, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.6, 'status': 'open', 'first_seen': 1792761781, 'last_seen': 1794776896, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 137, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 9 of 9 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794776896, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2015121, 'stale_s': 6}
- 2026-11-16 01:16:30 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794782700, 1794783600], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794782700&to=1794784267'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794782700&to=1794784267'}], 'provenance': {'source': 'rollup', 'collected_at': 1794784156, 'age_seconds': 117, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 55.6, 'status': 'resolved', 'first_seen': 1792761781, 'last_seen': 1794784267, 'resolved_at': 1794791790, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 140, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794776896, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2030014, 'stale_s': 7528}
- 2026-11-16 02:09:59 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794793500, 1794794400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794793500&to=1794794999'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794793500&to=1794794999'}], 'provenance': {'source': 'rollup', 'collected_at': 1794794939, 'age_seconds': 66, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 52.2, 'status': 'open', 'first_seen': 1792761781, 'last_seen': 1794794999, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 141, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794794999, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2033223, 'stale_s': 5}
- 2026-11-16 04:22:10 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794794100, 1794795000], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794794100&to=1794795483'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794794100&to=1794795483'}], 'provenance': {'source': 'rollup', 'collected_at': 1794795421, 'age_seconds': 67, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 56.8, 'status': 'resolved', 'first_seen': 1792761781, 'last_seen': 1794795483, 'resolved_at': 1794802930, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 142, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794794999, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2041154, 'stale_s': 7452}
- 2026-11-16 04:36:35 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 9 of 9 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 9 of 9 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 9, 'failed': 9, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794802500, 1794803400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794802500&to=1794803795'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794802500&to=1794803795'}], 'provenance': {'source': 'rollup', 'collected_at': 1794803755, 'age_seconds': 46, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 50.9, 'status': 'open', 'first_seen': 1792761781, 'last_seen': 1794803795, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 143, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794803795, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2042019, 'stale_s': 5}
- 2026-11-16 06:49:27 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 6 of 6 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 6, 'failed': 6, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794802800, 1794803700], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794802800&to=1794804225'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794802800&to=1794804225'}], 'provenance': {'source': 'rollup', 'collected_at': 1794804175, 'age_seconds': 56, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 56.4, 'status': 'resolved', 'first_seen': 1792761781, 'last_seen': 1794804225, 'resolved_at': 1794811767, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 144, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 6 of 6 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794803795, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2049991, 'stale_s': 7547}
- 2026-11-16 21:08:58 UTCreopened x-ray · severity=critical was=critical title=onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min level=high was_level=high finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 15, 'failed': 15, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794861900, 1794862800], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794861900&to=1794863338'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794861900&to=1794863338'}], 'provenance': {'source': 'rollup', 'collected_at': 1794863267, 'age_seconds': 78, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 53.7, 'status': 'open', 'first_seen': 1792761781, 'last_seen': 1794863338, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 145, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 6 of 6 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794863338, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2101564, 'stale_s': 7}
- 2026-11-17 00:58:16 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5476, 'fingerprint': '635f7e6df4b6146018f4', 'kind': 'endpoint.failing', 'severity': 'critical', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 8 of 8 calls failed (100 %) in the last 15 min', 'facts': {'endpoint': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'service': 'onboardingsvc', 'calls': 8, 'failed': 8, 'c4xx': 0, 'fail_pct': 100.0, 'window': [1794867900, 1794868800], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes'}, 'evidence': [{'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794867900&to=1794869382'}, {'label': 'Error budgets', 'href': '/budgets?cluster=automation&from=1794867900&to=1794869382'}], 'provenance': {'source': 'rollup', 'collected_at': 1794869290, 'age_seconds': 101, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 55.7, 'status': 'resolved', 'first_seen': 1792761781, 'last_seen': 1794869382, 'resolved_at': 1794877096, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 147, 'narrative': "onboardingsvc's GET /api/myprofile/visibility/study/:id/blindedScopes on automation failed 8 of 8 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of onboardingsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.", 'narrative_kind': 'template', 'discriminator': 'GET /api/myprofile/visibility/study/:id/blindedScopes', 'level': 'high', 'told_level': None, 'episode_at': 1794863338, 'chat_thread': None, 'kind_label': 'Endpoint failing', 'group': 'Errors', 'age_s': 2115324, 'stale_s': 7723}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794869290 |
| age_seconds | 101 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_failing |