onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4
warning
4xx spike (403/401/404…)
automation
resolved
impact 35
Owner: Prasad Limaye (backup: Sambasiva Bandarupalli)
opened 19d ago · last observed 3d ago · observed 11× · #12667 · b8f364a8e3f32885d766
What happened template
4xx spike (403/401/404…): onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4. Facts: endpoint GET /api/sites/staff/output, service onboardingsvc, calls 31, c4xx 31, share_pct 100.0, c401 0.
An endpoint answering far more 4xx than its own usual rate — a permissions change (403), expired or broken auth (401), a missing route or contract break (404). Its usual rate is its last seven days.
The facts
| endpoint | GET /api/sites/staff/output |
| service | onboardingsvc |
| calls | 31 |
| c4xx | 31 |
| share_pct | 100.00 |
| c401 | 0 |
| c403 | 0 |
| other_4xx | 31 |
| rejecting | yes |
| usual_per_window | 0.38 |
| baseline_days | 7 |
| window_min | 15 |
| discriminator | GET /api/sites/staff/output |
| window | 2026-11-16 07:10:00 UTC → 2026-11-16 07:25:00 UTC |
What is failing in onboardingsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from onboardingsvc. In this finding's window (2026-11-16 07:10:00 UTC → 2026-11-16 07:25:00 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 840× | request failed — 500 since 2026-09-14 00:17:50 UTC |
| 170× | Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-09-26 01:48:22 UTC |
| 170× | could not get study for blinded-scope lookup — objLabel Study: get with empty tildaId requires a struct with at least on… since 2026-09-15 02:42:25 UTC |
| 165× | Error getting folder manager — FolderManager(StudyInstance) not found: <nil> since 2026-11-07 05:46:31 UTC |
| 88× | Request completed — 500 · Could not initialize an organization: organization name cannot be empty since 2026-09-25 09:01:42 UTC |
All onboardingsvc errors, grouped → · service page · its logs
The story — No onboardingsvc deploy or hand change is recorded in the hour before this opened. Everything below is onboardingsvc only.
- 2026-10-31 11:10:58 UTCthis finding onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4
- 2026-11-14 15:59:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 16:17:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 16:28:38 UTCfinding · critical onboardingsvc: a new error shape, ≥24000× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 17:39:13 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 17:54:13 UTCfinding · critical onboardingsvc: a new error shape, 115× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = PermissionDenied desc = Us)
- 2026-11-14 18:05:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:09:05 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:15:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:29:23 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:32:55 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:35:48 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:46:19 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:46:45 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 18:50:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:00:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:01:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:11:17 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:11:41 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:20:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:21:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:26:07 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:27:00 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:40:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:40:56 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:48:21 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:48:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:53:44 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:54:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 19:58:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:07:58 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:18:43 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:19:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:24:55 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:39:29 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 20:46:26 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-14 21:02:54 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 00:00:00 UTCerrors onboardingsvc 5xx peaked at 87 in one five-minute bucket (median 0 in this window)
- 2026-11-15 07:17:18 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:28:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:43:49 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-15 07:59:30 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:46:11 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 05:56:46 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 06:49:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:02:22 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:18:08 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:32:52 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 07:49:06 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/onboardingsvc by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
History
- 2026-11-16 02:02:27 UTCreopened x-ray · severity=warning was=warning title=onboardingsvc GET /api/sites/staff/output: 4xx — 59 of 59 calls (100 %; other 4xx×59) in the last 15 min — 251× its usual 0.2 level=medium was_level=medium finding={'id': 12667, 'fingerprint': 'b8f364a8e3f32885d766', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/sites/staff/output: 4xx — 59 of 59 calls (100 %; other 4xx×59) in the last 15 min — 251× its usual 0.2', 'facts': {'endpoint': 'GET /api/sites/staff/output', 'service': 'onboardingsvc', 'calls': 59, 'c4xx': 59, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 59, 'rejecting': True, 'usual_per_window': 0.24, 'baseline_days': 7, 'window': [1794792900, 1794793800], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/sites/staff/output'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794792900&to=1794793800&service=onboardingsvc&status=4xx'}, {'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794792900&to=1794793800'}], 'provenance': {'source': 'rollup', 'collected_at': 1794794318, 'age_seconds': 235, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 33.4, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794794547, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': '4xx spike (403/401/404…): onboardingsvc GET /api/sites/staff/output: 4xx — 72 of 72 calls (100 %; other 4xx×72) in the last 15 min — 968× its usual 0.1. Facts: endpoint GET /api/sites/staff/output, service onboardingsvc, calls 72, c4xx 72, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/sites/staff/output', 'level': 'medium', 'told_level': None, 'episode_at': 1794794547, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1349494, 'stale_s': 5}
- 2026-11-16 02:59:47 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12667, 'fingerprint': 'b8f364a8e3f32885d766', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/sites/staff/output: 4xx — 72 of 72 calls (100 %; other 4xx×72) in the last 15 min — 306× its usual 0.2', 'facts': {'endpoint': 'GET /api/sites/staff/output', 'service': 'onboardingsvc', 'calls': 72, 'c4xx': 72, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 72, 'rejecting': True, 'usual_per_window': 0.24, 'baseline_days': 7, 'window': [1794793500, 1794794400], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/sites/staff/output'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794793500&to=1794794400&service=onboardingsvc&status=4xx'}, {'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794793500&to=1794794400'}], 'provenance': {'source': 'rollup', 'collected_at': 1794794939, 'age_seconds': 66, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 37.7, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794794999, 'resolved_at': 1794797987, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': '4xx spike (403/401/404…): onboardingsvc GET /api/sites/staff/output: 4xx — 72 of 72 calls (100 %; other 4xx×72) in the last 15 min — 306× its usual 0.2. Facts: endpoint GET /api/sites/staff/output, service onboardingsvc, calls 72, c4xx 72, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/sites/staff/output', 'level': 'medium', 'told_level': None, 'episode_at': 1794794547, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1352934, 'stale_s': 2993}
- 2026-11-16 07:34:54 UTCreopened x-ray · severity=warning was=warning title=onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4 level=medium was_level=medium finding={'id': 12667, 'fingerprint': 'b8f364a8e3f32885d766', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4', 'facts': {'endpoint': 'GET /api/sites/staff/output', 'service': 'onboardingsvc', 'calls': 31, 'c4xx': 31, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 31, 'rejecting': True, 'usual_per_window': 0.38, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/sites/staff/output'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794813000&to=1794813900&service=onboardingsvc&status=4xx'}, {'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814425, 'age_seconds': 76, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 30.4, 'status': 'open', 'first_seen': 1793445058, 'last_seen': 1794814494, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': '4xx spike (403/401/404…): onboardingsvc GET /api/sites/staff/output: 4xx — 72 of 72 calls (100 %; other 4xx×72) in the last 15 min — 306× its usual 0.2. Facts: endpoint GET /api/sites/staff/output, service onboardingsvc, calls 72, c4xx 72, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/sites/staff/output', 'level': 'medium', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1369443, 'stale_s': 7}
- 2026-11-16 07:51:56 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 12667, 'fingerprint': 'b8f364a8e3f32885d766', 'kind': 'endpoint.4xx_spike', 'severity': 'warning', 'cluster': 'automation', 'subject': 'onboardingsvc', 'title': 'onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4', 'facts': {'endpoint': 'GET /api/sites/staff/output', 'service': 'onboardingsvc', 'calls': 31, 'c4xx': 31, 'share_pct': 100.0, 'c401': 0, 'c403': 0, 'other_4xx': 31, 'rejecting': True, 'usual_per_window': 0.38, 'baseline_days': 7, 'window': [1794813000, 1794813900], 'window_min': 15, 'recent_deploys': [], 'discriminator': 'GET /api/sites/staff/output'}, 'evidence': [{'label': 'The 4xx calls (status, caller, path)', 'href': '/api-calls?cluster=automation&from=1794813000&to=1794813900&service=onboardingsvc&status=4xx'}, {'label': 'onboardingsvc service page', 'href': '/service/onboardingsvc?cluster=automation&from=1794813000&to=1794813900'}], 'provenance': {'source': 'rollup', 'collected_at': 1794814425, 'age_seconds': 76, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 35.4, 'status': 'resolved', 'first_seen': 1793445058, 'last_seen': 1794814494, 'resolved_at': 1794815516, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': '4xx spike (403/401/404…): onboardingsvc GET /api/sites/staff/output: 4xx — 31 of 31 calls (100 %; other 4xx×31) in the last 15 min — 81× its usual 0.4. Facts: endpoint GET /api/sites/staff/output, service onboardingsvc, calls 31, c4xx 31, share_pct 100.0, c401 0.', 'narrative_kind': 'template', 'discriminator': 'GET /api/sites/staff/output', 'level': 'medium', 'told_level': None, 'episode_at': 1794814494, 'chat_thread': None, 'kind_label': '4xx spike (403/401/404…)', 'group': 'Errors', 'age_s': 1370465, 'stale_s': 1029}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794814425 |
| age_seconds | 76 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | endpoint_4xx_spike |