studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^
warning
Background job failing
test27
open
impact 25
opened 21d ago · last observed 1d ago · observed 34× · #7584 · 8fd834585d2e3670ca5c
What happened template
The background job studioai.unknown on test27 failed 5 runs recently (532 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
A named background job whose runs keep failing (error-severity bursts) in the last hour — volume, not novelty; the novel-shape detector only fires the first time.
The facts
| job | studioai.unknown |
| service | studioai-test27 |
| failed_runs | 5 |
| error_lines | 532 |
| sample | Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 494, in _make_req |
| error_signature | Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File ?, |
| failing_at | generate_graph/import_global.py:1562 |
| trace | projects/tilda-trial-dev/traces/a28d19bdccf3a45a036c67d0b453203d |
| detected_via | ERROR-severity log lines |
| note | a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name. |
| discriminator | |
| window | 2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-27 13:53:04 UTCcypher ontologysvc shape 00000000 p95 went 47 ms → 717 ms across the window (364 runs)
- 2026-10-27 14:53:04 UTCthis finding studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^
- 2026-10-28 18:44:03 UTCfinding · warning onboardingsvc GET /api/organizations/myCro: permission denied — 85 of 85 calls (100 %; 403×85) in the last 15 min — it usually has none
- 2026-11-02 06:54:56 UTCfinding · warning studiosvc GET /api/studio/triggers: permission denied — 84 of 84 calls (100 %; 401×84) in the last 15 min — 5× its usual 16.3
- 2026-11-02 09:10:39 UTCfinding · warning pubsub.onboarding-test27-regsvc-sub: 19 failed runs in the last 6 h — failed to create triggers: failed to fetch existing triggers: failed to fetch triggers: Error from A
- 2026-11-05 16:33:42 UTCfinding · warning studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)
- 2026-11-05 22:27:19 UTCfinding · info studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)
- 2026-11-12 14:41:45 UTCfinding · warning execsvc POST /api/trialexec/v2/study/:id/activity/details: permission denied — 276 of 622 calls (44 %; 403×276) in the last 15 min — 522× its usual 0.5
- 2026-11-13 08:19:38 UTCfinding · warning enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)
- 2026-11-14 01:12:42 UTCfinding · warning regsvc (tilda) image pull failing for at least 117 min — 1 pod
- 2026-11-14 12:00:44 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 12:01:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:26:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 17:58:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 06:12:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 06:40:00 UTCerrors cluster 5xx peaked at 34 in one five-minute bucket (median 0 in this window)
- 2026-11-15 06:52:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-15 15:15:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 15:15:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:03:03 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:32:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 16:35:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:14:46 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:25:50 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 17:31:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 19:09:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:16:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 19:17:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 20:26:13 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 20:39:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 21:16:01 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 22:15:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-15 22:15:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 01:14:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 04:26:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 05:10:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 06:42:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:54:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 06:55:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 07:02:15 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:10:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:15:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:50:00 UTClatency cluster p95 peaked at 1.5 min
- 2026-11-16 08:20:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 08:30:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:14:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:15:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:16 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:16:19 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:18:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 10:23:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:24:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 10:25:07 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 15:00:56 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 15:18:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.33.4 (darwin/arm64) kubernetes/74cdb42,gzip(gfe))
- 2026-11-16 15:27:04 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 16:28:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 16:38:14 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:27 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:38:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:05 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:41:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:41:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:46:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 16:51:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed core/v1/namespaces/tilda/services/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:51 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/sheet-edit-svc by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 16:51:59 UTChand change [email protected] changed core/v1/namespaces/tilda/secrets/sheet-edit-svc-syncfusion by hand (kubectl/v1.36.4 (darwin/arm64) kubernetes/bb826b1,gzip(gfe))
- 2026-11-16 17:01:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 17:53:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:04:21 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 18:30:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:34 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:36 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:40:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:41:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:48:43 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 18:53:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:54:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:58:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 19:09:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:30:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 19:53:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.33.3 (darwin/amd64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 20:24:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:37:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 20:50:35 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:30:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/workplansvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:31:28 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:35:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 21:46:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:14:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:33:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:38:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 22:41:00 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 22:42:47 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docproc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-17 01:09:53 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 01:10:57 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 01:17:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 02:30:29 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 02:30:30 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 05:01:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-17 05:18:28 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 05:55:48 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-17 05:56:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 06:05:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 06:31:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 06:44:37 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-17 06:45:49 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-17 06:54:11 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 06:58:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-17 07:03:34 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-17 08:50:40 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-17 09:56:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
History
- 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^ level=medium was_level=medium finding={'id': 7584, 'fingerprint': '8fd834585d2e3670ca5c', 'kind': 'job.failing', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studioai.unknown', 'title': 'studioai.unknown: 5 failed runs in the last 6 h — Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^', 'facts': {'job': 'studioai.unknown', 'service': 'studioai-test27', 'failed_runs': 5, 'error_lines': 532, 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 494, in _make_req', 'error_signature': 'Traceback (most recent call last): File ?, line ?, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File ?,', 'failing_at': 'generate_graph/import_global.py:1562', 'trace': 'projects/tilda-trial-dev/traces/a28d19bdccf3a45a036c67d0b453203d', 'detected_via': 'ERROR-severity log lines', 'window': [1794890441, 1794912041], 'note': 'a run is a burst of lines that truly look like failures — an error/exception field, an error-worded message, or a failure status; lines within one burst count as one run. Lines a job dumps at ERROR that are neither (the steps that SUCCEEDED, banner rules) are ignored, so a service that logs its whole context on failure is not miscounted. `failing_at` is the call site those failures came from — for a coarse job (one set_job wrapping a pipeline) more specific than the job name.', 'discriminator': ''}, 'evidence': [{'label': "This job's page", 'href': '/jobs/studioai.unknown?cluster=test27&from=1794890441&to=1794912041'}, {'label': 'Background jobs', 'href': '/jobs?cluster=test27&from=1794890441&to=1794912041'}, {'label': 'Its error logs', 'href': '/logs?cluster=test27&from=1794890441&to=1794912041&severity=ERROR&q=studioai.unknown'}, {'label': 'Example trace', 'href': '/trace/projects/tilda-trial-dev/traces/a28d19bdccf3a45a036c67d0b453203d?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'error-severity job lines from BigQuery, last hour, burst-grouped per job'}, 'impact': 20.0, 'status': 'open', 'first_seen': 1793112784, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 34, 'narrative': "The background job studioai.unknown on test27 failed 5 runs recently (859 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'medium', 'told_level': None, 'episode_at': 1794912041, 'chat_thread': None, 'kind_label': 'Background job failing', 'group': 'Jobs', 'age_s': 1799384, 'stale_s': 127}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | error-severity job lines from BigQuery, last hour, burst-grouped per job |
| detector | job_failing |