Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 5 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:18:17.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 98detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
info Neo4j load anomaly master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 62d ago · 79×

onboardingsvc Neo4j load 48 q/min vs a normal 5 q/min ± 7 q/min for Mon 12:00 UTC (3 of the last 4 checks)

onboardingsvc's Neo4j load is 48 q/min on master, above a normal of 5 q/min ± 7 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Traffic anomaly master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 83d ago · 295×

onboardingsvc request rate 61.6 req/min vs a normal 6.3 req/min ± 9.3 req/min for Mon 12:00 UTC (3 of the last 4 checks)

onboardingsvc's request rate is 61.6 req/min on master, above a normal of 6.3 req/min ± 9.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Query shape degrading master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 2d ago · since 2d ago · 3×

studiosvc shape 8D3802C7: p95 1.73 s now vs 746 ms earlier (2.3×, 232 runs)

A studiosvc query shape on master has p95 1.7 s in the second half of the window against 746 ms in the first (2.3×, 232 executions). The shape reads: “MATCH (proto:FormV2 {tildaid: $protoId})-[pr:PROTOCOL_TO_GRAPH_NODE_REF]->(ref:GraphNodeRef) MATCH (ref)-[:GRAPH_NODE_RE…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

critical Endpoint failing master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 29d ago · 10×

docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min

docsvc's POST /api/documents/folders on master failed 8 of 8 calls (100 %) in the last 15 min. automated changed docsvc 23 minutes earlier — adjacency, not proof. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

First look low confidence
docsvc POST /api/documents/folders: 8 of 8 calls failed (100 %) in the last 15 min.
Cause not clear from logs and recent changes; checked: error lines (none stored), failing requests (8), deploys in the 24 h before (3), commits in the deploy's range (0), the code at the failing place (1), this finding's history, other findings at the same time (1). · its evidence →

warning Background job failing master resolved last seen 2d ago · since 2d ago · 7×

docproc.process: 8 failed runs in the last 6 h — No docsvc authorization (caller token or TildaSense bot token)

The background job docproc.process on master failed 8 runs recently (71 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

virtual-assistant: a new error shape 3 targets worst: warning 5× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e, 7× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at, 12× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er
warning Error never seen before master last seen · since · 3×

virtual-assistant: a new error shape, 5× in the last hour, never seen in 90 days (Token refresh failed: Login failed: status=? error=too_many_attempts e)

virtual-assistant logged “Token refresh failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. ” 5 times in the last hour and never before in 90 days on master (first seen 2026-11-16 00:37 UTC, raised at api/token_manager.py:155). Example trace: 3a84899fc4010a8af80b67f59dcdb5f9.

warning Error never seen before master last seen · since · 3×

virtual-assistant: a new error shape, 7× in the last hour, never seen in 90 days (Error fetching document info: Login failed: status=? error=too_many_at)

virtual-assistant logged “Error fetching document info: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login at” 7 times in the last hour and never before in 90 days on master (first seen 2026-11-16 00:37 UTC, raised at document_subscriber/document_events.py:165). Example trace: 3a84899fc4010a8af80b67f59dcdb5f9.

warning Error never seen before master last seen · since · 3×

virtual-assistant: a new error shape, 12× in the last hour, never seen in 90 days (HTTP request failed: Login failed: status=? error=too_many_attempts er)

virtual-assistant logged “HTTP request failed: Login failed: status=? error=too_many_attempts error_description=Your account has been blocked after multiple consecutive login attempts. W” 12 times in the last hour and never before in 90 days on master (first seen 2026-11-16 00:37 UTC, raised at api/token_manager.py:87). Example trace: 3a84899fc4010a8af80b67f59dcdb5f9.

docproc: a new error shape 9 targets worst: warning 5× in the last hour, never seen in 90 days (docsvc checksum update skipped: no agent token (configure TildaSense b, 5× in the last hour, never seen in 90 days ([?] PIPELINE FAILED — 101_1572_Ruiz_02Nov2026.pdf — Error: docsvc chec, 5× in the last hour, never seen in 90 days ([?] PIPELINE FAILED — 101_1572_Ruiz_02Nov2026.pdf — RuntimeError: docs, 5× in the last hour, never seen in 90 days ([?] Pubsub:Attribute Update failed: AIRun creation failed for document, 5× in the last hour, never seen in 90 days (docsvc call not possible, 16× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests and 3 more
warning Error never seen before master resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days (docsvc checksum update skipped: no agent token (configure TildaSense b)

The docproc service logged "docsvc checksum update skipped: no agent token (configure TildaSense bot token)" 5 times within a one-hour window on 2026-11-15 23:51 UTC, with no prior occurrences in the previous 90 days on master. The error originated from orchestration/document_orchestrator.py:950, with example trace 5f3319ddb12ce82aaafa6f250c5a20cb.

warning Error never seen before master resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days ([?] PIPELINE FAILED — 101_1572_Ruiz_02Nov2026.pdf — Error: docsvc chec)

The docproc service logged "[?] PIPELINE FAILED — 101_1572_Ruiz_02Nov2026.pdf — Error: docsvc checksum update skipped: no agent token (configure TildaSense bot token)" 5 times within the last hour. This error was never seen before in 90 days on master, first appearing 2026-11-15 23:51 UTC at utils/processing_logger.py:190. Example trace: 5f3319ddb12ce82aaafa6f250c5a20cb.

warning Error never seen before master resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days ([?] PIPELINE FAILED — 101_1572_Ruiz_02Nov2026.pdf — RuntimeError: docs)

docproc encountered a RuntimeError 5 times in the last hour, with no prior occurrences in 90 days on master. The error: "docsvc checksum update skipped: no agent token (configure TildaSense bot token)" affecting file 101_1572_Ruiz_02Nov2026.pdf. First seen 2026-11-15 23:51 UTC at utils/processing_logger.py:190. Example trace: 5f3319ddb12ce82aaafa6f250c5a20cb.

warning Error never seen before master resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days ([?] Pubsub:Attribute Update failed: AIRun creation failed for document)

docproc logged "[?] Pubsub:Attribute Update failed: AIRun creation failed for document ? (document processing): No docsvc authorization (caller token or TildaSense bot token)" 5 times in the last hour, with no prior occurrences in 90 days on master (first seen 2026-11-15 21:51 UTC, raised at utils/processing_logger.py:190). Example trace: 1100120a97ce52abaaa56c45db5be1b2.

warning Error never seen before master resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days (docsvc call not possible)

The docproc service logged "docsvc call not possible" 5 times in the last hour, with no prior occurrences in the past 90 days on master (first seen 2026-11-15 21:18 UTC). The error was raised at clients/docsvc_client.py:52, with example trace cc89b9ba100e89417f348a2fce85ee2b.

warning Error never seen before master resolved last seen · since · 2×

docproc: a new error shape, 16× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )

docproc logged “ServiceTokenManager refresh failed (? Client Error: Too Many Requests for url: https://login.tilda.pizza/oauth/token) with no usable cached token; backing off 6” 16 times in the last hour and never before in 90 days on master (first seen 2026-11-15 21:16 UTC, raised at tilda_auth/service_token_manager.py:185). Example trace: 9cded473aed184d50d489b938ac5b8e4.

warning Error never seen before master resolved last seen · since · 3×

docproc: a new error shape, 10× in the last hour, never seen in 90 days (Recovery failed — AIRun creation failed for document ? (document proce)

docproc logged “Recovery failed — AIRun creation failed for document ? (document processing): No docsvc authorization (caller token or TildaSense bot token)” 10 times in the last hour and never before in 90 days on master (first seen 2026-11-15 21:17 UTC, raised at handlers/pubsub_handler.py:875). Example trace: 0bd830b6db5a709ea508e58447ba8b64.

warning Error never seen before master resolved last seen · since · 3×

docproc: a new error shape, 19× in the last hour, never seen in 90 days (docsvc PATCH aiEnabled failed before creating AI run)

docproc logged “docsvc PATCH aiEnabled failed before creating AI run” 19 times in the last hour and never before in 90 days on master (first seen 2026-11-15 21:16 UTC, raised at queries/ai_run_queries.py:294). Example trace: 9cded473aed184d50d489b938ac5b8e4.

warning Error never seen before master resolved last seen · since · 3×

docproc: a new error shape, 19× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)

docproc logged “Failed to create document processing AIRun — aborting pipeline (refuse to run untracked) — No docsvc authorization (caller token or TildaSense bot token)” 19 times in the last hour and never before in 90 days on master (first seen 2026-11-15 21:16 UTC, raised at orchestration/document_orchestrator.py:4514). Example trace: 9cded473aed184d50d489b938ac5b8e4.

warning Error never seen before master last seen 2d ago · since 2d ago · 3×

studioai-master: a new error shape, 6× in the last hour, never seen in 90 days (Token refresh failed: Login failed: {?:?,?:?})

studioai-master logged “Token refresh failed: Login failed: {?:?,?:?}” 6 times in the last hour and never before in 90 days on master (first seen 2026-11-15 22:53 UTC, raised at handlers/token_manager.py:77). Example trace: 851e1a1616ecdb490874fff09d26be06.

warning Error budget burning master Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 11d ago · 63×

docproc is burning its error budget at 0.0× (1 h) / 6.0× (6 h) against a 99.50% objective

docproc is spending its error budget 0.0× faster than the objective allows over the last hour (6.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 6 bad events against 199 calls in the window.