Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 22:14:17.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 68detectors over stored rollups and live sources

What X-ray is prepared to defend about test27 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test27
just test27 · nothing ticked means test27, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 11 warning · 14 info detectors
warning Query shape degrading test27 last seen 1d ago · since 1d ago · 6×

authzd shape 00000000: p95 1.25 s now vs 10 ms earlier (126.8×, 274 runs)

A authzd query shape on test27 has p95 1.2 s in the second half of the window against 10 ms in the first (126.8×, 274 executions). The shape reads: “MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Query shape degrading test27 Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 5d ago · 9×

enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)

A enrollsvc query shape on test27 has p95 3.0 s in the second half of the window against 296 ms in the first (10.1×, 696 executions). The shape reads: “MATCH (d:Document {tildaid: $document_id}) MATCH (d)-[:DOC_TO_ORG]-(o:Organization) MATCH (d)-[:DOCUMENT_VERSION]->(dv:D…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.

warning Autoscaler blind test27 resolved last seen 1d ago · since 26d ago · 1071×

HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h

Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 15× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 15, last_message unable to get metric custom.googleapis.com|docproc_memory_demand_bytes: no metrics returned from custom metrics API.

warning LLM cost off the provider's bill test27 last seen 1d ago · since 37d ago · 73×

test27: LiteLLM's OpenAI cost is $2.03 (5.0%) over OpenAI's bill over 7 days (2026-11-10 to 2026-11-16)

Over 7 UTC days (2026-11-10 to 2026-11-16) OpenAI billed test27 $40.21; the LiteLLM log says $42.24, a difference of $2.03 (5.0%, log above the bill), past the 1.0% this alert allows. Largest gaps: gpt-5.6-terra billed $35.26, logged $37.23; gpt-5.4 billed $3.02, logged $3.04; gpt-6.1-terra billed $1.84, logged $1.86. Any charge worked out from the log inherits that difference; OpenAI's bill is the authority.

warning Background job failing test27 resolved last seen 1d ago · since 26d ago · 63×

pubsub.dashboards-test27-dashsub: 5 failed runs in the last 6 h — SyncForAIRun: merge recommendations: MergeRecommendations: failed to fetch existing IDs: failed to q

The background job pubsub.dashboards-test27-dashsub on test27 failed 5 runs recently (1,143 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning Error never seen before test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 1d ago · 3×

docproc: a new error shape, 10× in the last hour, never seen in 90 days (Pipeline failed - full context)

docproc logged “Pipeline failed - full context” 10 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 22:35 UTC, raised at orchestration/document_orchestrator.py:5044). Example trace: a4176774f1f34d27f33461722997e92e.

info Traffic anomaly test27 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 26d ago · 52×

docsvc request rate 6.4 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 20:00 UTC (3 of the last 4 checks)

docsvc's request rate is 6.4 req/min on test27, above a normal of 0.2 req/min ± 0.3 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 20.9, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Service outlier test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 2d ago · since 76d ago · 764×

chat-agent: p99 latency (ms) 42.90 s — 12.5× the fleet's usual 3.42 s (also p95 latency (ms) 9.61 s)

chat-agent differs from the other 8 services on 2 measures over six hours: p99 latency (ms) 42.90 s against the peers' 3.42 s (15.8 MADs above); p95 latency (ms) 9.61 s against the peers' 689 ms (9.8 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

warning Error never seen before test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 2d ago · 2×

tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held)

tmfsync logged “identical sync: document held” 6 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 19:10 UTC, raised at identical/pair_sync.py:326). Example trace: 32cac73727c15b60a6dff6e266c3f6cb.

warning Error never seen before test27 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 2d ago · 2×

tmfsync: a new error shape, 30× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )

tmfsync logged “ServiceTokenManager refresh failed (? Client Error: Too Many Requests for url: https://login.tilda.pizza/oauth/token) with no usable cached token; backing off 6” 30 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 04:26 UTC, raised at tilda_auth/service_token_manager.py:185). Example trace: e283b50f44a1b2de3e17c9d05e4c3c2a.