Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 00:17:50.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 118detectors over stored rollups and live sources

What X-ray is prepared to defend about test26 right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
test26
just test26 · nothing ticked means test26, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 0 critical · 10 warning · 8 info detectors
info Neo4j load anomaly test26 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 84d ago · 153×

docsvc Neo4j load 180 q/min vs a normal 1 q/min ± 1 q/min for Mon 06:00 UTC (3 of the last 4 checks)

docsvc's Neo4j load is 180 q/min on test26, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 301.2, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test26 resolved last seen 2d ago · since 84d ago · 1164×

test26 (whole cluster) Neo4j load 563 q/min vs a normal 110 q/min ± 95 q/min for Mon 06:00 UTC (3 of the last 4 checks)

the whole test26 cluster's Neo4j load is 563 q/min on test26, above a normal of 110 q/min ± 95 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 4.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Traffic anomaly test26 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 20d ago · 3×

docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)

docsvc's request rate is 28.8 req/min on test26, above a normal of 0.0 req/min ± 5.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test26 Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 2d ago · since 69d ago · 79×

execsvc Neo4j load 117 q/min vs a normal 0 q/min ± 20 q/min for Mon 06:00 UTC (4 of the last 4 checks)

execsvc's Neo4j load is 117 q/min on test26, above a normal of 0 q/min ± 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 5.8, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly test26 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 76d ago · 426×

virtual-assistant Neo4j load 427 q/min vs a normal 1 q/min ± 0 q/min for Mon 06:00 UTC (3 of the last 4 checks)

virtual-assistant's Neo4j load is 427 q/min on test26, above a normal of 1 q/min ± 0 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 2874.7, 5 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Error never seen before test26 Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 2d ago · since 2d ago · 3×

execsvc: a new error shape, 20× in the last hour, never seen in 90 days (could not provision managed-SMO org for site create — create doc seque)

execsvc logged “could not provision managed-SMO org for site create — create doc sequence counter: Object tildaobj.DocSeqCounter not registerd with db” 20 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 05:29 UTC, raised at repo/site.go:157). Example trace: 46cdd9f3abee4f38b4518b79951c72fb.

warning Endpoint failing test26 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 2d ago · 4×

docsvc PUT /api/documents/promote: 16 of 16 calls failed (100 %) in the last 15 min

docsvc's PUT /api/documents/promote on test26 failed 16 of 16 calls (100 %) in the last 15 min. One endpoint, not the service: the rest of docsvc is judged on its own. An alert in window mode counts these failed calls per observation window and resolves the endpoint after its quiet windows.

docproc: a new error shape 3 targets worst: warning 8× in the last hour, never seen in 90 days (needs_attention failed for ?: HTTP ?: {?:?}, 5× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse, 8× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests
warning Error never seen before test26 resolved last seen · since · 1×

docproc: a new error shape, 8× in the last hour, never seen in 90 days (needs_attention failed for ?: HTTP ?: {?:?})

docproc logged “needs_attention failed for ?: HTTP ?: {?:?}” 8 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 05:07 UTC, raised at processors/needs_attention.py:68). Example trace: fe55f3541b90ff52acc36c9fbb2ed5f2.

warning Error never seen before test26 resolved last seen · since · 1×

docproc: a new error shape, 5× in the last hour, never seen in 90 days (Failed to create document processing AIRun — aborting pipeline (refuse)

docproc logged “Failed to create document processing AIRun — aborting pipeline (refuse to run untracked) — No docsvc authorization (caller token or TildaSense bot token)” 5 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 00:10 UTC, raised at orchestration/document_orchestrator.py:4480). Example trace: f75d7c46a43d56a1196530409f70d087.

warning Error never seen before test26 resolved last seen · since · 2×

docproc: a new error shape, 8× in the last hour, never seen in 90 days (ServiceTokenManager refresh failed (? Client Error: Too Many Requests )

docproc logged “ServiceTokenManager refresh failed (? Client Error: Too Many Requests for url: https://login.tilda.pizza/oauth/token) with no usable cached token; backing off 6” 8 times in the last hour and never before in 90 days on test26 (first seen 2026-11-16 00:10 UTC, raised at tilda_auth/service_token_manager.py:185). Example trace: f75d7c46a43d56a1196530409f70d087.

info Neo4j load anomaly test26 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) resolved last seen 2d ago · since 84d ago · 113×

docproc Neo4j load 343 q/min vs a normal 13 q/min ± 4 q/min for Mon 05:00 UTC (3 of the last 4 checks)

docproc's Neo4j load is 343 q/min on test26, above a normal of 13 q/min ± 4 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 93.0, 6 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Error budget burning test26 Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 84d ago · 174×

docsvc is burning its error budget at 6.5× (1 h) / 1.2× (6 h) against a 97.00% objective

docsvc is spending its error budget 6.5× faster than the objective allows over the last hour (1.2× over six hours). The objective is 97.0%, derived — a baseline, not a promise. 26 bad events against 731 calls in the window.