Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:18:11.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 98detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
info Service outlier master Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) last seen 1d ago · since 26d ago · 932×

chat-agent: p99 latency (ms) 1.5 min — 23.7× the fleet's usual 3.87 s (also p95 latency (ms) 8.25 s)

chat-agent differs from the other 12 services on 2 measures over six hours: p99 latency (ms) 1.5 min against the peers' 3.87 s (29.4 MADs above); p95 latency (ms) 8.25 s against the peers' 1.28 s (6.7 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Service outlier master Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 26d ago · 1218×

workplansvc: Neo4j queries per request 622.4 — 140.7× the fleet's usual 4.4

workplansvc's Neo4j queries per request is 622.4 against the 12 other services' median of 4.4 (112.1 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.

info Neo4j load anomaly master Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 70d ago · 279×

studiosvc Neo4j load 118 q/min vs a normal 3 q/min ± 5 q/min for Tue 10:00 UTC (3 of the last 4 checks)

studiosvc's Neo4j load is 118 q/min on master, above a normal of 3 q/min ± 5 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 24.2, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning Background job failing master last seen 1d ago · since 71d ago · 326×

pubsub.training-assignment-master-trainingsub: 21 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj

The background job pubsub.training-assignment-master-trainingsub on master failed 21 runs recently (25 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

info Service outlier master Owner: Suresh Boddapati (backup: Madhava Mukkamala) last seen 1d ago · since 83d ago · 2381×

studiosvc: p95 latency (ms) 1.8 min — 85.2× the fleet's usual 1.28 s (also p99 latency (ms) ≥ 2.0 min)

studiosvc differs from the other 12 services on 2 measures over six hours: p95 latency (ms) 1.8 min against the peers' 1.28 s (102.6 MADs above); p99 latency (ms) ≥ 2.0 min against the peers' 3.87 s (38.8 MADs above). Peer comparison is structural more often than it is news — a service that does more work per request will sit here for as long as it does — which is why this is informational unless it is about failures.

info Neo4j load anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) last seen 1d ago · since 84d ago · 309×

execsvc Neo4j load 24 q/min vs a normal 0 q/min ± 1 q/min for Tue 10:00 UTC (3 of the last 4 checks)

execsvc's Neo4j load is 24 q/min on master, above a normal of 0 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 31.2, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

warning 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 7d ago · 16×

studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1

4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 210, c4xx 37, share_pct 17.61904761904762, c401 0.

critical 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 15d ago · 25×

studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8

4xx spike (403/401/404…): studiosvc GET /api/studio/events: permission denied — 36 of 49 calls (73 %; 401×36) in the last 15 min — 44× its usual 0.8. Facts: endpoint GET /api/studio/events, service studiosvc, calls 49, c4xx 36, share_pct 73.46938775510205, c401 36.

First look low confidence
Cause not clear from logs and recent changes; studiosvc GET /api/studio/events returned 401 on 21 of 49 calls (43 %) in 15 min, 28× its usual 0.8 per window.
Onset 15:25 UTC on 2026-11-16; the first 401 in the request log is at 15:28 UTC. This finding has recurred many times since 2026-11-03, and earlier episodes cleared on their own in about 20–30 min.
Checked: 0 stored error lines, 4 config-only autosync deploys (the last at 08:05 UTC, 7 h before) with no image or commit change, and the SSE handler. The handler returns 401 when authlib.GetRequestContext fails and logs a warning, but no such lines are stored.
Next step: open one of the 401 traces to see whether the auth context or Organization header is missing, and compare with the open sibling finding on /api/studio/triggers (100 % 401). · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 13d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; Neo4j shared, database matrix_b: 35 Cypher queries failed in the last 15 min (213 occurrences since onset).
Failing since 2026-11-04 14:19 UTC, still seen 2026-11-17 10:21 UTC; matrix_a also started failing 17 min later (7 queries).
Checked: 12 deploys in the 24 h before onset (11 config syncs at revision c8a04003ea3d by [email protected] 205 min before, dashrsvc autosync 67 min before). Nothing links them to Neo4j.
None of the stored cluster error shapes are Neo4j or Cypher errors, and no single service owns the queries.
Next step: open the Neo4j error logs for the window to get the failing Cypher error codes and the client issuing them. · its evidence →