Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 16:03:00.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 89detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
warning Autoscaler blind master resolved last seen 1d ago · since 26d ago · 571×

HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h

Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 2, last_message unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric custom.googleapis.com/docproc_pending_workload: googleapi: Error 503: Authentication backend unavailable., backendError.

info Traffic anomaly master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 84d ago · 246×

docsvc request rate 10.2 req/min vs a normal 0.7 req/min ± 1.0 req/min for Mon 16:00 UTC (3 of the last 4 checks)

docsvc's request rate is 10.2 req/min on master, above a normal of 0.7 req/min ± 1.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 9.2, 4 weeks of history). automated changed docsvc 47 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Latency anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 70d ago · 26×

execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)

execsvc's p95 is 4.2 s on master, above a normal of 380 ms ± 282 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 13.7, 4 weeks of history). automated changed execsvc 52 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Latency anomaly master Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 70d ago · 22×

enrollsvc p95 4.10 s vs a normal 312 ms ± 189 ms for Mon 16:00 UTC (3 of the last 4 checks)

enrollsvc's p95 is 4.1 s on master, above a normal of 312 ms ± 189 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 20.0, 4 weeks of history). automated changed enrollsvc 34 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Neo4j load anomaly master resolved last seen 2d ago · since 84d ago · 1035×

master (whole cluster) Neo4j load 942 q/min vs a normal 244 q/min ± 129 q/min for Mon 15:00 UTC (3 of the last 4 checks)

the whole master cluster's Neo4j load is 942 q/min on master, above a normal of 244 q/min ± 129 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 21d ago · 40×

enrollsvc Neo4j load 150 q/min vs a normal 0 q/min ± 1 q/min for Mon 14:00 UTC (3 of the last 4 checks)

enrollsvc's Neo4j load is 150 q/min on master, above a normal of 0 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 252.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Satish Ashok (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 21d ago · 4×

regsvc Neo4j load 173 q/min vs a normal 1 q/min ± 1 q/min for Mon 14:00 UTC (3 of the last 4 checks)

regsvc's Neo4j load is 173 q/min on master, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 290.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 2d ago · since 84d ago · 300×

docsvc Neo4j load 38 q/min vs a normal 2 q/min ± 3 q/min for Mon 14:00 UTC (3 of the last 4 checks)

docsvc's Neo4j load is 38 q/min on master, above a normal of 2 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 13.3, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

critical 4xx spike (403/401/404…) master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 31d ago · 38×

onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1

4xx spike (403/401/404…): onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1. Facts: endpoint GET /api/myprofile/visibility/v2, service onboardingsvc, calls 75, c4xx 36, share_pct 48.0, c401 36.

First look low confidence
Cause not clear from logs and recent changes; onboardingsvc GET /api/myprofile/visibility/v2 returned 401 on 36 of 75 calls (48%) in 15 min, 17x its usual 2.1.
Onset 12:30 UTC; the first 401 in the request log is 12:08 UTC.
Checked: 4 config-only autosync deploys (last at 08:03 UTC, 266 min before onset, no image or commit change), a hand edit of the deployment spec by [email protected] at 09:09 UTC, 0 stored error lines and no code found for myprofile/visibility. Nothing links any of these to the 401s.
feedsvc started returning 100% 403s at 12:06 UTC, and another onboardingsvc endpoint has 403s since 11:36 UTC. This finding has also recurred 5 times since Nov 5, which points to a shared auth or token problem rather than this endpoint.
Next step: open the 401 traces and the 4xx call list to see which caller and token is rejected, then check the auth path shared with feedsvc. · its evidence →

info Neo4j load anomaly master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 62d ago · 79×

onboardingsvc Neo4j load 48 q/min vs a normal 5 q/min ± 7 q/min for Mon 12:00 UTC (3 of the last 4 checks)

onboardingsvc's Neo4j load is 48 q/min on master, above a normal of 5 q/min ± 7 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.