Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 7 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 14:20:33.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 98detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 4 fleet-wide 12 critical · 14 warning · 13 info detectors
warning Autoscaler blind master resolved last seen 1d ago · since 26d ago · 571×

HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h

Autoscaler blind: HorizontalPodAutoscaler/docproc-hpa (tilda) cannot read its scaling metric — 2× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/docproc-hpa, failures_2h 2, last_message unable to get external metric tilda/custom.googleapis.com|docproc_pending_workload/nil: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric custom.googleapis.com/docproc_pending_workload: googleapi: Error 503: Authentication backend unavailable., backendError.

info Traffic anomaly master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 84d ago · 246×

docsvc request rate 10.2 req/min vs a normal 0.7 req/min ± 1.0 req/min for Mon 16:00 UTC (3 of the last 4 checks)

docsvc's request rate is 10.2 req/min on master, above a normal of 0.7 req/min ± 1.0 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 9.2, 4 weeks of history). automated changed docsvc 47 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Latency anomaly master Owner: Madhava Mukkamala (backup: Shashank Agrawal) resolved last seen 1d ago · since 70d ago · 26×

execsvc p95 4.25 s vs a normal 380 ms ± 282 ms for Mon 16:00 UTC (3 of the last 4 checks)

execsvc's p95 is 4.2 s on master, above a normal of 380 ms ± 282 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 13.7, 4 weeks of history). automated changed execsvc 52 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Latency anomaly master Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 70d ago · 22×

enrollsvc p95 4.10 s vs a normal 312 ms ± 189 ms for Mon 16:00 UTC (3 of the last 4 checks)

enrollsvc's p95 is 4.1 s on master, above a normal of 312 ms ± 189 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 20.0, 4 weeks of history). automated changed enrollsvc 34 minutes earlier — adjacency, not proof. Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.

info Neo4j load anomaly master resolved last seen 1d ago · since 84d ago · 1035×

master (whole cluster) Neo4j load 942 q/min vs a normal 244 q/min ± 129 q/min for Mon 15:00 UTC (3 of the last 4 checks)

the whole master cluster's Neo4j load is 942 q/min on master, above a normal of 244 q/min ± 129 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 5.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 20d ago · 40×

enrollsvc Neo4j load 150 q/min vs a normal 0 q/min ± 1 q/min for Mon 14:00 UTC (3 of the last 4 checks)

enrollsvc's Neo4j load is 150 q/min on master, above a normal of 0 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 252.9, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Satish Ashok (backup: Sambasiva Bandarupalli) resolved last seen 1d ago · since 21d ago · 4×

regsvc Neo4j load 173 q/min vs a normal 1 q/min ± 1 q/min for Mon 14:00 UTC (3 of the last 4 checks)

regsvc's Neo4j load is 173 q/min on master, above a normal of 1 q/min ± 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 290.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

info Neo4j load anomaly master Owner: Prasad Limaye (backup: Harish Kumar) resolved last seen 1d ago · since 84d ago · 300×

docsvc Neo4j load 38 q/min vs a normal 2 q/min ± 3 q/min for Mon 14:00 UTC (3 of the last 4 checks)

docsvc's Neo4j load is 38 q/min on master, above a normal of 2 q/min ± 3 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 13.3, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split.

critical 4xx spike (403/401/404…) master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 2d ago · since 31d ago · 38×

onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1

4xx spike (403/401/404…): onboardingsvc GET /api/myprofile/visibility/v2: permission denied — 36 of 75 calls (48 %; 401×36) in the last 15 min — 17× its usual 2.1. Facts: endpoint GET /api/myprofile/visibility/v2, service onboardingsvc, calls 75, c4xx 36, share_pct 48.0, c401 36.

First look low confidence
Cause not clear from logs and recent changes; onboardingsvc GET /api/myprofile/visibility/v2 returned 401 on 36 of 75 calls (48%) in 15 min, 17x its usual 2.1.
Onset 12:30 UTC; the first 401 in the request log is 12:08 UTC.
Checked: 4 config-only autosync deploys (last at 08:03 UTC, 266 min before onset, no image or commit change), a hand edit of the deployment spec by [email protected] at 09:09 UTC, 0 stored error lines and no code found for myprofile/visibility. Nothing links any of these to the 401s.
feedsvc started returning 100% 403s at 12:06 UTC, and another onboardingsvc endpoint has 403s since 11:36 UTC. This finding has also recurred 5 times since Nov 5, which points to a shared auth or token problem rather than this endpoint.
Next step: open the 401 traces and the 4xx call list to see which caller and token is rejected, then check the auth path shared with feedsvc. · its evidence →

critical 4xx spike (403/401/404…) master Owner: Prasad Limaye (backup: Sambasiva Bandarupalli) resolved last seen 3d ago · since 31d ago · 7×

onboardingsvc GET /api/myprofile/visibility/v2/study/:id: permission denied — 23 of 48 calls (48 %; 401×23) in the last 15 min — 73× its usual 0.3

4xx spike (403/401/404…): onboardingsvc GET /api/myprofile/visibility/v2/study/:id: permission denied — 23 of 48 calls (48 %; 401×23) in the last 15 min — 73× its usual 0.3. Facts: endpoint GET /api/myprofile/visibility/v2/study/:id, service onboardingsvc, calls 48, c4xx 23, share_pct 47.916666666666664, c401 23.

First look low confidence
Cause not clear from logs and recent changes; GET /api/myprofile/visibility/v2/study/:id returned 401 on 23 of 47 calls (49%) in 15 min, 73x its usual 0.3 per window, since about 2026-11-14 15:30 UTC.
No 403s and no stored onboardingsvc error lines; the sampled 401s carry no org, which suggests callers without valid auth rather than a permission bug.
The last deploy was config-only (autosync, f9914cf6181c, 06:33 UTC, 9 h before), as was a hand spec edit at 05:15 UTC, and no code changed.
Similar 401 spikes on /api/myprofile/v2, /visibility/v2 and /api/userprefs since late October resolved on their own.
Next step: open the 4xx calls list, see which caller sends these requests, and check whether its token is missing or expired. · its evidence →