Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-21 08:00:39.
› →|
updated 2d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 11detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity 6 critical · 5 warning · 8 info detectors
critical Pod cannot start master last seen 2d ago · since 3d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
Suspect: x-ray-services-master sync 556a243c8ac9 (autosync) · its evidence →

critical 4xx spike (403/401/404…) master last seen 2d ago · since 3d ago · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 174, c4xx 166, share_pct 95.40229885057471, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/news returned 403 on 44 of 47 calls (94 %) in 15 min, against a usual 0.01 per window. Onset was 2026-11-17 05:20 UTC, and the first failing request in the log is at 05:27. Every sampled 403 comes from one org, "Teammates Demo Sandbox", and returns fast (13–21 ms). That points to a permission check on that org or its users · its evidence →

critical Neo4j queries failing master last seen 2d ago · since 27d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j database 'automation' (project tilda-trial-dev) had 37 failed Cypher queries in the last 15 min, 366 occurrences since 2026-10-23 19:21 UTC.
Checked: 12 autosync config deploys in the 24 h before onset (closest: execsvc 57 min and studiosvc 55 min before). None has an image tag, commit or error line tying it to Neo4j.
None of the 6 error shapes in the bundle comes from neo4j or names a Cypher query. execsvc 'no result from branch evaluation' first appeared 2026-11-13, weeks after onset.
No earlier episodes and no other findings opened at the same time.
Next step: open the Neo4j error logs for this window, read the Cypher error text, and find which client service sends the failing queries to 'automation'. · its evidence →

warning Background job failing master last seen 2d ago · since 27d ago · 107×

pubsub.dashboards-master-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationForDocument, id ?

The background job pubsub.dashboards-master-dashsub on master failed 5 runs recently (91 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

critical Background job failing master last seen 2d ago · since 27d ago · 545×

workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?

The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

First look low confidence
workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?.
Cause not clear from logs and recent changes; checked: error lines (none stored), deploys in the 24 h before (12), commits in the deploy's range (0), this finding's history, other findings at the same time (1). · its evidence →

warning Autoscaler blind master last seen 2d ago · since 27d ago · 3259×

ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h

Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 35, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.

warning Background job failing master last seen 2d ago · since 73d ago · 326×

pubsub.training-assignment-master-trainingsub: 21 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj

The background job pubsub.training-assignment-master-trainingsub on master failed 21 runs recently (25 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

critical Neo4j queries failing master last seen 2d ago · since 15d ago · 213×

Neo4j shared Neo4j, database matrix_b: 35 Cypher queries failed in the last 15 min

In the last 15 minutes, 35 Cypher queries failed on the shared Neo4j database matrix_b. Environment: shared. Database: matrix_b. Failed queries: 35. Time window: 15 minutes. Project: tilda-trial-dev.

First look low confidence
Cause not clear from logs and recent changes; Neo4j shared, database matrix_b: 35 Cypher queries failed in the last 15 min (213 occurrences since onset).
Failing since 2026-11-04 14:19 UTC, still seen 2026-11-17 10:21 UTC; matrix_a also started failing 17 min later (7 queries).
Checked: 12 deploys in the 24 h before onset (11 config syncs at revision c8a04003ea3d by [email protected] 205 min before, dashrsvc autosync 67 min before). Nothing links them to Neo4j.
None of the stored cluster error shapes are Neo4j or Cypher errors, and no single service owns the queries.
Next step: open the Neo4j error logs for the window to get the failing Cypher error codes and the client issuing them. · its evidence →