Xray
hourly rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 48 hours▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 13:25:23.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Findings 10

stored age unknown all 33detectors over stored rollups and live sources

What X-ray is prepared to defend about master right now — each one a claim with its numbers, the evidence to check it, and what normal looks like. Detectors last ran never. ?A finding is opened by a detector — a statistical test over the stored rollups, or a live check of a source — and stays ONE row while the condition persists: it updates, escalates, and resolves itself fifteen minutes after the detector stops seeing it. Acknowledge to say you have it; snooze to hide it for a while; nothing here is a threshold somebody configured.

How to read this page — how findings work

Every card is a claim X-ray can back with numbers you can click through to. Findings rank by severity first, then by an impact score (severity band + the subject's share of the cluster's traffic + how long it has been open), so the top of the page is the thing to look at first.

anomaly
Three of the last four five-minute buckets far outside what this hour of THIS weekday normally sees — median ± MAD over the last 2–8 weeks, not a threshold.
outlier
A service unlike the rest of the fleet on one measure, over six hours. Compares with peers, not with its own history — the two catch different things.
budget
Burn rate against an error objective derived from the service's own last week, and the date the 30-day budget runs out if the trend holds.
resolves itself
A finding a healthy detector run no longer observes resolves after fifteen minutes. A detector whose SOURCE fails resolves nothing — it says so on the Detectors page instead.

What this page will not tell you

  • That a finding is a diagnosis. It is a measured departure from normal with adjacent changes listed; the narrative says 'adjacency, not proof' and means it.
  • Anything about hours the ingester did not cover. A bucket with no rows is a gap, not a quiet period, and the baselines skip it.
  • Anything on the first day for the novel-error detector: its 90-day memory learns silently for 24 hours before it will call anything new.
show active resolved all severity everything warning + critical only clusters
master
just master · nothing ticked means master, the cluster selected up top
or just close the list
group all AnomaliesReliabilityNeo4jChangesJourneyX-rayWeaviateAIPodsJobsPlatformErrorsCustomCostCapacity + 2 fleet-wide 12 critical · 7 warning · 13 info detectors
critical Pod cannot start master last seen 1d ago · since 1d ago · 14×

x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods

Pod cannot start: x-ray-argocd-collector (x-ray) image pull failing for at least 93 min — 2 pods. Facts: namespace x-ray, workload x-ray-argocd-collector, why image pull failing, pending_minutes 93, since 1794906422.

First look medium confidence
Pod x-ray-argocd-collector-29914927-d557w (namespace x-ray) cannot pull image us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/x-ray:3.7.1-56fc8bc: 51 events, Back-off pulling image, pending 35 min.
Pending since about 06:07 UTC; the finding opened at 06:29 UTC.
Likely cause: ArgoCD autosync of x-ray-services-master at 06:02 UTC, config revision 556a243c8ac9, about 5 min before the pod got stuck. It is the only x-ray change in the 24 h before.
The bundle does not show the image tag that sync set, so the link rests on timing and the matching app.
Next step: check that commit 556a243c8ac9 sets tag 3.7.1-56fc8bc and that this tag exists in Artifact Registry. If the tag is missing, push the image or revert the commit.
Suspect: x-ray-services-master sync 556a243c8ac9 (autosync) · its evidence →

feedsvc GET /studies/:id/ 3 targets worst: critical feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2, feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call, feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call
critical 4xx spike (403/401/404…) master last seen · since · 3×

feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/news: permission denied — 166 of 174 calls (95 %; 403×166) in the last 15 min — 1033× its usual 0.2. Facts: endpoint GET /studies/:id/feed/news, service feedsvc, calls 174, c4xx 166, share_pct 95.40229885057471, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/news returned 403 on 44 of 47 calls (94 %) in 15 min, against a usual 0.01 per window. Onset was 2026-11-17 05:20 UTC, and the first failing request in the log is at 05:27. Every sampled 403 comes from one org, "Teammates Demo Sandbox", and returns fast (13–21 ms). That points to a permission check on that org or its users · its evidence →

critical 4xx spike (403/401/404…) master resolved last seen · since · 57×

feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed/week: permission denied — 23 of 24 calls (96 %; 403×23) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed/week, service feedsvc, calls 24, c4xx 23, share_pct 95.83333333333333, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed/week returned 403 on 580 of 580 calls (100%) in 15 min, against a usual 2.3.
Onset 2026-11-16 07:10 UTC. This finding has recurred since 2026-11-12 18:03 (5 earlier episodes), and GET /studies/:id/feed is failing the same way (872 of 874 403s).
All 8 sampled 403s come from org "Tilda CRO" and return in 15-25 ms, so this is a fast permission rejection, not a crash. No feedsvc error lines are stored.
The only deploys were config-only autosyncs (81443681f910 at 01:17, c88737eb8000 at 22:28 the day before), 6+ hours before onset, with no commits. Nothing links them to the 403s.
Next step: open one failing trace to find which permission check rejects Tilda CRO. Then check that org's study access, or whether a client is hammering studies it cannot see. · its evidence →

critical 4xx spike (403/401/404…) master resolved last seen · since · 59×

feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call

4xx spike (403/401/404…): feedsvc GET /studies/:id/feed: permission denied — 108 of 108 calls (100 %; 403×108) in the last 15 min — it rejects nearly every call. Facts: endpoint GET /studies/:id/feed, service feedsvc, calls 108, c4xx 108, share_pct 100.0, c401 0.

First look low confidence
Cause not clear from logs and recent changes; feedsvc GET /studies/:id/feed returned 403 on 872 of 874 calls (99.8 %) in 15 min, 329x its usual 2.7.
Onset 07:10 UTC on 2026-11-16; the first 403 in the request log is at 07:31 UTC. GET /studies/:id/feed/week shows the same spike (580 of 580 are 403).
All 8 sampled 403s are fast (17-22 ms) and come from org "Tilda CRO", so a caller is hitting the feed many times without permission.
This is a recurring finding: 5 earlier episodes since 2026-11-12, all cleared on their own.
Checked: no feedsvc error lines stored, and the only 2 deploys were config-only autosyncs, 352 and 521 min before onset, with no link to the 403s.
Next step: open a trace and see which user or client in "Tilda CRO" is calling, then check its access to those studies. · its evidence →

critical Neo4j queries failing master last seen 1d ago · since 25d ago · 366×

Neo4j shared Neo4j, database automation: 37 Cypher queries failed in the last 15 min

In the last 15 minutes, 37 Cypher queries failed in the shared Neo4j database (automation) for project tilda-trial-dev. Environment: shared. Database: automation. Failed queries: 37. Time window: 15 minutes.

First look low confidence
Cause not clear from logs and recent changes; shared Neo4j database 'automation' (project tilda-trial-dev) had 37 failed Cypher queries in the last 15 min, 366 occurrences since 2026-10-23 19:21 UTC.
Checked: 12 autosync config deploys in the 24 h before onset (closest: execsvc 57 min and studiosvc 55 min before). None has an image tag, commit or error line tying it to Neo4j.
None of the 6 error shapes in the bundle comes from neo4j or names a Cypher query. execsvc 'no result from branch evaluation' first appeared 2026-11-13, weeks after onset.
No earlier episodes and no other findings opened at the same time.
Next step: open the Neo4j error logs for this window, read the Cypher error text, and find which client service sends the failing queries to 'automation'. · its evidence →

warning Background job failing master last seen 1d ago · since 25d ago · 107×

pubsub.dashboards-master-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationForDocument, id ?

The background job pubsub.dashboards-master-dashsub on master failed 5 runs recently (91 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

critical Background job failing master last seen 1d ago · since 25d ago · 545×

workplan.summarization-sweep: 30 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?

The background job workplan.summarization-sweep on master failed 30 runs recently (30 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

First look low confidence
workplan.summarization-sweep: 28 failed runs in the last 6 h — checkpoint scoring: unknown checkpoint source type ?.
Cause not clear from logs and recent changes; checked: error lines (none stored), deploys in the 24 h before (12), commits in the deploy's range (0), this finding's history, other findings at the same time (1). · its evidence →

warning Autoscaler blind master last seen 1d ago · since 26d ago · 3259×

ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h

Autoscaler blind: ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 35× in 2 h. Facts: namespace tilda, object ScaledJob/rag-agent-worker, failures_2h 35, last_message could not find stackdriver metric with query fetch pubsub_subscription | metric 'pubsub.googleapis.com/subscription/num_undelivered_messages' | filter (resource.project_id == 'tilda-trial-automation' && resource.subscription_id == 'tilda-document-master-rag-sub') | within 2m0s.

warning Background job failing master last seen 1d ago · since 71d ago · 326×

pubsub.training-assignment-master-trainingsub: 21 failed runs in the last 6 h — ReconcileTrainingAssignments failed for training ?: failed to fetch ActivityV2 ? for training ?: obj

The background job pubsub.training-assignment-master-trainingsub on master failed 21 runs recently (25 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.

warning 4xx spike (403/401/404…) master Owner: Suresh Boddapati (backup: Madhava Mukkamala) resolved last seen 1d ago · since 7d ago · 16×

studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1

4xx spike (403/401/404…): studiosvc GET /api/studio/protocols/:id/visits/:id: 4xx — 37 of 210 calls (18 %; other 4xx×37) in the last 15 min — 34× its usual 1.1. Facts: endpoint GET /api/studio/protocols/:id/visits/:id, service studiosvc, calls 210, c4xx 37, share_pct 17.61904761904762, c401 0.