Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-21 16:57:59.
› →|
updated 3d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Releases

What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.

running versionv3.9.21
commitdev
built—
sourcelocal checkout

This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.

The build you are running (v3.9.21) is on page 1.

v3.5.22 2026-09-16
The Neo4j ingest read names its fields, and stops retrying a pool that keeps overflowing

Changed

  • The five-minute Neo4j pass reads the twenty-one fields it uses instead of whole rows (its own projection; the trace page's Neo4j read already had one). BigQuery bills the JSON column per path referenced; proven on 15,591 live entries to return identical values for every field the rollups, shapes and slow-query details consume, for 30% fewer bytes.
  • A project whose Neo4j firehose overflows the pooled read (dev) is read per cluster for an hour before pooling is tried again, instead of paying the wasted pooled read every cycle on top.
  • The two log-reading detectors fetch only what is new. The job-failure detector re-read its whole six-hour window every hour, and the novel-error detector a full hour every fifteen minutes, per cluster (0.7–0.9 GiB a read, measured — about 10 GiB an hour in one project). Each now reads once per project, only the minutes since its last run, keeps the lines in its own table for the window it judges, and evaluates from there. Same windows, same findings, a tenth of the bytes.
  • A pooled read hands each cluster only the rows inside its own window. A sibling that started seconds later could align its window one bucket later and receive the earlier bucket's rows too, colliding with rollup rows it had not cleared (a duplicate-key error on the start-up cycle).
v3.5.21 2026-09-16
Audit logs are read through the Logging API, for free — the hourly ArgoCD collector stops scanning BigQuery

Fixed

  • Every reader of a Cloud Audit log now asks the Logging API instead of BigQuery. Admin Activity audit logs sit in each project's _Required bucket, kept 400 days and free to read; the sink also copies them into the log table, where a read bills the whole time window like any other log. The hourly ArgoCD change-trail collector (still running the 3.5.9 image) was reading its two-hour audit windows there: ten queries of 0.9–1.6 GiB per project every hour, plus a whole-table probe per project — about 30 GiB an hour fleet-wide, the largest item left on the bill after 3.5.19 and measured at 250–470 GiB a day in one project alone. The same filters, the same rows, no scan; the spot-pool lookup on the Pods and Placement tabs (a compute audit read) rides the same path.
The ArgoCD collector cronjob moves to this image with the web and ingester tiers; it had been left on 3.5.9, so it also inherits 3.5.18's partition probe and shorter publish slack.
v3.5.20 2026-09-16
Three small tightenings measured right after 3.5.19 went live

Fixed

  • The pod-events read is now really shared between sibling clusters. 3.5.19 pooled it, but the pool was keyed on the exact second each cluster asked, so three clusters collected seconds apart still read the same window three times (measured on the first cycle). It is keyed on the window's start bucket now.
  • Each ingest tick reads eight minutes of logs instead of twelve. The cycle runs every five; the measured publish lag is under a minute at the 99.9th percentile, and the current partial bucket is rewritten by the next tick anyway. A third off every routine read.
  • The spot-pool lookup reads the last fifteen minutes of the audit log, always. Learned pools are persisted and a spot pool inserts nodes continuously, so the scan back to the previous watermark (663 MiB per Pods-tab view after 3.5.19) bought nothing.
  • The job-failure detector reads each project's six-hour window once an hour, not once per cluster (seven 550 MiB reads an hour became three), splitting the lines by cluster afterwards.
  • The pooled Neo4j read has twice the row headroom before it is declared truncated — dev's firehose passed the old cap in one window, so the pool was thrown away and both clusters read again (three reads for two clusters).
v3.5.19 2026-09-16
Page loads stop scanning whole months of logs — the trace, pods, placement, logs and usage views are bounded, and the scheduler reads each window once per project

Changed

  • A trace is looked for where it ran, not across the whole retention. Every link to a trace now carries the moment it came from, and the trace page searches fifteen minutes either side of it (otherwise the selected time range). The old page ran two 30-day scans per view — about 90 GiB, measured — to find one request. 'Search all 30 days' is still there, explicitly, and walks the retention a day at a time, stopping when found. The Logs tab's trace search follows the same rule instead of a blanket two days.
  • The Pods and Placement tabs learn where a pod runs from Cloud Monitoring, not from a month of scheduling events. Each view of the Pods tab walked every Scheduled event in the cluster's log retention (about 110 GiB, the single most expensive page); the placement view did the same for long-lived pods. Monitoring already labels every pod's CPU samples with its node, for free.
  • X-ray's own usage page reads the page meter instead of re-scanning its access log (32–48 GiB a view), and now offers 7- and 30-day ranges.
  • Every BigQuery log read has a ceiling. A page may bill at most 20 GiB per read (about a day of the busiest cluster), the scheduler 12 GiB; a read that would exceed it is killed unbilled and the page says so in words, with what to narrow. Before this a single click could scan 220 GiB.
  • The scheduler reads each project's window once. The Neo4j pass and the pod-events pass read the same twelve minutes once per project and split the rows by cluster, as the API pass already did — three sibling clusters were paying three times for the same bytes. The first cycle after a restart reads back only as far as the gap (an hour at most), pooled too, instead of seven separate hour-long reads.
  • The cost mirror stops re-pulling five weeks of billing export on every start. A start pulls the recent two days; the nightly re-pull reaches a week, and the full 35-day adjustment horizon runs weekly (Sunday) instead of every night.
  • The journey's session view and the placement scan name the fields they read instead of pulling whole rows (BigQuery bills the JSON column per path referenced), and the Logs tab's filter sampler reads an hour every fifteen minutes instead of six hours every three.
Nothing stored changes and no cluster is left out: the same seven clusters are ingested every five minutes, the same pages render the same facts. What changes is how much of the log table each question is allowed to touch. Follows 3.5.18, which fixed the scheduler's hourly month-long scans; together they address the $690 September BigQuery bill measured from the job history.
v3.5.18 2026-09-15
The scheduler stops scanning a month of logs every hour — BigQuery cost cut where it was actually spent

Fixed

  • The disk forecast no longer scans a month of logs every hour. The hourly detector asked the Neo4j and Weaviate tabs' data gatherers for 30 days of volume metrics and got the tabs' log-derived feeds (disruption events, app errors) with them — a scan of the entire log table of each environment (210 GiB per query on the shared dev cluster), 24 times a day, for feeds the detector never read. It now asks for the Monitoring series only. Measured from BigQuery's job history, this was the single largest item on the September bill (about $230 of the dev project's $325 in the first half of the month).
  • Readers of past windows stop padding the window's end with an hour of slack. Entries may publish later than their timestamp, so every query extended its publish_time upper bound by an hour and paid for that hour of logs; the pipeline's measured lag is 3 s median, 46 s at the 99.9th percentile. The slack is ten minutes now. (A live ingest tick ends at 'now' and never paid this; log pages, gap repair and detectors reading a past hour did.)
  • The job-failure detector runs hourly instead of every 15 minutes. It looks back six hours each time; four scans an hour of the same six hours per cluster bought nothing.
  • The coverage probe reads partition metadata instead of the whole table. The hourly check of how far back each log table reaches took the minimum publish_time over the entire table (0.8–1.4 GiB per project); it now asks the partition list for the earliest populated partition and takes the minimum inside that one partition (16–36 MiB).
Nothing changes in what is stored or shown: the same windows are ingested, the same findings are raised, the same pages render; only the bytes BigQuery is asked to scan change. For the record, the five-minute ingest itself was measured cheap (the pooled API pass bills 30–70 MiB per tick; the Neo4j pass 90–150 MiB per cluster per tick). What remains after this release is mostly page loads that scan the whole retention — trace lookups, the pods and placement fragments — which live in the web tier and are the next fix. 3.5.16 and 3.5.17 are the remediation branch's numbers and were not kept in production.
v3.5.17 2026-09-15
3.5.16 stalled ingestion on PostgreSQL deployments — the coverage probe and the compaction planner are fixed

Fixed

  • Ingestion, pod health and the live/archive wall work again. 3.5.16's BigQuery coverage probe took the minimum publish_time over the whole entries table — 0.8–1.4 GiB per project, not the few megabytes it assumed — under a 64 MiB ceiling, so every probe failed closed and, with it, live ingest, pod-health collection and the wall between live and archived logs. The probe now reads partition metadata for the earliest populated partition and takes the minimum over that one partition (16–35 MiB today, 256 MiB ceiling, cached an hour). On master the 3.5.16 ingester was rolled back to 3.5.15 within twelve minutes; no window was lost.
  • Hourly and daily compaction and retention run on PostgreSQL again. The compaction planner grouped by a bucket expression with separately bound parameters, which PostgreSQL cannot match to the selected column (column "c.bucket_start" must appear in the GROUP BY clause); SQLite accepted it. It groups by the output column now, with a real-PostgreSQL regression.
Both faults were invisible to the offline suite: BigQuery is mocked, and the PostgreSQL module never called the planner. Both now have regressions; the probe's byte cost was measured with free dry runs against the four real tables before shipping.
v3.5.16 2026-09-15
The remediation release — a number is right or says unknown, an alert is delivered once, AI spend is one shared budget

Changed

  • AI spend is one budget for all of X-ray. Every model call — Ask, alert composer, widgets, wording, narratives, optimizer — reserves against a shared, durable daily ledger before it is sent, so two replicas cannot both spend the last dollar; unsettled attempts hold their amount for the day.
  • Ask is exactly-once per turn and approval is exactly what you saw. A reconnect replays the turn instead of running it again; a PR or Linear action executes only the full server-stored preview you approved, for the same target and base, and only once.
  • Collectors report attempts, and the Ops page shows them per unit. Argo coverage is certified by completed attempts (legacy ranges are labelled unverified, not bridged), Weaviate identity math shows unknown where it cannot be reconstructed, and Skillgraph snapshots live in the shared database.
  • Log archive reads are bounded and honest: bounded scans, correct substring search, de-duplicated shards, and late-arriving shards are revalidated instead of silently ignored.

Fixed

  • A gap in the data reads as unknown, never as zero traffic. Stored metrics are now built from complete, aligned five-minute windows with a coverage certificate per window; hourly and daily tiers only compact certified children, retention only prunes what a clean parent covers, and every page that used to show 0 for a missing window now says what it does not know.
  • Cost pages stop inventing. Missing billing export, an unknown commitment utilisation or an unallocated share render as unknown rather than $0 or “within target”; recommendation observations are no longer summed as if they were measured savings; evaluation effort is counted once per run, not once per artifact.
  • One alert, one message. Every destination of a send is its own durable job: a failed webhook retries on its own without resending the ones that worked, a retry re-checks mute, disable, quiet hours and the destination list, a warning held overnight is consumed by exactly one lane in the morning, and the hourly cap counts messages, not rows. Delivery is at-least-once, never twice from our side.
  • A detector that could not see a source no longer resolves the findings it could not see. Partial or failed sources keep their findings open, a failing detector cannot poison its siblings on PostgreSQL, and a finding keeps its identity when an alert is renamed.
  • AI wording actually happens. With a model configured, the polished narrative on a new finding and the plain-English notification text were silently skipped; both are written now, after the facts are committed, and never overwrite newer facts.
  • /healthz works on PostgreSQL. It counted detector errors with SQL PostgreSQL rejects, so the probe reported a database error on every Postgres deployment.
First start adds tables and columns (collector state, coverage certificates, the AI ledger, Ask events) and back-fills finding identities where the facts allow; nothing is dropped. History that was already wrong is not rewritten — legacy totals and old coverage stay labelled unverified until an explicitly scoped repair. The finding-by-finding record is docs/REMEDIATION.md.
v3.5.15 2026-09-08
Log-alert notifications group into one thread per subject — the bell stops showing 975 identical lines

Fixed

  • A 'notify me on any error' alert no longer floods the bell with a row per log line. Those notifications carry no finding, and the inbox threaded them one-per-row — so one alert showed as 975 separate entries. They now fold into ONE thread per alert and subject (e.g. 'Error in docproc' with a running count), and the bell counts threads, not lines. Existing notifications regroup on their own — nothing to clear.
  • A recurring error on many documents is one notification, not hundreds. The live log-alert de-dup now normalises file paths and names out of the error 'shape', so the same failure across different files collapses at the source instead of one message per document.
v3.5.14 2026-09-08
Error-rate anomalies no longer fire on 404s — a 4xx is the caller's problem, a 500 is ours

Added

  • A new informational finding, "Endpoint rejecting calls". When an endpoint rejects almost every call with a 4xx (a broken client or a changed contract — not a server error), it now shows as an *info* signal instead of vanishing into the noise or masquerading as a critical error. So a genuinely-wrong caller stays visible without paging anyone.

Changed

  • The error-rate anomaly (anomaly.errors) now watches SERVER errors (5xx), not 4xx. A burst of 404s — a scanner, a missing resource, a client calling a removed endpoint — used to trip a *critical* "error rate 11.67%" finding with zero server errors, because the detector counted 4xx + 5xx. Every other part of X-ray already excluded 4xx (error budgets, outliers, endpoint-failing all say "a 404 is the caller's problem, a 500 is ours"); this detector now agrees. The error-rate CHART still shows the full 4xx + 5xx rate — only the alert changed.
Existing anomaly.errors findings driven by 404s will resolve on their own once the 5xx baseline is in place (rebuilt from stored history, no warm-up).
v3.5.13 2026-09-07
Ask X-ray can read code and open PRs in production — the tools that were local-only are provisioned on the pod

Added

  • Ask X-ray's code and act packs now work on the deployed instance. They were built and only ever ran locally, so on the pod the chat truthfully said it had no way to read source or open a PR — the packs hide themselves when their prerequisites are absent. Now the image carries git and gh, an init container clones the platform repo onto the pod, the GitHub App key and the Linear key are mounted from Secret Manager, and the repos are pointed at. So the chat can search and read code, and PROPOSE a draft GitHub PR or a Linear ticket — every write still gated on your approval in the turn, opened as tildabot[bot].
Secrets come from GCP Secret Manager via External Secrets (ash-nazg): a read-only Cloudflare token, the Linear key, and the existing GitHub App key. Nothing long-lived is baked into the image; the App mints a short-lived installation token per PR.

Versions are judged by effect on the reader, not by how much code moved: major when a page is gone, moved or now means something different; minor for a new page or a new answer on an existing one; patch for a fix to something the page already claimed to do. The same notes are in CHANGELOG.md, generated from the same source so the two cannot disagree.