Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-21 12:08:15.
› →|
updated 3d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Releases

What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.

running versionv3.9.21
commitdev
built—
sourcelocal checkout

This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.

The build you are running (v3.9.21) is on page 1.

v3.6.9 2026-09-24
Traffic pages read up to 50 GiB, and ask with a price beyond it

Added

  • Past 50 GiB, a view says how much it would read and what that likely costs — BigQuery's own estimate, from a free dry run of the exact query — and offers "Run anyway" for exactly that amount. One approval covers at most 1 TiB (about $6.25); anything larger asks for a narrower window.

Changed

  • A page may now read up to 50 GiB of logs (about $0.31) without asking, up from 1.6 GiB (a cent). 6h and 24h of API calls, errors, traces, consumers and jobs load again on every cluster — 24h of test26 is about 17.6 GiB (about $0.11).
v3.6.8 2026-09-24
API calls and errors load again for the last hour

Fixed

  • An hour of API calls or errors on master was refused as "more than 20 GiB" and showed nothing, with or without a service filter. BigQuery checks each read against its advance estimate, and the way filters were written made that estimate about four times the real scan (1.67 GiB for an hour that bills 0.03 GiB) — just over the one-cent page budget. Filters are now written so the estimate matches the scan (0.40 GiB for that hour), returning exactly the same rows.
  • When the page budget, not the per-query ceiling, is what stops a read, the message now says so and shows the budget, instead of quoting a 20 GiB limit that did not apply.
v3.6.7 2026-09-22
3.6.6 could not start on a database that already had the 3.6.3 table

Fixed

  • 3.6.6 crash-looped on startup and never served traffic. It widened cursor_usage_events and, in the same change, indexed one of the new columns from schema.sql. But init_db runs schema.sql first and the column migrations after it, so on a database that already had the table from 3.6.3 the index named a column that did not exist yet and the whole startup script failed with column "repo" does not exist. Web and ingester both crash-looped; the rolling update kept the previous pod serving, so the site stayed up and the rollout simply never completed. The index is now created after the migration, and a test upgrades a 3.6.3-shaped table to catch this class of change.
v3.6.6 2026-09-22
Cursor spend is real, and Bugbot turns out to be two thirds of it

Changed

  • The Cursor page now reports Cursor's own per-event charges. With a Team Admin key installed it reads 3,634 usage events for the last 30 days: $3,021.66 charged in total, of which $2,290.86 is usage-based (billed on top of the seats) and $730.80 is consumption the plan already covers. Those two are shown separately and never added into one headline, because only the first lands on an invoice.
  • Bugbot is the single largest line of Cursor spend: $1,992.56, about 66% of the total. It is the thing the old page claimed to measure at $86, so the figure it reported was roughly 23 times under. Spend is now attributed by model, by person, by bot, and — for Bugbot events, which name the PR they reviewed — by repository.
  • The page reads chargedCents from Cursor verbatim and never recomputes it. That value is usually tokens + Cursor's fee, but the identity failed on 36 of 3,630 measured events, and the number that has to reconcile with an invoice is Cursor's, not a formula of ours.

Fixed

  • X-ray classified a working Team Admin key as invalid. The credential probe asked /v0/me first and gave up when it failed — but Cursor's two APIs are mutually exclusive, not nested: a Team key is refused by /v0/me with "this is a team API key ... which only works with the Cursor Admin API", and a personal key is refused by every /teams route. The probe now asks the Admin API first and falls back to /v0/me only to name a personal key, so each key type is identified by the surface it actually belongs to.
  • The /teams/spend figure is no longer a headline tile. It reports hard overage beyond the plan's pools and read $0.00 for a cycle that carried $388 of usage-based charges, so presenting it as "current billing cycle" invited exactly the wrong conclusion. It is kept, below the fold, labelled for what it measures.
v3.6.5 2026-09-22
The Cursor page tells you where the Admin key actually is

Fixed

  • The instructions for enabling Cursor spend named a menu that does not exist. 3.6.3 shipped "Settings → Cursor Admin API", which was written from memory rather than checked, and an admin following it could not find the option. The verified path is cursor.com/dashboard → Team Settings → scroll to Advanced → Cursor Admin API Keys → New API Key, scope Admin — it is not a sidebar item, which is why it is easy to miss. The page also now says a Team plan is enough (only AI Code Tracking requires Enterprise), and gives a one-line curl to check a new key against /teams/spend before installing it — both key types start crsr_, so the prefix cannot tell an Admin key from a Background Agents key.
v3.6.4 2026-09-22
The Cursor collector actually collects the spend it was built to collect

Fixed

  • The Cursor collector cronjob synced reviews but not spend. 3.6.3 wired the new Admin API sync into the in-process loop that runs inside the ingester, and the job that exists precisely to do this work — x-ray-cursor-collector, twice daily — ran only the GitHub half. The effect would have been invisible and expensive: install a Team Admin key, wait, and the page still says spend is unavailable. The collector now runs both halves through one entry point, and a test pins the manifest to it.
v3.6.3 2026-09-22
The Cursor page asks Cursor what it cost, instead of guessing from GitHub

Changed

  • The Cursor spend figure was never Cursor's. It was computed as the number of PRs carrying a Cursor or Bugbot status check multiplied by a hardcoded $0.40 — a PR counter wearing a dollar sign. That is how the page came to report about $86 for a month that actually cost north of $2,000. Spend now comes from Cursor's Admin API: per-event usage with Cursor's own costs, plus the per-member billing-cycle total the Cursor dashboard shows. The two are reported side by side and never blended, because a window and a billing cycle are not the same period.
  • Three further numbers on that page were decoration, and are gone. "Spend Avoided" was $0.40 times the count of PRs whose branch name contained "cherry-pick"; no gate was skipping anything. "Unit Cost Efficiency" displayed the hardcoded constant itself. "Active Model — Composer 2.5 Fast, Bugbot (Smart Routing)" was two string literals that had never been read from Cursor, sitting under a connection status that returned "connected" without asking. The page now reports which Cursor credential it holds, whether that credential can read spend, and when it last succeeded.
  • When spend cannot be read, the page says so and shows no number. X-ray currently holds a personal Background Agents key, which authenticates fine for agents and is refused by every Admin API route — so spend reads as "—" with the reason and the fix, not as a zero or an estimate. A zero under a working key is shown as $0.00, because that one was measured.
  • The GitHub review ledger stays, as review activity. Which PRs Bugbot checked, what it concluded, and which cherry-picks were never reviewed are all real and still there. They are just not dollars: GitHub does not know what Cursor charged.

Fixed

  • Opening the Cursor page no longer runs a GitHub crawl inside the request. On a cold database it fired up to fourteen gh subprocesses at a 25-second timeout each while the reader waited, contradicting the page's own promise to read only from the database. The twice-daily collector owns backfill; the page renders what is stored.
v3.6.2 2026-09-22
Ask runs an investigation to its end, and asks before spending your money on a long look

Changed

  • The step cap is gone. Ask stopped mid-investigation three turns in a row with "I reached the step limit before finishing" — and not because the questions were hard: the cap was 8, and the agent had spent its steps retrying a tool that kept refusing. A turn now takes the steps it takes. What bounds it is money, not an arbitrary number of thoughts: every model call is admitted against the daily budget before it leaves. The 120-second turn deadline, which a real investigation exceeds on tool time alone (one argocd_in_sync call measured 19.4 s), is now 30 minutes.
  • A tool that could not afford the window used to hand back a dead end. errors may ask for up to 72 h, and 72 h of production logs is far past the read ceiling, so it failed with a refusal the agent could not act on — the same failure in three consecutive turns. Live reads now start at 6 h, may reach 24 h freely, and past that STOP and ask you first, saying what it may cost (BigQuery bills the whole window scanned, about 20 GiB — $0.12 — per day of the busiest cluster). Answer yes and it reads the longer window. The rule is injected into all nine live windowed tools, so a tool cannot forget it.
  • Ask is exempt from the per-page byte budget. A turn is an investigation across many tools, not a page view; holding it to one page's worth of bytes would have refused the reads the question needs. It stays bounded by the daily dollar budget instead.
v3.6.1 2026-09-20
A page load has one budget, so no view can quietly bill a dollar

Changed

  • One page load may now bill about a cent of BigQuery, in total, across every read it makes. The two ceilings that existed were per QUERY, which is not the number a bill cares about: a Logs view with every source selected makes seven to seventeen reads, so it could bill several times any single-query limit and did — 46 GiB, $0.28, in one measured load against a 20 GiB per-query ceiling. Each read is now admitted for only what is LEFT of the page's budget, which makes the total a ceiling rather than a hope, and a read that cannot fit is refused unbilled with a sentence saying to narrow the window or pick one source. Ingest and the detectors are not a page and keep their own ceiling; nobody is waiting on them.
  • What this costs you, stated plainly: an hour of the busiest cluster's logs is 1.2–1.95 GiB, which is roughly the whole budget, so a Logs view over a wide custom window or across all four sources will now come back partial and say why. That is the intended trade — it was the shape of load that produced every bill over a cent. OBS_BQ_PAGE_BUDGET_BYTES raises or removes the bar without a redeploy.
  • The per-request query ledger is cleared when the request ends. It never leaked in the server, where each request gets its own context, but anything reusing a context would have inherited a spent budget and had its reads refused.
v3.6.0 2026-09-20
A failed read says so instead of reporting zero, and the instrument wears Apple's design language

Changed

  • A panel that could not read its source no longer renders a confident zero. The owner stopped trusting the numbers after the cluster dashboard showed an empty Logs tile, Workload placement 0 and CPU 0m / 0m. Proven by running the same build against the same database with and without credentials: a total outage rendered as six measured zeros and the sentence "nothing moved". The Logs tile swallowed every failure and returned a hole, which made its own "the logs could not be read" branch unreachable for the only failure mode that occurs; the Memory tile coerced a missing reading to 0.0 GB while the CPU tile beside it, fed by the same failed read, correctly said —. Absence is not zero, which this page had been claiming it was.
  • Placement counts workloads again, not pods. A Deployment's ReplicaSet hash is six to ten characters but a Job's generated segment is five, so Job pods never matched the pattern and each became its own "workload". Production reported 2368 workloads where there are 28. Flipping 0 was arithmetically guaranteed rather than measured the whole time — a workload of one pod has one placement and can never be seen to flip.
  • A change the current scope says is not a finding can now close. Hand-change findings re-confirm out-of-scope subjects so they resolve, but only while the source row stays inside the detector's 24-hour window; once it aged out the finding was stranded open for good. Eight x-ray namespace findings were sitting open, last seen before the scoped build shipped.
  • Four detector descriptions said something the code does not do. The job-failure detector claimed error-severity lines over the last hour and actually reads sub-error lines carrying an error field over six hours — it described the bug that was fixed, not the fix. The disk forecast claimed to cover X-ray's own store and never touches it. The budget forecast credited Theil–Sen for a date that comes from a recent-median scenario. Pod instability promised closure fifteen minutes before the resolve grace allows it.
  • The whole instrument now uses apple.com's design language, in both themes — its palette, its type scale and tracking, pill controls and flat cards, with every uppercase label returned to sentence case. Apple's own greys sit below WCAG AA on these surfaces, so the dark palette is raised to the faintest values that still clear 4.5:1; measured in the browser across 33 pages, dark went from 43 unreadable pairs to none.

Versions are judged by effect on the reader, not by how much code moved: major when a page is gone, moved or now means something different; minor for a new page or a new answer on an existing one; patch for a fix to something the page already claimed to do. The same notes are in CHANGELOG.md, generated from the same source so the two cannot disagree.