Releases
What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.
This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.
The build you are running (v3.9.21) is on page 1.
Fixed
- Quality Control runs showed about 40% where the team reports 91-93% for the same runs (ENG-4826). A result file that declares its score (
headline_rate) now shows that score as its accuracy in Evals, Accuracy over time and on the run page; documents passed and scored are still listed beside it. QC declares the Critical/Major score: precision and recall on Critical and Major findings, and the overall score as the mean per-document F1. Every other eval is unchanged: documents right / documents scored.
Added
- Evals is one table of every run, one row per category it scored: date, category, model, dataset, accuracy (documents right / documents scored), the change against the previous full run of the same category, dataset and model, cost, time, service tier, and where it ran (Jenkins build, or a laptop and whose).
- Models on the same dataset: pick a dataset and see every model's latest full run on it, one bar per model per category (documents right / documents scored), best first — dataset3 puts seven models side by side, the QC set three. It opens on the dataset the most models have run.
- Each category and dataset in detail: per category and dataset, one line per model with its latest and best accuracy, runs, a trend, cost per run and per document, time and service tier — from full runs only. Under it, accuracy over time and accuracy against cost per run, with a line through the runs nothing beats on both.
- Ran, but never published results: runs that made LLM calls through the eval gateway and uploaded nothing (16 in the last 30 days, all from laptops), with who, what and what it cost.
- Run an eval (/evals/run): starts the tmf Jenkins job with a dataset, categories, model, documents per category and workers, as the shared Jenkins bot, naming you as the person who asked. Off until X-ray's account may read the bot's token.
- The evals MCP tool returns the same table, leaderboard and never-published runs, with days, category and model filters.
Changed
- Cost, time, service tier and the models that answered come from the eval gateway's usage log (LiteLLM on shared-neo4j, through its BigQuery link), split per category by each call's operation, so they no longer depend on the run's own calls.jsonl — which was empty for 20 of 21 tmf runs from 2026-09-09.
- Model, category and dataset come from each result file and the run's metadata, not from file names: Accuracy over time read them from one name pattern and showed 1 of 21 tmf runs since September. /evals/scores now opens Evals.
- A run where at least half the documents got no answer is marked broke, not scored 0%; a run that scored a LIMIT or a fraction of the dataset is a sample, listed but kept off the leaderboard.
- Evals has its own range (24h to 180d, 30 days by default) instead of X-ray's hour-long default, which showed the page empty on most visits.
- A run page with an empty ledger rebuilds it from the usage log, says so, and marks a cost that includes unpriced calls as a floor. Its time-window estimate reads the eval gateway, not the selected cluster's proxy.
Fixed
- A failed call (not billed) no longer counts as unpriced; only a successful call with no price does (gpt-6.1-terra, which LiteLLM has no rate card for).
- A web replica no longer sits Pending for good after a repository fetch times out: the clone step removes the lock files the killed git left behind (one replica was stuck for 20 hours on 2026-10-28).
Added
- Docs Studio (/docs-studio, BETA): Tilda Academy's docs (tildabio/whats-up-doc) as GitHub has them. Overview: what is waiting for you (reviews you are asked for or own via CODEOWNERS, your own pull requests and requests), what happened, and whether main is reaching Academy. Requests: every open request and pull request by stage, each with its checks, preview links, files, reviews and conversation. Guides, Screenshots (pictures a page asks for that aren't on main, and pictures nothing uses), Release notes (Academy's beside X-ray's), Publishing (the deploy runs and what merged since the last one that worked) and Access (who can merge what). Read with the Tilda Prs app; nothing is stored or made up.
- The docs agent: every ticket in Linear's Tilda Sense Documentation project that is in Ready for Development becomes a pull request on the docs repo. X-ray reads it every two minutes; if anything is unclear it asks on the ticket and waits for the answer, however long it takes, and once it is clear it starts a pod beside X-ray where Claude Code follows the docs repo's own skills, takes the screenshots from the environment the ticket names (production too, with a release named) and runs its checks. The pull request is opened as the person who asked when they have connected GitHub, otherwise by Tilda Prs, and linked on the ticket, which moves to In Review (or On Hold, with the reason).
- New request in Docs Studio makes that Linear ticket, in Ready for Development, with the guide, the environment (test26, test27, master, nexus, delta, production, automation) and the release; the Requests board shows each ticket with what the agent is doing. Nothing becomes a GitHub issue any more.
- Docs Studio › Screenshots shows every screenshot on main as a gallery, by guide or by words; each one has its page (the picture, where it's shown, its recipe and file, open requests about it) and Request a change, which files a ticket with all of that for the agent.
- The agent answers each ticket in one thread (its first comment starts it; a person's new thread takes over), in a line or two, and its pull requests say "Issues Fixed: ENG-…" so Linear links them.
- Every model call a docs ticket causes is tagged for LiteLLM's usage log: the triage and the agent's run in its pod (which reached it as system/xray and unnamed_anthropic_messages) name the operation (xray_docs_agent_triage, xray_docs_agent_run), the person who asked, the ticket as the session, and the job xray.docs_agent.
- Docs Studio › Release notes follows Academy's What's New, which mirrors the help center (help.tilda.bio): a release that is one page is listed, a build record (a folder guides name for their screenshots' app build) is not, and each release links to its Academy address (release-2-11.html).
- The docs agent can use X-ray's own Linear app (OAuth, client credentials) instead of a person's key: tickets, comments and moves are the app's, and a ticket someone asked for in Docs Studio, a screenshot or Academy reads "Their Name (via X-ray)" with their avatar, not as whoever owns the key.
- Academy's suggestions: on Academy's dev sites, select text or point at a screenshot to suggest a change; suggestions pile up across pages and file as one ticket (POST /docs-agent/suggestions, through Academy's own server, tilda.bio people only).
- Connect GitHub (Docs Studio › Access): let the Tilda Prs app act for you once, and what X-ray writes to GitHub carries your name. Tokens are encrypted, renewed while you use them, and withdrawn on GitHub when you disconnect.
Changed
- Ask X-ray's draft pull requests are opened as the person who approves them, when they have connected GitHub: their commit, their push, their pull request ("ramyala opened this", with the app's badge), which they can't approve themselves. Without a connection they come from the app's bot, as before; the approval card says which.
Fixed
- Every eval run published since tilda_gcs began writing items.jsonl and manifest.json said "This prefix holds another run's files" and set those two aside, Jenkins runs and laptop runs alike. They are the run's own report, like run.json and calls.jsonl, and are kept with it.
Added
- Each request on a run page opens in place to the app's side of it: the service, endpoint, status and user, every line the app logged under its trace and the Neo4j queries it ran (or ran after responding).
- A suite or a test has its own log in a run: the run page's suite links, the suite picker and each test row open /tests/run/<job>/<build>/logs?suite=…&test=… — the runner's lines, the app's lines and the Cypher of exactly those tests. A suite's history across runs is still one click away ("history").
Changed
- The Tests page lists every run in the selected window, newest first, with who ran it — all jobs, or one job's runs under its tab — instead of one row per job showing only its last run. Jobs with no run in the window are named in one line.
- The run log reads a suite or a test as its own question. It used to read the whole run's newest rows and filter them on the page, and a whole run fills every cap: on api-tests-master #568 two failing suites had no line at all, and EDLTracking had no runner line. A scoped read takes up to 5,000 rows a source; app and Neo4j lines start at INFO ("debug lines" reads the rest — DEBUG was 64% of one suite's app lines), and Cypher is off until ticked.
Fixed
- A run with failed tests reads UNSTABLE, not SUCCESS. The api-tests pipelines publish their report before the junit step marks the build, so every api-tests report said SUCCESS — #568 with 7 failures, #586 with 330. Where the runner's own log has the verdict it wins over the report's SUCCESS.
Added
- Ask and MCP read Cloudflare (tilda.bio, tilda.pizza, tildasense.ai) with X-ray's own read-only token — nothing to configure on a laptop. cf_overview, cf_firewall_events / cf_firewall_top (who was blocked, by which rule, from where), cf_traffic / cf_traffic_daily (requests by host, path, status, country, cache), cf_rules (WAF, rate-limit, transform and IP access rules), cf_dns_records (proxied or DNS-only), cf_dns_analytics, cf_zone_security (SSL mode, minimum TLS, certificates) and cf_history; cf_capabilities says what the Free plan can and cannot answer.
- A chat or run link with the wrong cluster (a test27 chat opened with ?cluster=master) says which cluster holds it and links there, and /ai/find and /ai/runs go there on their own, instead of reporting no rows.
- Twelve pages that no tool could read now have one, for Ask and MCP alike: llm_chats (the chat-agent conversation list), llm_calls (individual LLM calls, filterable like the AI Logs tab), ai_documents (LLM cost per document), pods and pod_detail (request/limit/usage per workload, restarts and why), jobs and job_detail (background jobs, their runs and spend), alerts and notifications (what is defined, what was actually sent — read as the caller, so a private alert stays private), neo4j_health and weaviate_health (the servers' CPU, memory, disk, restarts and internals, as last/min/max/mean) and node_incidents (spot preemptions and what they took down).
- Every tool can be paged. A tool without a page of its own takes
pagefor its result's longest list — 25 rows a page on MCP — and the result's_pagesays how many pages there are and what to ask next; before, MCP cut most results at 25 rows and pointed at a page=N that most tools did not accept. cf_firewall_top names a rule whose ruleset the read-only token cannot open (Bot Fight Mode) from the event's ruleset and phase instead of leaving it blank, and xray_catalog lists Jev's primary pack first. - xray_catalog: give it the question and Jev routes it to the few tools that answer it, so an MCP client need not read the whole catalogue to choose.
- The tools that can read the shared Neo4j/eval gateway cluster (logs, errors, ai_usage, reliability) list shared-neo4j-5-26 in their cluster choices, and a parameter that means something different in one tool (a cost tool's service is the service as billed) keeps its own wording in the MCP catalogue.
- AI traces read the same way everywhere: thread → runs → messages. Every message the model saw is one row — user query, assistant, tool call (paired with its result), injected context (labelled by tag and slot, e.g. system-reminder · current_date), system prompt, LLM call, side call — shown once, at the run and model call where it first reached the model, with a toggle per kind (remembered per browser) and presets for the conversation, + tools, or everything the model saw. The chat and run pages both use it. Stable links by id alone: /ai/chats/<thread id> and /ai/runs/<run id> find their own time; the Traces tab has a find-by-id box. Agents get the same list as JSON: the llm_thread and llm_run tools (MCP and Ask).
- A standing check that the LiteLLM log has not drifted from OpenAI's bill. The
openai_cost_driftdetector compares, every 6 hours, what the log says a cluster spent on OpenAI with what OpenAI billed its project over the last complete UTC days, in either direction, and raises "LLM cost off the provider's bill" past a tolerance (1% by default), or when OpenAI billed a model the log never saw. It refuses to judge on stale or partial OpenAI figures. Each day's pair is kept inopenai_reconciliation_dailyand never deleted, so a usage-based figure can be shown with the check that vouched for it. It checks the clusters listed in itsclusterssetting (test27 to start); an admin adds more without a deploy. - Machine identities. Each key of the optional
x-ray-service-tokensSecret is one service's API key, registered at startup as<key>@x-ray-- TildaBot publishes release notes astildabot@x-rayinstead of on a person's key, so a release it posts no longer says that person made it. Adding a service is a new Secret key; no code change. - Ask reads OpenAI's own bill.
openai_costsreturns what OpenAI charged per UTC day and per OpenAI project (project names equal cluster names), the org's dollars per model line item, and when the copy last synced.openai_vs_litellmsets one cluster's OpenAI bill against the LiteLLM proxy log day by day, with both cut to the same UTC days, the log restricted to OpenAI models and names merged (openai/gpt-5.4isgpt-5.4), so the answer is where the two part, not a list of possible reasons. - AI → Traces lists chats, one line per conversation thread (first question, user, turns, tokens, share read from the prompt cache, cost), not one line per question. Opening a chat reads it back and forth: each turn's question, the tools the agent called (folded, with arguments and results) and the answer, with that turn's calls, time, tokens and cost, and a link to the turn's own trace.
- AI → Traces: read a chat-agent conversation end to end. Each run lists its LLM calls (model, tokens, latency, cost, the tools offered, the response and the raw record) beside the conversation as the model saw it: the system prompt, every message, tool calls with their arguments and results, and the answers. History is folded so the current turn reads first. Context the agent injects as messages (<user_context>, <memories>, <system-reminder>) is shown folded and is never taken for the question. Runs are titled by what the person asked. The rows come from the LiteLLM callback's full-trace events, which sit in their own dataset (tilda_llm_traces_<cluster>_link) so they never add to what the usage pages scan.
- A request's trace page shows everything the request caused, in order. The waterfall has one lane per service (the request on top, every service it reached below), with API calls, LLM calls and Neo4j queries on it; clicking a bar opens its row. One chronological table lists each API call (endpoint, status, time) above the queries it ran, and the LLM calls where they happened; hover a lane for what that service did in total. LLM-trace log lines read as one line each and link to the conversation.
Fixed
- Injected context led by a label ("(current_date; replaces earlier current_date)" before a <system-reminder>) was shown as the user speaking; chat links no longer carry from/to twice.
- Eval spend shows the person who ran it. Eval runs logged before main#16621 recorded the runner only as an OS login in
extra_metadata.eval_userand left the user assystem/background, so their cost sat under "system identity" -- Akash's 7 days of evals on shared-neo4j showed as $0.21 instead of $18.03. The AI tab now reads those rows as the person's email. It is a one-off correction: only rows older than 2026-10-15 06:00 UTC, only a system user, only the seven logins it names; it removes itself from relevance once the logs age out (2027-04-13). - The OpenAI sync no longer stores its oldest day partial. The window opened at "now minus 45 days", mid-day, so OpenAI returned that first UTC day in part and the sync overwrote a complete day with it; it now opens at a UTC midnight.
- The OpenAI Spending page and Ask no longer lose most of a day's per-model dollars. OpenAI returns several results for one project, day and line item (one per API key or user), and the sync kept only the last: one day held $1.96 of its $111.02, and the per-model table covered about a third of the bill. They are now added, stored per project (so the per-model table follows the project the page is scoped to), and a step that fails marks the sync
partialinstead ofokand leaves the stored days as they were. - The OpenAI sync follows OpenAI's
next_pagefor every request and also stores OpenAI's own request and token counts per project, model and service tier, so the proxy log can be set against them. - Ask no longer treats the OpenAI and Cursor cost pages as the GCP bill. Asked from /costs/openai it was told "the GCP bill" and a 1 h window, so it searched the billing export for OpenAI charges that are never in it. It now knows the page is OpenAI's own billing data and which tools read it.
- When the per-model rows of OpenAI's bill add up to far less than the per-project totals,
openai_costssays the per-model breakdown is incomplete instead of letting it explain a project's total.
Added
- Release notes record where they start and end. The Releases table has Start and End columns ("—" when a release does not say); hover either to see the range per repository, with commit counts. A release's page shows "Start → End" under its title. The release-notes generator sends them with
start,endandrangesonPOST /api/v2/release-notesandPUT /api/v2/release-notes/by-key/{key}, and so can the New release form. A regenerated draft keeps its own range, and adopting the draft makes that the release's range.
Changed
- Approving a release now locks it. Its text, title, start and end cannot be changed — not from the page, the API, MCP or Ask — and it cannot be deleted, restored to an older version or have a draft adopted into it. The page shows it read-only, with "Approved vN by X — Unapprove to edit" and an Unapprove button; unapproving is recorded in the history. Comments, replies, resolving threads, new generated drafts, dismissing a draft and archiving all still work. A write to a locked release is refused with 403 (MCP and Ask say NOT SAVED and why). "Revoke approval" is now called Unapprove. Releases edited after approval before this change still say so.
Fixed
- The welcome card shown on first sign-in could not be closed on a laptop or phone: it grew taller than the screen inside an overlay that does not scroll, so "Got it" (its only button) sat below the bottom edge — 980 px down on a 768 px screen. The card now fits the screen and only its list of groups scrolls; "Got it" and a new × in the corner are always visible, and a click on the dimmed background closes it too (Esc still works).
Added
- Shared Neo4j (the shared-neo4j-5-26 cluster: the shared dev Neo4j, Weaviate and the eval gateway) is in the cluster picker on the pages where it means something — AI usage, logs and documents (the eval gateway's LLM calls), Neo4j and Weaviate health, logs, errors, pods, reliability and placement. It is view-only: nothing new is ingested or polled for it, and request pages say it has no app traffic.
- Shared Ask chats are collaborative. Anyone who opens a chat's share link can ask in it: their first question joins them, after which the chat is in their own list and needs no link. Every question is labelled with who asked it, and the model is told too, so "as Priya asked above" works. Everyone with the chat open sees new questions arrive and watches the answer stream in. Every turn is billed to the chat owner's daily Ask budget; the turn still runs as the person who asked it, so the audit case and any report are theirs. Only the owner, or the person who asked a turn, can re-run it. It reads as a group chat: each question carries its asker's avatar, name and colour, yours on the right and theirs on the left, and each answer says whom it is answering. Changes are pushed to everyone who has the chat open (about a second), rather than waiting for the next poll, and a colleague's question appears the moment they send it, with its answer streaming beneath it.
Fixed
- An image changed by hand is no longer a finding, in any namespace — rolling an image is how people ship and fix. It is still on the change trail. A hand change that carries no image (a scale, an HPA, a ConfigMap) in production's tilda or neo4j-system is still a finding; the open image-change findings resolve on the next pass. And a hand change now stops being a finding once it is older than the detector's window, as the window promised: one whose record had aged out was never re-checked and stayed open forever.
- Weaviate content (/weaviate-stats) works again. Production's nightly collector had been rejected since 2026-09-21 — first because its old image computed stale_docs the old way, then because the server accepted collector status only from master — freezing production at 2026-09-17; both are fixed and the collector re-run. The shared view led with an eight-day-stale test27 card (its collector was retired) and ~8,000 lines of dead org rows before master's; it now leads with the cluster you picked, lists snapshots no collector has refreshed in three days instead of rendering them, and renders an old-format snapshot instead of calling every figure unknown.
- Pages and the ingester read less BigQuery. Every narrow log read asked for the whole
labelsandresourcesubtrees, mostly GKE metadata nothing here reads; they are now read as the fifteen label keys and six resource fields the code uses (measured 35% less per narrow read, ingest included). The request lists — an endpoint's traces page, the traces heatmap, API calls — and the Weaviate event and indexing feeds read the full log row for a handful of fields; they now name them (the endpoint traces page billed ~17.7 GiB for a day of one service). X-ray usage now opens sorted by cost per load. - Asked "why did this take so long?" from a trace page, Ask was refused the trace as a 311 GiB (about $1.90) read, while the page itself had opened it for under a GiB. The Ask tool, MCP and the REST API searched the cluster's whole log retention in one read; the page searched the moment its link came from. There is now one lookup for all of them: the time the trace ran (a link's at=, a row's timestamp, or the page Ask was asked from) ±15 minutes; else X-ray's own record of recent traces; else the last day, a day at a time. Reading the whole retention is available, and asks first.
- The same trace no longer tells two stories: the page and the API split "waited for" from "after the response" Cypher at different thresholds (500 ms vs 1,500 ms); both now use one.
- A trace lookup reads its app log once: the request's summary is taken from its own "Request completed" line instead of a second read of the same window — about 40% less BigQuery per trace on every surface.
- Every question is now answered once, for every surface. The pages, the REST API, MCP and Ask had drifted: each called the same readers with its own caps, filters and thresholds, so the same question gave different answers at different cost. Now they share one layer (answers.py), and a test fails the build if any surface reads the data itself. Measured live: the error groups for an hour of master are the same SQL and the same 0.998 GiB whether asked from the page, the API or Ask.
- Ask and MCP asked for 'error' requests with a filter the reader did not know, so they got every request; the status filter now means the same everywhere.
- Ask could not read a week of FREE stored data (a 24-hour rule refused it, then rejected its own retry). Cost is now judged by bytes, exactly as a page judges it: under the read budget it runs; over it, every surface refuses with BigQuery's price — the page offers Run anyway, the API answers 413 with the exact approval, Ask and MCP are told to ask the person.
- The API and Ask log search now matches the Logs page: stdout-only services (rag-agent, litellm) are reachable, Neo4j and Weaviate lines are included, text matches anywhere, and noise events no longer use up the line cap.
- Cost anomalies use the cost-anomaly detector's own thresholds on the page, in Ask and in the findings (they used three), and Ask now folds in project-level spikes under a cluster filter as the page does. Outliers everywhere use the outlier detector's settings.
- The API's attribution and pods answers now sample the last 10 minutes as the pages do (the attribution read over 6 h cost up to 36x more); a session opened by id is read in its own minutes on every surface when its times are known.
- Cost comparisons in Ask flag a previous period that is only partly in the ledger instead of reporting it as a change, and cost search and commitments honour their filters.
- Logs are read from BigQuery for as long as BigQuery holds them — its measured reach (21.9 days on automation and dev, 24.4 on production today, rising to its 30-day expiry) — and only the part of a window older than that goes to the GCS archive. The API and Ask now fall back to the archive past that point as the Logs page does, and say where the line was; the last helpers still assuming the Logging API's old 3 days were changed.
Added
- Ask for a report and Ask X-ray composes a branded PDF: a summary, KPI tiles, tables, charts (bar, stacked bar, line, area, stacked area, donut, scatter) and live network diagrams. Every chart, table and computed figure is read by X-ray from the tool results of that turn — every row the tool returned, not the shortened copy the model sees — and each one names its source. A closing "How this report was built" section lists the sources, any block that could not be drawn, and every number in the text no tool returned (highlighted where it appears). Open it in the browser or download the PDF.
- gcp_network: a project's VPCs, peerings, subnets, firewall rules (flagging the ones open to 0.0.0.0/0), NAT, load balancers, addresses, VPN, GKE and Cloud SQL, read live. A network_diagram block draws it: what faces the internet, the open firewall rules, and one card per region that holds something.
- scc_findings: active Security Command Center findings per project, with exact counts by severity, category and class, and for GKE vulnerabilities the workload, image, package, installed and fixed versions, CVE and CVSS.
- gcp_assets: any resource in a project via Cloud Asset Inventory — counts by type, or a filtered list.
- linear_issues: list and count Linear issues by team, state, label, assignee, project and recency, up to 500.
- Three one-click skills: network diagram report, security findings report, monthly cost report.
- Reports wear the Tilda brand, read from tilda.bio: the tilda research wordmark, a cream page, serif headlines, Inter text, mono labels in outlined pills, a deep-teal KPI band with mint figures, and teal-and-sand charts and diagrams. The fonts (Inter, Source Serif 4, IBM Plex Mono — all open-licence) are bundled, so every PDF looks the same wherever it is made. Brands are a table, ready for more than one.
- Network diagrams draw a Shared VPC a project's clusters sit on (automation's shared-network) as its own marked box, and bundle many entry points into one edge per region instead of a tangle.
Changed
- Tools may now return up to 500 rows when asked (default stays 50), and a result may be up to 256 KB, up from 64 KB. Earlier tool results in a turn are now prompt-cached, so each is billed once at the cache-write price and then at a tenth of the input price on every later step, instead of full price every step.
- Ask X-ray's daily budget is $10 per person, up from $5 — a PDF report costs about $0.30 to $1.60. The X-ray-wide $50 day is unchanged.
Fixed
- A chat's PDF (/ask/chat/<id>/pdf) could be downloaded by anyone signed in who guessed its number; it is now served only to the chat's owner or a holder of its share link, like the chat itself.
Versions are judged by effect on the reader, not by how much code moved:
major when a page is gone, moved or now means something different;
minor for a new page or a new answer on an existing one; patch
for a fix to something the page already claimed to do. The same notes are in
CHANGELOG.md, generated from the same source so the two cannot
disagree.