Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-23 06:11:52.
› →|
updated 4d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Releases

What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.

running versionv3.9.21
commitdev
built—
sourcelocal checkout

This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.

The build you are running (v3.9.21) is on page 1.

v3.2.3 2026-08-28
Spikes under a second are noise

Changed

  • Latency findings need a second. A query shape whose p95 went from 2 ms to 43 ms is 21× and means nothing to anyone; 31 of the 34 *query shape degrading* findings ever opened were like it. The shape detector and the p95 anomaly detector now ignore anything whose p95 is still under 1 s — the owner's rule, written on the System alerts page as a setting (*ignore while the late p95 is under · 1000 ms*, adjustable), in each detector's description, in each finding's narrative, and here. Above a second the ratio and the deviation decide, as before.

Fixed

  • Tables in Ask X-ray rendered as pipe soup some of the time. A table was recognised only when it stood alone in its paragraph, one row per line; the model sometimes glued it to the sentence before it, or put every row on one line, and the pipes came through as text. The renderer now reads a table wherever its |---| separator row is: the separator fixes the column count, the cells before it are the header, the cells after are taken a row at a time, and the prose around it becomes its own paragraphs. The prompt also asks for a blank line around every table.
v3.2.2 2026-08-28
A message has one number, and says what it said

Added

  • Message numbers. Every send carries a number — *X-ray message #131* — printed on the last line of the Chat card (linked), on the bell row and the Notifications page, in the webhook body, and on the alert page, where each entry is that number. Every channel's copy of one send shares it, and /alerts/messages/131 lands on the alert page scrolled to that message, highlighted, with a line saying you came from it. A reader holding a Chat message can now find exactly that message in X-ray, and what it was about.
  • What it has told you about. The alert page has one line per finding across the alert's messages: every message that mentioned it, oldest first, with the severity *as sent* and the message's number — *opened warning 15:28 #101 → still firing 15:45 #102 → escalated critical 16:21 #103* — then where the finding stands now: still open at what severity, or resolved when, and whether that was ever told.

Changed

  • The alert page and the Notifications page read a message from the payload frozen when it was sent — headline, severity, the findings as they were — and show *→ now critical* or *→ now resolved 16:28* beside a finding that has moved on. Before, a message about one finding showed the finding's *current* title, so two messages fifteen minutes apart about the same finding read as two unrelated problems. A single-finding send stores its summary like a group's does; rows from before read their headline off the card.
  • One entry per send: a message that went to two channels is one entry with both deliveries beside it (*#dev-notifications sent · #in-app delivered*), on the alert page and on the Notifications page. One line per thread in your notifications: a grouped alert's running summary is one line with its earlier updates folded, marked read together; the bell counts threads. The 16 became 3.

Fixed

  • A finding's resolution was announced only if a message happened to go out in the same tick it resolved; otherwise it was dropped, and the next message — an hour later, after the group's back-off — said nothing about it (#248: told at 10:51, resolved 10:58, silence at 11:53). What the last message called firing stays owed until a message closes it.
v3.2.1 2026-08-28
A pod that keeps dying is a finding, not a page

Added

  • Pod keeps restarting. execsvc on delta crash-looped for a day — seventy restarts, ninety BackOff events — and nothing opened, because every detector measured what was happening and a crashing pod is an absence: it serves nothing, logs no 5xx, burns no budget. Now the ingest cycle also pulls each cluster's container restarts (the Monitoring counter as five-minute deltas — in-place restarts, so a rollout or a spot preemption is churn, not a crash) and the Kubernetes events that say why, into a rollup kept thirty days. The pod_unstable system alert opens a warning at three restarts in thirty minutes (or one OOM kill), goes critical once the run has lasted an hour and is still going, resolves after thirty quiet minutes, and says in words what it knows: *execsvc on delta has restarted 108 times in 9 h — crash loop, image v2.10.13-1fe888e*, with the last event verbatim, the containers involved, the deploys just before, and its share of the cluster's traffic for the impact. A stale rollup is refused, not read as calm.
  • Service gone silent. The other half of absence: a service whose hour-of-week baseline says it should be serving and that has served nothing for thirty minutes opens a finding, critical after two hours, with how long it has been quiet and what it normally serves at this hour. Only where the baseline expects enough calls to judge; an ingest gap is never read as a quiet period.

Fixed

  • Costs: a day is complete when the ledger says so, not the clock. At 07:40 UTC the export still held 21 of yesterday's 24 hours, trailing off to $0, and the anomaly page read the missing hours as a cost drop. 'Complete through' is now measured from the hourly ledger (all 24 hours present, none trailing off), and every page, tool and detector derives its window from it.
  • Costs: no comparison against a hole. A previous period only partly in the ledger is not compared with — an 8× 'increase' was nine days against thirty; the pages say how many days are present instead, and the month tiles say when last month is not fully loaded yet.
  • Costs: day-grain charts label UTC calendar days (a day epoch in the viewer's zone read '05:30'); the hour-range picker is hidden on usage-day pages; the footer names the ledger; the anomaly page defaults to z ≥ 4 and moves of $20+ with its own controls; the explain panel reads X-ray's own signals from whichever rollup tier holds the day.
Both thresholds are editable under Alerts → System like every other detector; the pod-health pull is one Monitoring call and one paced Logging read per cluster per cycle, and the ingest-stale alert watches its heartbeat. Nothing deployed.
v3.2.0 2026-08-28
The bill, with every penny tied to the thing that incurred it

Added

  • Costs, from the billing export itself. A new Costs group reads the billing account's BigQuery export (all twelve projects) into a ledger X-ray keeps: one row per usage day × project × service × SKU × region × cluster × namespace × workload, plus an hourly table for the last three weeks and a per-resource table (this disk, that bucket, this log bucket) for the last five. Refreshed every six hours, re-read nightly for late adjustments, partition-pruned and byte-capped; the page footer says when it was refreshed and what refreshing it cost in BigQuery bytes (about three dollars a month).
  • Overview: this month so far and the month-end projection (Theil–Sen over the last 28 complete days, with a band), the period two ways (service, project) and per cluster, what moved against the previous period by service and by workload, the days that were flagged, and what the recommendations are worth. Net = billed at the effective price + credits — the number on the invoice — with gross and credits always beside it.
  • Explore: break the bill down by anything — project, service, SKU, region, cluster, namespace, workload, workload type, node pool, cost type, invoice month, or the named resource — and drill by clicking: a row becomes a filter chip and the remainder regroups by the natural next dimension. Every row carries its share, the previous period's value, the delta and a trend. Rows with no value in a dimension are a named bucket ('not cluster-attributed (project-level)'), never dropped.
  • Commitments, read the way the fee-based model actually bills: usage is charged at the discounted price; the part of a commitment fee that funded usage is attributed to a project or cluster and nets to zero against its offset credit; the unattributed remainder is what the commitment cost for nothing. Each family shows utilisation, what it funded by cluster and project and subscription, fee vs funded per day, and dollars per month that funded nothing. On this account today: the three GKE commitments are fully used, the Compute Flexible one is 83 % used in August, and the Cloud SQL commitment (about $90 a month) has funded nothing since it began.
  • AI on the bill: Gemini, Vision and Translate by model (parsed from the SKU) and token kind (input, output, cached input, image input), the cached share per model, and a reconciliation against X-ray's own AI log — the proxy's price-table estimate beside what Google charged, per project — with the billed Gemini line apportioned to orgs by their share of logged Gemini spend, labelled as such.
  • Anomalies: days a project × service × cluster cell sat far outside its own trailing normal (robust z on median/MAD over 28 days, with a dollar floor). Open one and the day is decomposed — which SKU, namespace, workload and project moved and what share of the delta each explains — beside what X-ray saw on the same cluster that day: API calls, Neo4j queries, 5xx, deploys, findings opened. Adjacent facts, not causes; *Ask X-ray why* opens the dock with the question already scoped.
  • Forecast & targets: month-end and 90-day projections for the total, each project and each cluster, and monthly targets you set on the page (the Budget API is not enabled on the account) — with the day the projection crosses each.
  • Recommendations, by source. *Google says*: the Recommender API (idle disks, over- and under-provisioned GKE workloads, PDB and webhook diagnosis) with Google's own monthly saving. *X-ray found*: rules over the ledger — a commitment funding nothing, an under-used one, workloads growing 40 % fortnight on fortnight, Cloud Logging's share, Gemini input never cached, a cluster billed with no traffic — each with the numbers it was computed from and its estimate marked as one. *A model suggested*: on request, through Ask X-ray, never stored as fact. Done and dismissed live in X-ray; nothing is written to GCP. A project without the API enabled, or a role the account lacks, is stated on the page.
  • Ask X-ray knows the bill. Nine stored tools — cost_overview, cost_breakdown, cost_movers, cost_explain, cost_anomalies, cost_commitments, cost_ai, cost_by_org (apportioned by call share, and it says so), cost_recommendations — with routing for 'why did cost spike', 'what does org X cost', 'where could we save', and three skills on the cost pages.
  • Three system alerts: cost_anomaly (a cell's daily cost far above its normal, with the SKUs and workloads that moved), cost_target (a month-end projection over its target), cost_commitment (fee that funded nothing) — thresholds editable under Alerts → System like every other detector. Query alerts can be written over cost_daily.
  • Five Home cards: the cost overview (fleet-wide or for a pinned cluster), commitments, anomalies, recommendations.

Changed

  • The org page's cost is apportioned — the bill knows clusters and workloads, not customers, so an org's share of a cluster's directly attributed cost follows its share of API calls, and every place that shows it says so.
The export lives in tilda-trial-dev.GKEusagemetric (a dataset made for the GKE usage-metering template, never renamed) and covers the whole billing account. Days are UTC usage days and a day is complete the next morning; the console's daily bars use Pacific days, so day-level figures differ slightly and months agree. Nothing deployed.
v3.1.0 2026-08-26
The rest of the roadmap, a year early

Added

  • Did that deploy make it worse — answered with a test, not a glance. For every deploy, each busy Cypher shape's latency histogram from the hour before is compared with the hour after (a chi-square test on the stored buckets); a finding opens only when the distributions differ AND the p95 moved by half again and by 20 ms or more, on two hundred executions a side. It names the deploy and says *tested, not blamed*. A day with three times the usual number of new Cypher shapes opens a plan-churn finding.
  • Dates, not trend lines. Every Neo4j and Weaviate node's disk has a date for its 90 % watermark (Theil–Sen over 30 days of daily medians), shown under the disk chart and as a finding inside 30 days; obs.db has the date it fills its own volume; the AI page projects the month-end spend per cluster, weekends weighted.
  • Long windows are fast, and reach back a year. Five-minute rollups compact into hourly and daily tiers — sums add and histograms merge, so every figure stays exact — and windows over three days read from them: a 30-day overview renders in 40 ms. The range picker offers 60 d, 90 d, 180 d and 1 y as the tiers fill; a pill says which tier a page is reading. Hourly is kept 400 days, daily forever.
  • Every organisation has a page. /org/<alias>: traffic, error rate, p95 and active users from per-org rollups (no log scan), AI spend, Weaviate coverage, recent journeys, and the findings that mention it.
  • Every service has a scorecard. Above its endpoints: the error-budget verdict and the date it runs out, p95 against this hour's normal, open findings, the last deploy, the Neo4j footprint — and *owner: not declared* until ownership metadata exists.
  • A board, a ledger, and the paths people take. /board is every cluster × service coloured by its worst open finding, refreshing live. /waste is the weekly N+1 ledger: which screen × call pairs re-fetch the same thing and what that costs in Cypher, with one real session to open per row. Journey gains a screen-cost leaderboard and /journey/paths: the common screen sequences, and where sessions end after an error.
  • Optimizer suggestions. For the busiest slow shapes a model proposes an index, a rewrite, batching or caching — each with the effect it expects and the evidence that would confirm it. Suggests only; a human marks it applied, dismissed or observed.
  • Charts you can interrogate. A crosshair with real values on every time series, one shared time cursor across the page, and drag-to-zoom: a selected range becomes the page's window. Saved views live on the server for signed-in readers, with permalinks whose window is frozen so the link shows tomorrow what it shows now. Live fragments update over a server-sent event stream. A tablet layout for on-call. Journey sessions replay at 10×.
  • Explanations that read well without inventing anything. New warning and critical narratives are rewritten for flow by the fast model and kept only when a validator finds no number, identifier or address that is not in the facts. Warning and critical findings carry the story around them — deploys, hand changes, the 5xx and p95 peaks, degrading shapes, overlapping findings — labelled adjacency, never cause; Ask X-ray can tell it too (incident_story).
  • Watches you write in plain English. "Tell #ops when docsvc p95 on production stays above 2 s for 10 minutes" becomes a watch: the model (or the built-in reader) fills a closed spec — cluster, service or org, one of ten measures, an operator, a threshold in the measure's unit or as a multiple of this hour's normal, how long it must hold — never a query. You see it in words with a 24-hour backtest (how many buckets it would have fired on) before it exists; then it is evaluated every five minutes against the rollups, opens a finding while it holds, resolves when it stops, and routes to the channel you named. Sustain semantics: every bucket in the window, on buckets that have data — a gap is not a breach.
  • Admin → Baselines shows every hour-of-week profile's coverage and one profile as a week strip with this week's dots on it, with a rebuild button.
  • /healthz reports each cluster's ingest age, detector state and stale clusters; the ingester logs one quota line per cycle.

Changed

  • Slow statements are stored once per distinct text. The slow_query table carried the full Cypher on every one of a million rows — most of the file. New rows carry a hash into slow_detail; a batch script moves the old text (1,091,733 rows became 133,921 distinct statements on the local copy). Reads are unchanged; the file shrinks at the next VACUUM.
  • Frozen evidence windows work: from/to given as epoch seconds were not being parsed, so the links on findings fell back to the default range. They hold now, and so do brush-to-zoom and permalinks, which use the same form.
Every ticket in the year's roadmap except the DuckDB gate (deferred to January 2027 by the owner) landed on the local branch on the day after 3.0.0. Nothing deployed; the deploy manifests and the kustomization image tag are untouched. New background work — compaction, the weekly N+1 ledger, the optimizer — is bounded per scheduler tick and logs one line when it does something.
v3.0.0 2026-08-26
X-ray finds things, tells you, and answers back

Added

  • Findings. X-ray now makes claims it can defend. A finding is one line with the numbers in it, the facts behind them, links whose time windows are frozen so they show tomorrow what they show now, and the provenance of the data. One row per condition — it updates while the condition persists, escalates when it worsens, and resolves itself fifteen minutes after a healthy detector stops seeing it. Acknowledge, snooze or resolve from the page; the history is kept. /findings, /findings/<id>, and the count sits on the Findings tab on every page.
  • Thirteen detectors, and a page that says whether each one ran. Peer outliers, burning error budgets, degrading Cypher shapes, hand changes that bypassed ArgoCD, N+1 fan-out, stale ingest and stale collectors read the stored rollups every five minutes; novel errors, stalled Weaviate queues and unattributed AI spend read the live sources. A detector whose source fails records the failure and resolves nothing — a GMP outage must not read as "every queue is draining". /detectors shows last run, errors and a *run now* button.
  • What is normal for a Tuesday at 21:00. Every service and measure has an hour-of-week baseline: the median and a robust spread over the last two to eight weeks, only from buckets the ingester actually wrote (a gap is a gap, not a quiet period). An anomaly is a value far outside that for three of the last four buckets — not a threshold anyone configured. The overview charts draw the expected range as a quiet band behind the line, and the fleet grid says *+40% vs normal* instead of comparing the second half of a window with the first.
  • A date, not a trend line. The error budget page forecasts when each service's 30-day budget runs out at the current rate (a Theil–Sen fit, so one bad day does not drag the line), and opens a finding when that date is within two weeks.
  • Every finding is explained in words — what happened, what normal is, what landed just before it (labelled *adjacency, not proof*), and what the finding cannot tell you — written from the finding's own facts. A test checks that no number appears in the text that is not in the facts.
  • Errors X-ray has never seen. A 90-day memory of error shapes per cluster; a shape appearing for the first time, five or more times in an hour, opens a finding with an example trace. It learns silently for its first day.
  • Findings reach Google Chat — through an outbox, in dry-run by default (superseded in 3.1.0 by the Alerts tab, which adds in-app pop-ups and webhooks). A space's incoming webhook is added on Admin → Notifications with a short name; rules pick the channel from a dropdown and the clusters and kinds from checkboxes. One Chat thread per finding, so an escalation lands under the message that opened it. Duplicates within a day, rate limits and quiet hours are recorded as rows rather than dropped; warnings can wait for a daily digest while criticals go now. Nothing is sent until a human sets OBS_NOTIFY_MODE=live after a test send reads right; the outbox shows every message, sent or held, and why.
  • Alerts — one tab for the whole lifecycle. An alert says who gets told about what: either a filter over the findings X-ray opens (kinds, severity floor, clusters, services) or a condition you write yourself. Create it in plain English — "pop up in X-ray and tell #ops when docsvc p95 on production stays above 2 s for 10 minutes" — or by form; the reading and, for conditions, a 24-hour backtest are shown before it exists. Then list them, search them, filter by creator, by recipient and by cluster, turn them on and off, edit, duplicate, delete. Every enabled alert that matches delivers — alerts belong to different people and do not compete — holding warnings for its digest in quiet hours, capping its own rate, and muted for an hour, a day or a week when you say so.
  • Many findings, one message. "Notify me whenever there is an error in doc service" must not become a hundred notifications when the error happens a hundred times. Each alert now sends at most one message per group per interval (five minutes by default; "at most once every 15 minutes" in plain English), and only when the group changed: *docsvc errors: 40 findings firing (40 new) · 1,212 events in the last hour*, then the ten worst, each with its own count — "×100 in the last hour" — and a link. Nothing changed, nothing said, until the reminder (every four hours while findings stay open; "no reminders" turns it off). Resolutions ride along in one line; a finding that flip-flops stays counted as firing until it has been quiet for half an hour rather than producing open/closed pairs all night; a group that keeps changing backs off (×2, ×4, ×8). A single new finding still reads as the plain message it always was. Groups can be per alert, per cluster, per service or per finding kind. The design is taken from a survey of twelve open-source alerting systems (Alertmanager's group interval, repeat interval and content hash; Sentry's per-rule frequency and "seen N times in an hour"; ElastAlert's exponential re-alert; Prometheus's keep_firing_for) and written here, in one module with its own tests, so X-ray owns it.
  • Alert on anything X-ray records. A notification is not only about errors: a user hitting a screen, an org going quiet in one area, a journey repeating calls, an endpoint making a hundred queries per request, a test job failing. A third kind of alert — a *query* — makes that reachable: the model reads the request with the full catalogue of what X-ray stores (every table and column, in the words the schema was written with) and of the read-only tools that reach journeys, traces, logs, evals and tests, writes the query, RUNS it and looks at the rows before proposing it; you see the query, its result right now and a backtest before pressing create. Every query — the model's or yours — goes through a guard that cannot be argued with: a read-only connection, an allow-list of tables (token hashes, webhook URLs, chat transcripts and prompts are never readable), one SELECT only, a row cap and a deadline. When X-ray does not record what you asked, the answer is "X-ray does not record that", not a plausible query over the wrong table. Query alerts fire as findings, so grouping, channels and the inbox apply; the built-in reader still handles the closed set when no API key is configured.
  • Channels anyone can add, that any alert can link to. A pop-up in X-ray (the bell in the header, a toast when it lands, an inbox per person), a Google Chat space (one thread per finding), or any webhook URL (a JSON POST with the finding and a link, HMAC-signed when a secret is set). External channels stay in dry-run — rendered into the outbox, nothing sent — until a human sets OBS_NOTIFY_MODE=live after a test send reads right; in-app always delivers.
  • Fewer, weightier findings. The more rows a board shows, the less each one means. Peer outliers collapse to one informational row per service (they are structural more often than news, and only a 5xx-rate outlier is a warning); a Neo4j poller going quiet or traffic moving on a test cluster is informational; impact is weighted by cluster tier so a production warning outranks a test-cluster oddity; and the board and Home show warning-and-above by default, with the informational rest counted, one click away.
  • A home page that briefs you. / now opens Home: what changed since you last looked, the open findings ranked by impact, the forecasts that have a date, a fleet grid coloured by each cluster's worst finding, whether X-ray itself is keeping up, and questions to ask.
  • Ask X-ray. A chat that answers only from X-ray's own data. The model is given twenty-eight read-only tools — each a function a page already calls — and a rule that it may state no number a tool did not return. Every tool call it made, with row counts and latency, and every query behind them are shown beside the answer; citations link to the calls; any number no tool returned is marked *unverified*. Every call is recorded with its cost, under a daily cap per person and per instance. Needs an API key; the page says so when none is configured.
  • MCP. Claude Code and Claude Desktop can connect to X-ray at /api/v2/mcp with an existing API key and use the same tools Ask X-ray uses. The command is on the API tokens page.
  • API: GET /api/v2/findings, /findings/<id>, POST /findings/<id>/ack and /resolve, GET /detectors.

Changed

  • / lands on Home. This is why 3.0 is a major: the front door means something different. The overview you had is unchanged at /overview and one click away; OBS_HOME_DEFAULT=false restores the old landing.
  • Archive scans are bounded twice. At most 20,000 rows and 64 MiB of kept rows per scan, eight scans at once. Counting never stops — the totals stay honest — only keeping does. Twenty-four scans of fifty thousand parsed entries did not fit a 2 Gi pod.
  • One sanitiser for every value that enters a Cloud Logging filter. The API's /logs and the live service filters interpolated query-string values raw, so a crafted service= could widen a cluster-scoped read to the whole project. Every path now passes through sources.sanitize_filter_value, and the API's severity is an enumeration.

Fixed

  • /export/<unknown view> was a 500, not the intended 404: HTTPException was raised without being imported.
  • The __main__ guard sat above fifteen routes, so python -m obs_dashboard.web served a dashboard with no Tests, Evals, Guide, Releases or Tokens pages. It is last now, and a test keeps it there.
  • uv.lock said 2.9.1 while the badge said 2.9.18; a test now checks the three places the version is written.
Built on the local branch xray/v3, tested against a copy of the production data (seven clusters, 13 detectors and the baselines run in 7 seconds), and NOT deployed. Everything new is off or inert by default: detectors read SQLite only until OBS_DETECT_LIVE is trusted, notifications are dry-run, the LLM needs a key, and the deploy manifests for the secrets are committed but not applied. The plan, the decisions and the year ahead are in the obs v3 folder: DuckDB was considered and deferred to a January gate; the rollup tiers that make long windows fast come in 3.2.
v2.9.18 2026-08-25
A theme switch you can find, and release notes you can page through

Changed

  • Light and dark are now a visible control, not a pill that said "auto". The appearance switch has been in the header since 2.9.4 as one small pill cycling auto → light → dark on click, and nobody recognised it as a theme switch — the owner asked for one while it was on screen. It is now a three-way control with icons and words: *Light*, *Auto* (follow the system — still the default, so a first-time viewer never gets a theme they did not choose) and *Dark*. The pressed state is drawn from the theme actually applied, so the control can never disagree with the page.
  • Release notes are paged, ten releases at a time, newest first. Fifty-five releases had become one very long page. A pager sits above and below the notes; page 1 is the newest, so the arrows read as time — *newer* to the left, *older* to the right. page=all still gives the whole ledger on one page for anyone who wants to search it with the browser, and when the build you are running is not on the page you are looking at, the page says where it is. A stale link to a page that no longer exists lands on the last real page rather than an empty one.
v2.9.17 2026-08-25
A repeated call is one you made twice in the same breath

Changed

  • The Journey page called a call "repeated" if it happened twice anywhere on a screen, and that number was mostly wrong. On one real session it claimed 85 repeats on a screen where 3 were real. A screen visited seven times shows every landing call seven times, and each extra one was booked as waste; POST /api/myprofile/stillActive — a keepalive whose entire job is to fire periodically — was reported as 8 wasted calls. A count that is wrong 96% of the time trains you to ignore the pill, which costs more than not having it.
  • A repeat is now scoped to one trigger: the same method, path, query string and body fingerprint twice inside a single press, or inside a single background burst. That is the boundary the page already draws as a fold, and the count is computed from the folds themselves so the number cannot disagree with the rows beneath it. Crossing a trigger is not a repeat — the page fetching what it needs again after you did something else is not a defect.
  • What this surfaced immediately: GET /api/v1/threads firing 22 times in two seconds in one background burst on a trainings screen — identical query and body every time. Under the old rule that screen read "30 repeats" and the 22 were indistinguishable from landing-call noise.
  • Same call, later in the same visit, is reported separately as refetched. Polls and keepalives belong here, and so does the borderline case worth a look: the same screen asked GET /api/taskfilters/ twice six seconds apart, which the three-second fold boundary puts in adjacent folds. It is reported rather than judged, because waste and legitimate periodicity are not separable by elapsed time alone.
  • "Dark" on the Skills tab says what it means in words. The label read *wired but gated off until run request enables it*, which is accurate and answers nothing: it names a mechanism instead of a consequence. A dark skill is code that ships and works, that an ordinary request cannot reach — the caller has to ask for it explicitly. The tab now says that, and the counts beside each kind read "available to an ordinary request" and "off until a caller asks" rather than live/dark.
  • The row badge now names its scope — *same call, 2 of 2 in this burst* rather than *2 of 6 on this screen*. Two adjacent rows that were different questions both read "2 of 2" before, which looked like they were duplicates of each other. The scope is read from the fold, not the row, because only the row that opens a fold carries the interaction that triggered it.
v2.9.16 2026-08-25
A kustomize apply silently rolled the site back to 2.7.3

Changed

  • kubectl apply -k deploy/ reverted production from 2.9.14 to 2.7.3. The kustomization carried newTag: latest with a note to "bump at deploy time", and :latest on this repository still points at a build from 2026-08-19. The only symptoms were a version number in the header and an API endpoint answering 404 — no error, no failed rollout. The tag is pinned to the real build now: a kustomization that reverts production when applied is worse than one that is slightly stale.
  • An authenticated API call could 500 because SQLite was busy. verify_token updates a last_used_at timestamp after the SELECT that actually authenticates, and a write lock there took down the whole request — the skill-graph CronJob's first push died on database is locked while the ingester held the lock. That write is bookkeeping: it now waits briefly and then gives up quietly, because a missing timestamp is cosmetic and a failed API call is not.
v2.9.15 2026-08-25
The Skills tab gets its snapshots from a CronJob instead of my laptop

Changed

  • "No snapshots yet" was the honest answer. The graph is built by IMPORTING chat-agent's workflow registry, so it needs that branch's source and an interpreter that can import it — 1.6 GiB of repo and a 577 MiB environment. The deployed pod has neither, and the only snapshots that ever existed were on a laptop.
  • A CronJob now builds them every six hours for origin/master and origin/production, in a purpose-built runner image with git and uv. Its workspace is a persistent volume, so the first run clones and every later one is a git fetch — on an emptyDir it would re-clone 1.6 GiB four times a day to observe something that only changes on commit.
  • It POSTs to a new /api/v2/skill-graph/{branch} rather than writing the volume, because x-ray-data is ReadWriteOnce and a job on any other node cannot write it. Same route the ArgoCD collector already takes. Snapshots go through the READER's own directory function, so a push cannot succeed into a directory the page does not read — and a snapshot with no workflows is refused, because an extractor that imported nothing looks exactly like an empty capability surface.
  • The data volume is 30Gi, up from 10Gi. 10Gi was "generous (the DB is ~100s of MB)" when it was written and then quietly stopped being true.

Versions are judged by effect on the reader, not by how much code moved: major when a page is gone, moved or now means something different; minor for a new page or a new answer on an existing one; patch for a fix to something the page already claimed to do. The same notes are in CHANGELOG.md, generated from the same source so the two cannot disagree.