Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Max · 30 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-21 17:32:31.
› →|
updated 3d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Releases

What changed in X-ray, and which build you are looking at right now. X-ray reads live infrastructure, so when a number looks wrong the first question is whether the fleet changed or the dashboard did — this page answers the second half.

running versionv3.9.21
commitdev
built—
sourcelocal checkout

This instance is running from a source checkout, not a released image — the commit is read from git and there is no build timestamp. A deployed build stamps both at image build time.

The build you are running (v3.9.21) is on page 1.

v3.5.1 2026-09-03
Every page shows what it cost to load

Added

  • A cost line in the footer and in the "show queries" panel. It sums the per-request query ledger: how many BigQuery queries the view ran, the bytes they scanned, how many were served from cache ($0), and the dollar cost at $6.25/TiB — e.g. *$0.0214 to load · 4 BigQuery queries · 3.51 GiB scanned*. BigQuery reports total_bytes_billed once a job finishes, before any row is read, so the number is exact rather than estimated. SQLite and Monitoring reads are counted but shown as unbilled, because they are not charged by bytes.
The recorder that powered the existing "queries behind this view" modal now carries each query's billed bytes; the BigQuery reader fills them in where the job runs, so the four scattered record points collapsed to one and can never double-count a page's cost. A fragment reports its own cost; the page shell reports the shell's.
v3.5.0 2026-09-03
BigQuery is the only log reader — the Cloud Logging client is uninstalled

Added

  • One table answers every log question. Each project's logs_raw.entries holds every entry it routes — app logs, k8s events, the Neo4j forwarder, Auth0, and now the Admin Activity / System Event audit trail through a second sink — as one JSON column per row. Nothing is inferred, so a service changing a field's type tomorrow changes nothing; the filter language maps onto the entry's own key paths, so all 46 read sites kept the filters they had.
  • The change trail became a query that costs a fraction of a cent. Audit entries used to be answerable only through a view that could not prune — ~2.5 GiB billed per question, whatever the window. Landed in entries the same question prunes to its hour. tilda-control-plane — where the ArgoCD sync trail lives — gained the same pipeline; it had none.

Changed

  • Every timestamp bound now prunes the partition column for free. timestamp >= X implies publish_time >= X exactly — an entry cannot publish before it exists — so the translator emits the prune beside every comparison, and refuses any filter with no lower bound: the difference between $0.0001 and scanning a month.
  • The scheduled/interactive backend split is retired, because the reason for it is gone. Scheduled work read the free API so the 288-cycles-a-day ingest would not bill ~$10/day against the linked views. On entries the same cycle's queries bill about $0.0001 each — measured, not estimated — so one backend serves both.

Fixed

  • A refused filter now says so instead of falling back. With no second backend, sample(insertId, 0.5) surfaces as an explicit error on the page rather than a confident zero or a silent 60-second retry elsewhere.
google-cloud-logging is removed from pyproject, not merely unused — a dependency that is merely unused comes back. Coverage starts at the sinks' birth (2026-09-02) and grows to a rolling 30 days; the wall between BigQuery and the ten-year GCS archive is measured from the table hourly, never assumed, so the two can neither double-serve a window nor lose a band of it. Logs Explorer itself, the _Default sinks, and the archive are untouched.
v3.4.2 2026-09-02
The collectors stop paying for BigQuery, because nobody is waiting on them

Fixed

  • The ArgoCD collector was most of X-ray's BigQuery bill, and it never needed to be. It reads the audit log in its own process, so the server's unattended marker did not cover it, and it took the billed path: $0.107 a run at 96 runs a day — about $10/day — measured from BigQuery's own billed bytes. Every scheduled entry point now marks itself, and a test notices when a new one does not.
  • The free backend is not even slower here. The 214-second timeout that first pushed these reads to SQL was a 48-HOUR audit window. Over the two hours this job actually asks for, the Logging API answers in 10.7s against BigQuery's 12.4s, returning identical rows — so this was cost with nothing bought.
  • Hourly instead of every fifteen minutes, and a two-hour window. Reducing the window is the obvious lever and it is the wrong one: bytes billed hardly moved between a 20-minute and a 6-hour lookback, because the cost of a log query here is the columns it names, not the rows that match. How often it runs is the lever.
  • The audit response blob is no longer fetched. Nothing reads it — request is what says which field of a ConfigMap changed — and it was a tenth of every audit query's scan, retrieved for no reader.
  • The collectors talk to X-ray inside the cluster. --via-api pointed at the public host, so a job running beside the dashboard went out through the load balancer and back; that is why its five failures on 2026-09-01 were HTTP 502 while the dashboard was restarting. The Weaviate job on the production cluster keeps the public URL, because it runs somewhere the in-cluster name does not resolve.
A caution for whoever tunes this next: BigQuery's dry-run estimate is not trustworthy on a Log Analytics view. The same query reported 15.9, then 26.6, then 31.3 GiB within twenty minutes, while a controlled A/B was stable at 2.7. Only INFORMATION_SCHEMA.JOBS_BY_PROJECT.total_bytes_billed on real runs was reliable, and every figure quoted here comes from it.
v3.4.1 2026-09-02
The pages say which reader answered them, and how far back it can see

Fixed

  • Every page that explained itself in terms of Cloud Logging was describing the backend it no longer uses. The jobs and Consumers pages still said job is not an indexed field *so Cloud Logging scans the window* — the exact thing 3.4.0 fixed — and the Logs page announced a three-day wall. Reworded to name the reader that actually answered, which the page computes rather than has written into it.
  • The three-day wall was wrong on the two clusters most people look at. It was the Cloud Logging API's _Default retention, and true while the API was the only reader. Read through a Log Analytics link the reach is whatever the bucket keeps — thirty days on automation and dev, three on production — so a wall computed from three days also handed the archive twenty-seven days the live reader could answer itself. Every page wall now asks the bucket, and the sentence beside it says the real number.
  • Cost data stopped on 2026-08-31, so September read $0.00 with no explanation. The cost sync had been switched off on the cluster by hand, undeclared and unexplained, so nothing said why the Costs pages were frozen. It is on again and declared, at a measured $0.02 per refresh.
  • The Weaviate collector on the production cluster was still running 2.7.3 from 2026-08-19. 3.4.0 pinned the two collectors on master and missed this fourth job, which lives on a different cluster. All four now run the same build as the app.
Auditing every job that fills X-ray found no duplicate work: the two Weaviate collectors cover disjoint clusters (master's does master, automation, nexus, test26, test27; production's does production), delta runs no Weaviate so its absence is correct, and the ArgoCD collector runs from master only by design. Its five earlier failures were HTTP 502 from the dashboard being down 21 hours ago, not a fault in the collector.
v3.4.0 2026-09-02
Log questions answered in SQL — and the laptop stops paying for them

Added

  • Every log question is now asked in SQL, against a Log Analytics view of the log bucket Cloud Logging already fills. The read API allows 60 reads a minute per user and cannot index a jsonPayload field, so every question about one was a full scan: measured on master, 189 seconds to find 4,153 entries carrying jsonPayload.job in a ONE-hour window. Two of six filters measured could not be answered at all — 504 Deadline Exceeded after 60 seconds, the page throwing, caching nothing, and re-running the same doomed query on the next refresh. Those two now answer in 7 and 22 seconds. Where both backends could answer they returned identical row sets, compared by insert_id.
  • Who changed a ConfigMap or a Secret, and what changed inside it, over 400 days. Audit entries arrive parsed into typed columns, so the change trail is a query rather than a scan. The read API cannot do this at all: asked for 48 hours of it the read timed out after 214 seconds, while the SQL form returned 2,878 rows in 7.

Changed

  • Scheduled work reads the free backend; pages read SQL. The two are priced in opposite directions — Cloud Logging bills ingestion only, so a read is free however slow, while BigQuery bills bytes scanned and a log question cannot prune below the whole json_payload column for its window. So the deciding question is the one from the kind axis: is anyone waiting? The ingester, its backfill, podhealth, the detectors and maintenance are marked unattended as one thread and take the free path; a page render takes SQL, which is where a 189-second scan is felt.

Fixed

  • X-ray no longer downloads the log archive when it is not running in a pod. In the cluster these reads are same-region and free — eleven months of billing show $0.00 to $2.07 a month. On a laptop the identical code is metered internet egress: 2,589.8 GiB and $283.11 of Download APAC in 2026-08, 715.5 GiB on the 28th alone, and ~430 GB/day off the developer's own connection until it exhausted their allowance. run_scheduler and archive_backfill now default off unless KUBERNETES_SERVICE_HOST is set, the guard the compactor got in 3.3. In-pod behaviour is unchanged.
  • The archive gate now sits where the bytes leave, not where someone remembered to check. archive_backfill was consulted in exactly one place while the Logs page reached GCS through its own route and the timeline through a second HTTP stack, so switching the archive off still let an ordinary page view stream 256 MB shards with a prefetcher warming the window behind it. Both urllib primitives now refuse, and a trip-wire test asserts that no path attempts a download — it caught two that the first fix had missed.
  • The two collectors were running fourteen-day-old code. x-ray-argocd-collector and x-ray-weaviate-stats asked for :latest, which points at 2.7.3 built 2026-08-19 and has not moved since, while the app ran 3.3.1. They sat outside the kustomization that pins every other image, so nothing bumped them and nothing said so. The tag is pinned in one place per kustomization now.
The BigQuery half is billed on bytes scanned, and the honest figure is a few cents a page: a jobs render measured 4 queries and 12.72 GiB, about $0.078, with a ten-minute scan cache so refreshes inside that window are free. Background work adds nothing, because it does not use it.
v3.3.1 2026-09-01
Background jobs and Consumers: two pages, real run counts, and one honest ceiling

Added

  • Background jobs and Consumers are two pages now, split on a rule that makes the difference checkable: *anything with a schedule is a background job*. The test is a week of zero traffic — whatever still fires is unattended work. That puts 32 jobs on one page (21 scheduled, 11 housekeeping) and 88 consumers on the other, and it matters because a consumer is the REST OF SOMEBODY'S REQUEST running in another process: it follows traffic, so a quiet hour there is correct, while a gap in a sweep's cadence is an incident. Listed together, every consumer read as a background job and the headline was roughly five times too big.
  • Runs, failures and durations for every job that logs anything. These columns used to read one source — the Job completed record Go's RunJob writes — and almost nothing writes it, so most rows were dashes while the job's own log lines sat right there. A run is a BURST of them: workflow prints "cron run started", works, prints "cron run completed", then goes quiet for two minutes. Counting bursts counts runs, a burst's span is the duration, and a burst containing an ERROR is a failed run. Jobs with a run count went 9 → 16 of 21, with a duration 2 → 9, and every figure checks against its declared cadence (502 runs of a 30-second sweep in six hours, 72 of a five-minute one). The measured interval is printed beside the declared one, because a cadence read out of a config file is a claim.
  • Search, service filters, sortable columns, pagination, and a page per invocation. Search covers name, service, subscription and evidence; sorting is server-side and in the URL so a link carries your view; blanks sort last in both directions, because a job that cannot report a duration is not the fastest job. Clicking a run shows what it said and the trace it belongs to — two sections, because neither joins the lines alone: a sweep that never leaves its process has no trace, and a time window stops at the service boundary.
  • Spend by job on the AI tab. It reported a total it could not decompose: 93% of master's LLM calls name a job and there was no way to see which. Verified the two pipes agree rather than assuming it — BigQuery $71.86 / 10,015 calls against Cloud Logging $73.03 / 10,164 rows over the same window.

Changed

  • Log rows on a job's pages are the Logs page's own rows. They were a five-column table, so any field the log carried and the table had no column for was invisible — transactionID on every Go line that touches the database, the per-instance job name, the raw payload. logbody also gained job, trigger and transaction, which every page using it now shows.
  • The jobs come first on the page. Four paragraphs and a full-width pill sat above the table, so the first row started below the fold. The explanations are still there, folded under the table, because several of these columns need a sentence before they mean anything — but they are reference material, not the headline.

Fixed

  • The index and the detail page disagreed about what a run is. workplan.summarization-sweep said 698 runs and then listed none, because the index derived runs from log bursts while the detail page listed only Job completed records. finsvc.budget-scheduling writes those, looked correct, and hid the fault. Both derive from one function now: 240/240, 120/120, 121/121.
  • Healthy jobs were reported as outages. "Not heard · past due" was printed for jobs ticking perfectly — va.in-progress-recovery logged 297 lines in 24 h against 288 expected from its five-minute interval. Silence now has five states instead of two: *observed*, *heard* (a per-run line, checked against cadence before it is trusted), *registered* (a startup line proves the scheduler holds it), *unobservable* (it only logs when it finds work, so silence is the success case) and *silent* — the one state worth an alarm. False "past due" claims went from 6 to 1.
  • Rows with no name were notifsvc naming every subject-visit reminder by its database UUID. Each fires once and is removed, so there is one row per reminder and thousands of them crowding out the real jobs. They are folded into notifsvc.visit-reminders. Those lines carry a JobName field that looked like the obvious thing to display instead — it reads "Subject <name> Study …", so it is deliberately never read, and a test enforces that.
  • The page looked like it hung; it was failing and retrying. A six-hour scan sits on the log client's 60-second deadline — 51 s when it lands, a timeout when it does not — and a timeout cached nothing, so every refresh re-ran the same doomed query while the route returned 200 throughout. Failures are now remembered for a minute, and the too-large-window state says so and links to shorter ranges instead of showing a blank error.
  • Nothing was cached on these pages at all, so every refresh and every column click re-ran three live log queries. The scan is now held for ten minutes and search, sort and paging are arithmetic over its rows: 25 s cold, 8 ms warm. Ten minutes rather than two because a single scan takes longer than two — a cache whose TTL is shorter than the work it caches is not a cache.
  • Seven scheduled jobs were missing from the inventory entirely (the virtual assistant's apscheduler crons), and six entries were keyed on a prose label or a guessed spelling, so the row and its own log line could never meet — each producing two rows, one under "not yet classified" and one reading "never heard". Unclassified rows: 0.
The ceiling, stated rather than hidden. These pages read Cloud Logging live and jsonPayload.job is not an indexed field, so the API scans the whole window: measured at 189 s to find 4,153 entries in one hour on master. Caching hides that on the second view and cannot fix the first, and a window much past six hours exceeds the client's deadline. Making this genuinely fast means ingesting into a rollup the way every other fast page here works. That is not in this release.
v3.3.0 2026-08-31
PostgreSQL as an option, three pages that ask far less, and the outage that taught us why

Added

  • X-ray can run on PostgreSQL. OBS_PG_DSN set means Postgres; unset means SQLite, which is every deployment today and stays the default. Nothing else in the app changed — the 456 conn.execute(...) sites keep the sqlite3 API and a new dialect layer makes psycopg answer to it, so there is a way back if this turns out wrong. All 62 tables and 131 indexes are built from the same schema.sql; 25 of 25 pages serve from Postgres, and 6.6 million rows import in 601 s. WHY, given SQLite is not slow: the real indexed page queries measure ~11 ms and tuning cache_size and mmap_size on the 8.5 GiB file moved that to 10.4 ms. It is a concurrency problem. X-ray has 30 user-facing write endpoints, so N replicas means N writers, and SQLite's WAL cannot be shared across hosts — it coordinates through shared memory, and on a network filesystem the documented outcome is corruption. One SQLite file on a ReadWriteOnce volume is what pins this app to one replica and makes every deploy an outage. Not yet switched on: nothing ingests into Postgres yet, so pointing the live site at it would give fast pages and frozen data.

Changed

  • Three pages ask a fraction of the SQL they did — /home 282 statements to 23, /findings 142 to 12, /fleet 179 to 102. detector_settings was read once per detector inside loops, 260 queries for one /home render and every one returning nothing, because that table is empty until an admin changes a threshold. baselines was read once per bucket per service on /fleet. It costs little against local SQLite and it is the whole difference between a page that survives a network database and one that does not.
  • The dashboard pod now asks for 4 CPU and 6 GiB, up from 2 and 2. Nothing leaked: this pod ingests seven clusters, keeps a 60-table rollup, runs the log-archive compactor, and runs a cost extractor that had never executed here before. The old limits were written for 2.9.18.
  • Liveness and readiness probe /livez, which touches nothing. /healthz keeps its freshness body for external alerting, but it opens the database and therefore cannot be a probe.

Fixed

  • “Internal server error”, repeatedly, was the health check ejecting a working pod. Every one was 502 failed_to_pick_backend — the load balancer with no healthy backend — while the app itself had returned no 5xx at all. The BackendConfig named no timings, so GKE defaulted timeoutSec: 1 with unhealthyThreshold: 2, and /healthz opens a connection whose PRAGMA journal_mode=WAL queues behind the ingester's write lock: 5,102 ms measured, against a one-second timeout. With one replica an ejection is a total outage. Every timing is now written down — 15 s, 3 strikes, 45 s of grace — and the probe touches nothing.
  • Pages taking tens of seconds: /events held a worker thread per open browser tab. It was a sync generator calling a blocking 25-second wait, Starlette runs sync iterators in the anyio threadpool, and FastAPI runs every plain def route in that same pool — default size 40. Enough open tabs and the whole application queued: a finding page measured 71,417 ms while its own handler ran in 917 ms, and the numbers were multiples of 25 s because that is what they were waiting on. The stream is async and non-blocking now, stops when a tab closes, and the pool is raised to 256 so the next blocking caller degrades one request instead of everything.
  • The pod was OOM-killed three times in twelve minutes after the 3.2.6 upgrade. The cost module had never run on this deployment — cost_daily was empty — so on first start it backfilled the billing export from scratch, and the extractor holds a whole window in memory before inserting. Memory went 192 MiB to 1.9 GiB in two minutes through a 2 GiB ceiling.
  • deploy/deployment.yaml had not parsed for days — envFrom: and its first list item were joined onto one line, so kubectl apply failed outright. That is why this Deployment had only ever been changed imperatively. Every manifest in deploy/ now validates.
  • Two schema bugs a stricter parser found. chat_messages had a comma stranded inside a comment, so status and context had no separator — SQLite accepts that and produces the right columns, PostgreSQL does not. And cost_commit_daily.offset is a reserved word in PostgreSQL; exactly one of 548 columns collides, so it is quoted rather than renamed.
  • A pasted /frag/... URL lands on the page it belongs to instead of collapsing to the first path segment.
Several of these fixes are repairs to damage done while shipping 3.2.6. They are written down rather than quietly folded in, because the next person to see a 502 from a healthy pod should find the reason in the ledger.
v3.2.6 2026-08-31
Documents, a searchable bill, and one sentence for “is production in sync?”

Added

  • AI → Documents. What each document cost the fleet in model calls, and how far it got — uploaded, TMF pipeline, validated in a work order, indexed for search — said in words rather than stage codes. A document's own page carries its totals (tokens in/out/cached, model time against wall time, spend, failures), says whether it is new or was already here, and charts every call on a time axis from the run's start so a slow step is visible instead of inferred. It also names the work that cannot be attributed: operations that expect a document and named none. That list is the honest edge of the data, not a gap to hide.
  • Costs → Search. One box over every dimension of the bill: a word (autopilot — $4,217, 39% of 30 days; neo4j — $1,935), a service, a cluster, a workload, or a pasted gs:// or console link, which reduces to the bucket and prices that bucket alone. Resources are counted once across columns rather than summed twice. Label search came with it: 43,110 label-days for about $0.06, so an istio search stops answering “nothing matches” when the answer is $7,993.
  • ArgoCD → In sync? One sentence per environment: does what is running match git, or did somebody change it by hand. Read from each cluster's own audit log, so it catches a kubectl edit that never touched an Application and appears in no sync history. Three depths on one page — the verdict, then the services with the image each is running, then config, secrets, RBAC and CRDs. A replica move is reported as capacity, not drift: git does not define replica counts, and 23 of 155 hand-writes in a 72-hour sample were scale alone.
  • Alerts are written for a reader. Every notification now goes facts → template → a model rewrite under rules that forbid new numbers or names, so a card says what happened and to whom instead of listing raw titles.

Changed

  • ArgoCD → Clusters is the fleet breakdown it always claimed to be — every environment with what ArgoCD holds, how much it reconciled and when it last did anything, read live from the instance's own audit log. It needs no pushed table, so it is true on a laptop and after a collector stops. It also reports an environment that is live but undeclared, which found dev on its first run.
  • Costs → Search is in the sidebar. It existed, worked, and was in no nav group, so it appeared neither in the menu nor in the guide's page list and could only be reached by submitting a form.
  • The guide covers the jobs added since 3.1 — the bill going up, what one document cost, whether production is in sync, a screen that feels slow, and being told when something happens again. Tests now fail if a nav item has no blurb, if a guide link dies, or if a page is missing from the menu.

Fixed

  • “Create alert” works again. A hidden trigger block still validated, so a query proposal's for_minutes: 0 failed a min=5 field inside a closed section and the browser refused the submit with nothing on screen. Inactive blocks are now disabled, and a refused submit opens the section and says why.
  • “Trace” on the Waste ledger opens the session it points at. Three faults, each enough alone: the link carried the reader's window instead of the week the ledger sampled; one session was fetched by scanning the window and filtering afterwards, so anything outside the newest 4,000 rows was unreachable; and an unfinished archive scan was cached and then reported as a finished, empty one. The session in the report had 180 rows the whole time.
  • “Who changed a secret” was missing most secret changes. This fleet manages secrets through External Secrets, and externalsecrets was not in the tracked kinds — 87 of them against 15 raw secrets in seven days on production.
  • Audit reads no longer fall back to the log archive. Audit logs live in _Required (400 days); the archive backs _Default (3 days) and holds no protoPayload, so the fallback scanned shards that could never match and printed “still reading the archive — 0%” over an answer that was already complete.
  • A page with no query shapes renders its empty state instead of a 500. shape_trends dropped the (rows, total) shape on the empty path, which made /neo/queries fail on a fresh install, a quiet cluster or a narrow window — found by a new test that walks every page in the menu.
  • The laptop stopped paying for log egress. The archive compactor defaulted to on everywhere, so a local run pulled 1.4 TiB out of Cloud Storage in four days ($159, heading for ~$2,400/month). It now runs in-cluster only.
  • Timeline and “who changed what” no longer call an uncollected page a quiet one. “That is a real answer, not a gap” was printed on pages that simply had no data; the two cases are now separated, and the empty one points at the pages that need no collection.
v3.2.5 2026-08-28
Log search beyond three days: compact once, search small

Added

  • The archive compactor. The raw log archive is one object per hour for the whole project — on dev 2.25 GiB an hour, 96 % of it one cluster's Neo4j firehose — so a 14-day search on one cluster downloaded 25× its own data and never returned. The pod now reshapes each finished hour once: per-cluster gzip parts (13.7× smaller on production), a token bloom filter per hour ("could this hour contain the word?", ~1.2 bytes per distinct token), and — when a day closes — one composed day object and one index with byte offsets, all under compact/v1/ beside the raw archive. A resumable walker backfills newest-first to 30 days, six hours a tick, without starving ingest. Measured on 24 real production hours: 1,163 MB → 85 MB; a search over the day for a word no line has: 0.00 s, every hour ruled out by the index without a download; a common word: 50 rows in 0.06 s from 9 hours read. On the pod the reads are in-region. Research: 16-LOG-SEARCH-STRATEGY.md.
  • The Logs page says what the archive read did — compacted hours read, hours ruled out by the word index, raw shards for hours not compacted yet, seconds — and raw-shard reading stops at a 20-second budget per page and says so, instead of spinning for ten minutes.

Changed

  • The archive timeline accepts a bare search word: it was refused as untranslatable offline, so the chart stopped at three days while the rows below covered the window. A bare or quoted term is now the same substring-over-values test the rows do, and the compacted hours' indexes rule hours out for the chart exactly as for the rows.

Fixed

  • The compactor writes once and never deletes: the production archive bucket carries a locked ten-year retention policy, under which nothing can be overwritten or removed (a re-compose answered 403 retentionPolicyNotMet mid-backfill). Every write is create-if-absent, an existing object counts as done, hourly parts stay beside the composed day, and a day whose roll-up failed is retried on the next tick. Storage: the compact set is about a fifth of the raw set on top of it.
  • One download per raw shard: every retry of a page and every background warmer downloaded the same 2 GiB object at the same time (fourteen half copies were found, none finished, 17 GB of partials invisible to eviction). Readers now wait for the one download in flight, a dropped connection resumes by Range, and partial files from a dead process are swept at startup.
v3.2.4 2026-08-28
One message per observation window, with the counts

Added

  • Observation windows. An alert can now *say it every window*: while anything it routes is failing, one message per window (30 minutes by default) listing every member — a failing endpoint, a service, a new error shape — with how many times it happened *in this window*, its total since the episode began, and its state: new · still failing · seen again · quiet 1 of 2. Two failing endpoints of one service are two lines in one service-level message; five failing services in a namespace-level alert are five lines in one message. A member not seen for N windows (2 by default, the alert's setting) is resolved in its own message, then watched: if it fails again the message says *seen again* instead of pretending it is new. Nothing is said in a window where nothing happened. The old way — only when the set of findings changes, with reminders — stays as *say it only when something changes*. Research: 15-OBSERVATION-WINDOW-NOTIFICATIONS-RESEARCH.md.
  • Endpoint failing. A new system alert opens one finding per endpoint failing most of its calls (5xx ≥ 50 % of ≥ 5 calls in 15 minutes; critical at 95 % of ≥ 20). 4xx are shown beside it and not counted as failing — a 404 is the caller's problem, a 500 is ours. Alerts can group per endpoint.
  • Counts come from the rollups. Failed calls per endpoint or service (the API rollup), executions of a Cypher shape, restarts of a workload, occurrences of an error shape — each read for the window, never estimated. A kind without a counter reads *still open this window*.
  • Private alerts. An alert is *private* (only its creator and recipients see it — on the Alerts tab, its page, its messages on the Notifications page, the design runs, the home widgets, the counts) or *public* (everyone with X-ray). New alerts are private unless you say otherwise; an alert the composer makes is always private. Alerts from before that only popped up in X-ray became private; those posting to a shared channel stayed public.

Changed

  • The alert form speaks in the owner's words: *observation window · say it every window / only when something changes · resolved after N quiet windows · one message per alert / cluster / service / endpoint*. New alerts default to a 30-minute window, every-window mode and two quiet windows; existing alerts keep only-when-changes. The demo alert on the dev instance was moved to the new mode.

Versions are judged by effect on the reader, not by how much code moved: major when a page is gone, moved or now means something different; minor for a new page or a new answer on an existing one; patch for a fix to something the page already claimed to do. The same notes are in CHANGELOG.md, generated from the same source so the two cannot disagree.