Detectors 31 runs on test26
The alerts X-ray ships with — the detectors that open findings for everyone. Their logic is code; their numbers (thresholds, windows, severity cut-offs), how often they run, and whether they run at all are yours to change here, like any alert. A change takes effect on the next run; reset returns to the shipped defaults.
Every setting has a system default: ↺ on a setting, a detector's reset button, or all of them here.
| Detector | Opens findings of kind | Runs | Last run | Settings | |
|---|---|---|---|---|---|
| outliers Services unlike their peers on 5xx rate, p95, p99 or Neo4j queries per request — one row per service; a warning only on the measures you choose (by default the 5xx rate), informational otherwise. |
outlier.service |
every 5 min per cluster |
never | 9 numbers · defaults | |
| budgets Error budgets burning faster than the objective allows (1 h and 6 h burn rates). |
budget.burning |
every 5 min per cluster |
never | 1 number · defaults | |
| shapes Cypher shapes whose p95 in the second half of a 6 h window is ≥1.5× the first half — and is at least 1 s now. Spikes under a second are ignored as noise, whatever the ratio: 2 ms → 43 ms is 21× and means nothing to anyone. |
shape.degrading |
every 15 min per cluster |
never | 7 numbers · defaults | |
| neo4j_errors live A Neo4j database failing Cypher executions, from Neo4j's own metric (db_query_execution_failure_total, via Managed Prometheus) -- the failures the Neo4j tab charted but nothing raised (2026-10-23). Load and plan speed are anomaly.neo4j and shape.*; this is errors. |
neo4j.errors |
every 5 min fleet-wide |
never | 3 numbers · defaults | |
| endpoint_4xx_spike One finding per endpoint for its 4xx: answering far more 4xx than its own usual rate, or rejecting nearly every call. Permission denied (401/403) is critical; other 4xx a warning. A permissions change (403), broken or expired auth (401), a removed route or contract break (404). endpoint_failing only counts 5xx, and its 4xx signal needs ~every call rejected; a 403 storm on a busy endpoint was invisible (2026-10-23: docsvc folders 96 and onboardingsvc orgs 102 4xx in five minutes, no finding). The usual rate is the endpoint's own average over the last seven days (hourly rollup), so a chronically noisy endpoint is not flagged for being itself. |
endpoint.4xx_spike |
every 5 min per cluster |
never | 6 numbers · defaults | |
| endpoint_failing One endpoint of a service failing most of its calls (5xx) over the last 15 minutes — the unit the observation-window alerts count and resolve per endpoint (15-OBSERVATION-WINDOW-NOTIFICATIONS-RESEARCH.md §3.5). 4xx are shown beside it, not counted as failing: a 404 is the caller's problem, a 500 is ours — but an endpoint that rejects almost EVERY call with a 4xx is its own INFO signal (a broken client or contract), separate from the 5xx alert. |
endpoint.failing endpoint.rejecting |
every 5 min per cluster |
never | 6 numbers · defaults | |
| hand_changes kubectl/Helm/console writes that bypassed ArgoCD in the last 24 h — production only, only in the tilda and neo4j-system namespaces, and never an image change. |
change.hand |
every 5 min per cluster |
never | 1 number · defaults | |
| fanout Services averaging ≥25 foreground Neo4j queries per API request over the last hour. |
n1.fanout |
every 15 min per cluster |
never | 3 numbers · defaults | |
| ingest_stale X-ray's own ingest has not succeeded for this cluster in 45 min (warning) or 3 h (critical). |
ingest.stale |
every 5 min per cluster |
never | 2 numbers · defaults | |
| collector_stale A push collector that has posted before has gone quiet for 3× its cadence. |
collector.stale |
every 15 min fleet-wide |
never | 2 numbers · defaults | |
| baseline_refresh Rebuilds the hour-of-week baselines for the cluster once a day (no findings of its own). |
none (maintenance) | every 24 h per cluster |
never | no numbers to tune | |
| seasonal Three of the last four complete 5-minute buckets far outside the hour-of-week baseline. A p95 that stays under 1 s is never reported — below a second nobody notices, whatever the deviation. |
anomaly.traffic anomaly.errors anomaly.latency anomaly.neo4j |
every 5 min per cluster |
never | 4 numbers · defaults | |
| budget_forecast The date the 30-day budget runs out, if within 14 days — a constant recent-median scenario (Theil–Sen is only the gate that says the trend is upward at all, not the source of the date). |
budget.forecast |
every 1 h per cluster |
never | 2 numbers · defaults | |
| shape_changepoint For each deploy in the last 6 h, a chi-square test of every busy shape's latency histogram ±60 min around it. |
shape.regressed |
every 15 min per cluster |
never | 4 numbers · defaults | |
| shape_churn New Cypher shapes today vs the median of the prior 14 days — literals inlined into statements show up here. |
shape.churn |
every 1 h per cluster |
never | 2 numbers · defaults | |
| disk_forecast live Theil–Sen on 30 days of volume usage for every Neo4j and Weaviate node; a finding when a watermark is within 30 days. X-ray's own store is NOT covered here — that is obsdb_growth. |
forecast.disk |
every 1 h fleet-wide |
never | 1 number · defaults | |
| obsdb_growth A daily size sample of X-ray's own store — and, on the SQLite PVC, the date it fills 30 GiB (minus a 2 GiB reserve). |
forecast.obsdb |
every 24 h fleet-wide |
never | 1 number · defaults | |
| llm_spend_forecast live Month-end AI spend projection per cluster (BigQuery daily cost, Theil–Sen, weekend-weighted); a finding at >1.2× last month. |
llm.spend |
every 1 h per cluster |
never | 2 numbers · defaults | |
| k8s_events What the cluster itself reports going wrong, from its events (every namespace): pods that cannot start for 10+ minutes (image pull, config error, unschedulable), evictions, kernel OOM kills (including a process inside a running container), Jobs out of retries, ExternalSecrets not syncing, cert-manager or other platform pods down, autoscalers that cannot read their metric, and node problems. Judged on the current state: still being reported, and a normal wait (a pod pending two minutes, then scheduled) is not an incident. |
pod.cannot_start pod.evicted pod.oom job.backoff secret.sync_failing cert.unhealthy platform.unhealthy scaler.failing node.unhealthy |
every 5 min per cluster |
never | no numbers to tune | |
| pod_unstable A workload whose containers restart in place: warning at three restarts in 30 minutes (or one OOM kill); critical once the run has lasted an hour and is still going. Stops being reported after 30 quiet minutes and closes about 15 minutes after that, once the resolve grace passes. Says why — crash loop, OOM, liveness, image — and since when. |
pod.unstable |
every 5 min per cluster |
never | 5 numbers · defaults | |
| service_silent A service that normally serves requests at this hour of the week (its baseline) and has served none for the sustain window. Absence as a signal: a crashed or cut-off service produces no anomaly, only silence. |
service.silent |
every 5 min per cluster |
never | 4 numbers · defaults | |
| cost_anomaly Daily net cost of each project × service × cluster cell against the median/MAD of its previous 28 days; a finding for the last two complete days. |
cost.anomaly |
every 6 h fleet-wide |
never | 3 numbers · defaults | |
| cost_target Every monthly target set on Costs → Forecast against the month-end projection; a finding when the projection is over it. |
cost.target |
every 6 h fleet-wide |
never | 2 numbers · defaults | |
| cost_commitment Commitment fees over the last 30 complete days that funded no usage; a finding per commitment family above the floor. |
cost.commitment |
every 24 h fleet-wide |
never | 2 numbers · defaults | |
| custom_watches Every enabled watch (a condition written in plain English, read into a closed spec) evaluated against the five-minute rollups. |
custom.watch |
every 5 min per cluster |
never | no numbers to tune | |
| query_alerts Every enabled query alert — a request in plain English that the model turned into a query over X-ray's own tables or one of its read-only tools — run and compared with its threshold. |
custom.query |
every 5 min per cluster |
never | no numbers to tune | |
| errors_novel live Error shapes seen for the first time in 90 days, ≥5 times in the last hour (learns silently on day one). |
error.novel |
every 15 min per cluster |
never | 2 numbers · defaults | |
| job_failing Failure lines of named background jobs, clustered into failed runs; a finding when a job keeps failing within the configured window (6 h by default). Severity alone is not the filter — a line carrying an error/exception field or a failure status counts too, because rag-agent.upload logs its failures at WARNING. The complement of errors_novel, which only sees NEW shapes. |
job.failing |
every 1 h per cluster |
never | 4 numbers · defaults | |
| weaviate_stalled live Weaviate tenant queues that are deep and have not drained in the last hour (GMP). |
weaviate.stalled |
every 15 min fleet-wide |
never | 1 number · defaults | |
| llm_unattributed live ≥25% of the last 24 h of LLM spend carries no organisation (BigQuery). |
llm.unattributed |
every 1 h per cluster |
never | 2 numbers · defaults | |
| openai_cost_drift live What the LiteLLM log says a cluster spent on OpenAI against what OpenAI billed its project, over the last complete UTC days. Either direction. Each day's pair is kept in openai_reconciliation_daily as the audit trail. |
llm.cost_drift |
every 6 h per cluster |
never | 5 numbers · defaults |
Two of these evaluate what people wrote themselves: custom_watches (condition alerts) and query_alerts (query alerts). Turning them off silences every alert of that kind.