Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 17:36:15.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Deliveries on production 65 what went out for production · all time · only production, last 1 hour · for you (0 unread)

Every message X-ray produced for an alert — delivered in X-ray, sent to a channel, rendered in dry-run, held or dropped — and why. One row per message: a group of a hundred findings is one row. Every row carries its message number (#N), the same one printed on the card and the bell row — click it to see the message on its alert's page, with what it was about. external channels dry-run

WhenAlertMessageDelivered toEventWhy
#52134
2026-11-17 10:41:25 UTC
channel test info dashrsvc Neo4j load 61 q/min vs a normal 0 q/min ± 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "reopened",
 "finding": {
  "id": 1088,
  "fingerprint": "86ecf8a5e876f925b7e2",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "dashrsvc",
  "title": "dashrsvc Neo4j load 61 q/min vs a normal 0 q/min \u00b1 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 61.0,
   "expected_median": 0.2,
   "expected_mad": 0.29652,
   "z": 205.04519088088492,
   "rel": 303.99999999999994,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 10:00 UTC",
   "bucket": 1794910500,
   "window": [
    1794909600,
    1794910800
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/dashrsvc?cluster=production&from=1794888000&to=1794910800"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794888000&to=1794910800"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794911420,
   "age_seconds": 639,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 7.0,
  "status": "open",
  "first_seen": 1788850771,
  "last_seen": 1794912041,
  "resolved_at": null,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 95,
  "narrative": "dashrsvc's Neo4j load is 50 q/min on production, above a normal of 1 q/min \u00b1 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 33.2, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794912041,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 6061288,
  "stale_s": 18
 },
 "title": "dashrsvc Neo4j load 61 q/min vs a normal 0 q/min \u00b1 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52126
2026-11-17 10:19:19 UTC
channel test warning Any error → notify me (master): 2 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in schedsvc (tilda) on master; Error in tilda on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in schedsvc (tilda) on master; Error in tilda on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794910789,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 52126,
 "message_url": "https://xray.tilda.pizza/alerts/messages/52126"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#52113
2026-11-17 10:00:20 UTC
channel test info virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min ± 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 5498,
  "fingerprint": "9e46498162c3f35b85bf",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "virtual-assistant",
  "title": "virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min \u00b1 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 300.4,
   "expected_median": 2.2,
   "expected_mad": 1.4826,
   "z": 201.1331444759207,
   "rel": 135.54545454545453,
   "buckets_hit": 4,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 09:00 UTC",
   "bucket": 1794906900,
   "window": [
    1794906000,
    1794907200
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/virtual-assistant?cluster=production&from=1794884400&to=1794907200"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794884400&to=1794907200"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794907808,
   "age_seconds": 598,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 12.3,
  "status": "resolved",
  "first_seen": 1792763357,
  "last_seen": 1794908390,
  "resolved_at": 1794909577,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 259,
  "narrative": "virtual-assistant's Neo4j load is 300 q/min on production, above a normal of 2 q/min \u00b1 1 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 201.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794908390,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 2146236,
  "stale_s": 1203
 },
 "title": "virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min \u00b1 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52108
2026-11-17 09:23:11 UTC
channel test info execsvc Neo4j load 83 q/min vs a normal 0 q/min ± 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 153,
  "fingerprint": "2bd5ad0e4658eb2d6436",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "execsvc",
  "title": "execsvc Neo4j load 83 q/min vs a normal 0 q/min \u00b1 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 82.8,
   "expected_median": 0.0,
   "expected_mad": 20.0,
   "z": 4.14,
   "rel": Infinity,
   "buckets_hit": 4,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 08:00 UTC",
   "bucket": 1794903900,
   "window": [
    1794903000,
    1794904200
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/execsvc?cluster=production&from=1794881400&to=1794904200"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794881400&to=1794904200"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794904797,
   "age_seconds": 973,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 12.4,
  "status": "resolved",
  "first_seen": 1787759210,
  "last_seen": 1794905754,
  "resolved_at": 1794907325,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 371,
  "narrative": "execsvc's Neo4j load is 83 q/min on production, above a normal of 0 q/min \u00b1 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 4.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794905754,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 7148131,
  "stale_s": 1587
 },
 "title": "execsvc Neo4j load 83 q/min vs a normal 0 q/min \u00b1 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52107
2026-11-17 09:23:11 UTC
channel test info docsvc Neo4j load 471 q/min vs a normal 48 q/min ± 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 649,
  "fingerprint": "5f1819d07c793bfe2483",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "docsvc",
  "title": "docsvc Neo4j load 471 q/min vs a normal 48 q/min \u00b1 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 470.6,
   "expected_median": 47.9,
   "expected_mad": 70.72002,
   "z": 5.977091069827186,
   "rel": 8.82463465553236,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 08:00 UTC",
   "bucket": 1794903900,
   "window": [
    1794903000,
    1794904200
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/docsvc?cluster=production&from=1794881400&to=1794904200"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794881400&to=1794904200"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794904797,
   "age_seconds": 973,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 21.6,
  "status": "resolved",
  "first_seen": 1788412396,
  "last_seen": 1794905754,
  "resolved_at": 1794907325,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 1029,
  "narrative": "docsvc's Neo4j load is 471 q/min on production, above a normal of 48 q/min \u00b1 71 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794902649,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 6494945,
  "stale_s": 1587
 },
 "title": "docsvc Neo4j load 471 q/min vs a normal 48 q/min \u00b1 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52105
2026-11-17 09:19:08 UTC
channel test warning Any error → notify me (master): 1 more message held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794907178,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 52105,
 "message_url": "https://xray.tilda.pizza/alerts/messages/52105"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#52092
2026-11-17 08:57:00 UTC
channel test info production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min ± 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 155,
  "fingerprint": "34db0855651c5c451b99",
  "kind": "anomaly.traffic",
  "severity": "info",
  "cluster": "production",
  "subject": "*",
  "title": "production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min \u00b1 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "api_rpm",
   "unit": "req/min",
   "value": 46.0,
   "expected_median": 4.2,
   "expected_mad": 6.22692,
   "z": 6.712788987171828,
   "rel": 9.95238095238095,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 07:00 UTC",
   "bucket": 1794902100,
   "window": [
    1794901500,
    1794902700
   ],
   "recent_deploys": [],
   "discriminator": "api_rpm"
  },
  "evidence": [
   {
    "label": "Cluster charts",
    "href": "/charts?cluster=production&from=1794879900&to=1794902700"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794879900&to=1794902700"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794903342,
   "age_seconds": 677,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 40.0,
  "status": "resolved",
  "first_seen": 1787759909,
  "last_seen": 1794904001,
  "resolved_at": 1794905754,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 799,
  "narrative": "the whole production cluster's request rate is 46.0 req/min on production, above a normal of 4.2 req/min \u00b1 6.2 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.7, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "api_rpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794902649,
  "chat_thread": null,
  "kind_label": "Traffic anomaly",
  "group": "Anomalies",
  "age_s": 7145861,
  "stale_s": 1769
 },
 "title": "production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min \u00b1 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.traffic"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52091
2026-11-17 08:57:00 UTC
channel test info docproc Neo4j load 118 q/min vs a normal 14 q/min ± 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 648,
  "fingerprint": "e44edcffd892b971d1a5",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "docproc",
  "title": "docproc Neo4j load 118 q/min vs a normal 14 q/min \u00b1 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 118.0,
   "expected_median": 13.5,
   "expected_mad": 4.89258,
   "z": 21.358874050092183,
   "rel": 7.7407407407407405,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 07:00 UTC",
   "bucket": 1794902100,
   "window": [
    1794901500,
    1794902700
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/docproc?cluster=production&from=1794879900&to=1794902700"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794879900&to=1794902700"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794903342,
   "age_seconds": 676,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 13.6,
  "status": "resolved",
  "first_seen": 1788412396,
  "last_seen": 1794904001,
  "resolved_at": 1794905754,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 1235,
  "narrative": "docproc's Neo4j load is 118 q/min on production, above a normal of 14 q/min \u00b1 5 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 21.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794904001,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 6493374,
  "stale_s": 1769
 },
 "title": "docproc Neo4j load 118 q/min vs a normal 14 q/min \u00b1 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52090
2026-11-17 08:57:00 UTC
channel test info production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min ± 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 49,
  "fingerprint": "7a1fc9a9706dadb35d51",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "*",
  "title": "production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min \u00b1 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 302.0,
   "expected_median": 57.2,
   "expected_mad": 36.17544,
   "z": 6.767022045896332,
   "rel": 4.27972027972028,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 07:00 UTC",
   "bucket": 1794902100,
   "window": [
    1794901500,
    1794902700
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Cluster charts",
    "href": "/charts?cluster=production&from=1794879900&to=1794902700"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794879900&to=1794902700"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794903342,
   "age_seconds": 677,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 29.5,
  "status": "resolved",
  "first_seen": 1787697056,
  "last_seen": 1794904001,
  "resolved_at": 1794905754,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 1081,
  "narrative": "the whole production cluster's Neo4j load is 302 q/min on production, above a normal of 57 q/min \u00b1 36 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794904001,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 7208714,
  "stale_s": 1769
 },
 "title": "production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min \u00b1 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52089
2026-11-17 08:57:00 UTC
channel test info docsvc request rate 20.2 req/min vs a normal 1.0 req/min ± 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 223,
  "fingerprint": "ee625f6aa4ec4f81cd50",
  "kind": "anomaly.traffic",
  "severity": "info",
  "cluster": "production",
  "subject": "docsvc",
  "title": "docsvc request rate 20.2 req/min vs a normal 1.0 req/min \u00b1 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "api_rpm",
   "unit": "req/min",
   "value": 20.2,
   "expected_median": 1.0,
   "expected_mad": 1.4826,
   "z": 12.950222581950628,
   "rel": 19.2,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 07:00 UTC",
   "bucket": 1794902100,
   "window": [
    1794901500,
    1794902700
   ],
   "recent_deploys": [],
   "discriminator": "api_rpm"
  },
  "evidence": [
   {
    "label": "Service",
    "href": "/service/docsvc?cluster=production&from=1794879900&to=1794902700"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794879900&to=1794902700"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794903342,
   "age_seconds": 676,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 26.8,
  "status": "resolved",
  "first_seen": 1787816487,
  "last_seen": 1794904001,
  "resolved_at": 1794905754,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 574,
  "narrative": "docsvc's request rate is 20.2 req/min on production, above a normal of 1.0 req/min \u00b1 1.5 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 13.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "api_rpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794902649,
  "chat_thread": null,
  "kind_label": "Traffic anomaly",
  "group": "Anomalies",
  "age_s": 7089283,
  "stale_s": 1769
 },
 "title": "docsvc request rate 20.2 req/min vs a normal 1.0 req/min \u00b1 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.traffic"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52049
2026-11-17 07:35:33 UTC
channel test medium [MEDIUM] execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
show the card
🟡 MEDIUM · execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
production · Background job failing · medium
The background job execsvc.auto-query-reeval-sweep on production failed 11 runs recently (358 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The facts
job: execsvc.auto-query-reeval-sweep
service: execsvc
failed_runs: 11
error_lines: 358
sample: no result from branch evaluation for branch, protocolId=77bcfccb-cc59-4aea-bff8-0991eaa0c788 activityRefId=76d84261-cdb3-4d29-aba0-234a24c8652d
error_signature: no result from branch evaluation for branch, protocolId=? activityRefId=?
live · observed 11m ago · 5× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52049
#in-app delivered
#prod-notifications sent
opened in-app: 1 inbox row(s) for [email protected]
ok
#52047
2026-11-17 07:35:31 UTC
channel test info execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
show the card
✅ resolved: execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
production · Background job failing · medium
Open for 9h, observed 7 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 3h ago · 7× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52047
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#52045
2026-11-17 07:35:31 UTC
channel test info pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
show the card
✅ resolved: pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
production · Background job failing · medium
Open for 70d, observed 570 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 4h ago · 570× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52045
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#52034
2026-11-17 06:43:15 UTC
channel test warning onboardingsvc is burning its error budget at 0.0× (1 h) / 13.3× (6 h) against a 99.50% objective
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 1530,
  "fingerprint": "9e19b05a63df8239d9bc",
  "kind": "budget.burning",
  "severity": "warning",
  "cluster": "production",
  "subject": "onboardingsvc",
  "title": "onboardingsvc is burning its error budget at 0.0\u00d7 (1 h) / 13.3\u00d7 (6 h) against a 99.50% objective",
  "facts": {
   "objective": 99.5,
   "objective_source": "derived",
   "achieved": 93.33333333333333,
   "calls": 225.0,
   "bad": 15.0,
   "c5xx": 15.0,
   "budget": 1.125,
   "spent": 13.333333333333334,
   "remaining_pct": 0.0,
   "burn_1h": 0.0,
   "burn_6h": 13.333333333333332,
   "verdict": "burning",
   "p95_ms": 165.0,
   "window": [
    1794874505,
    1794896105
   ],
   "recent_deploys": [],
   "discriminator": ""
  },
  "evidence": [
   {
    "label": "Error budgets",
    "href": "/budgets?cluster=production&from=1794874505&to=1794896105"
   },
   {
    "label": "onboardingsvc service page",
    "href": "/service/onboardingsvc?cluster=production&from=1794874505&to=1794896105"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794895685,
   "age_seconds": 424,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 51.6,
  "status": "resolved",
  "first_seen": 1789163502,
  "last_seen": 1794896105,
  "resolved_at": 1794897755,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 199,
  "narrative": "onboardingsvc is spending its error budget 0.0\u00d7 faster than the objective allows over the last hour (13.3\u00d7 over six hours). The objective is 99.5%, derived \u2014 a baseline, not a promise. 15 bad events against 225 calls in the window.",
  "narrative_kind": "template",
  "discriminator": "",
  "level": "low",
  "told_level": "medium",
  "episode_at": 1794892665,
  "chat_thread": "xray-alert-11-748fd5a928-9e19b05a63df8239d9bc-1793224828",
  "kind_label": "Error budget burning",
  "group": "Reliability",
  "age_s": 5734258,
  "stale_s": 1655
 },
 "title": "onboardingsvc is burning its error budget at 0.0\u00d7 (1 h) / 13.3\u00d7 (6 h) against a 99.50% objective",
 "severity": "warning",
 "cluster": "production",
 "kind": "budget.burning"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52020
2026-11-17 05:35:00 UTC
channel test info Tilda namespace findings (master): all clear — 1 resolved
show the card
🔵 Tilda namespace findings (master): all clear — 1 resolved
alert: Tilda namespace findings (master) · changed
1 resolved
✅ execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx)
message 2 for this group · last one 48m ago · X-ray message #52020
#dev-notifications sent
#in-app delivered
group ok
in-app: 52 inbox row(s) for 52 people on X-ray
#52018
2026-11-17 05:30:00 UTC
channel test info execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
show the card
✅ resolved: execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
production · Background job failing · medium
Open for 7h, observed 7 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 1h ago · 7× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52018
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#52016
2026-11-17 05:30:00 UTC
channel test info pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
show the card
✅ resolved: pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
production · Background job failing · medium
Open for 70d, observed 570 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 2h ago · 570× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52016
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#52011
2026-11-17 05:17:58 UTC
channel test info production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min ± 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 49,
  "fingerprint": "7a1fc9a9706dadb35d51",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "*",
  "title": "production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min \u00b1 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 570.2,
   "expected_median": 39.4,
   "expected_mad": 10.08168,
   "z": 52.64995516620246,
   "rel": 13.472081218274115,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 04:00 UTC",
   "bucket": 1794890400,
   "window": [
    1794889500,
    1794890700
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Cluster charts",
    "href": "/charts?cluster=production&from=1794867900&to=1794890700"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794867900&to=1794890700"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794891309,
   "age_seconds": 352,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 29.5,
  "status": "resolved",
  "first_seen": 1787697056,
  "last_seen": 1794891647,
  "resolved_at": 1794892665,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 1080,
  "narrative": "the whole production cluster's Neo4j load is 570 q/min on production, above a normal of 39 q/min \u00b1 10 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 52.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794891647,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 7195621,
  "stale_s": 1030
 },
 "title": "production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min \u00b1 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#52000
2026-11-17 04:27:47 UTC
channel test info production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min ± 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)
show the payload
{
 "event": "resolved",
 "finding": {
  "id": 49,
  "fingerprint": "7a1fc9a9706dadb35d51",
  "kind": "anomaly.neo4j",
  "severity": "info",
  "cluster": "production",
  "subject": "*",
  "title": "production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min \u00b1 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)",
  "facts": {
   "measure": "neo_qpm",
   "unit": "q/min",
   "value": 96.2,
   "expected_median": 38.2,
   "expected_mad": 9.04386,
   "z": 6.413190827810249,
   "rel": 1.5183246073298429,
   "buckets_hit": 3,
   "n": 48,
   "weeks": 4,
   "slot": "Tue 03:00 UTC",
   "bucket": 1794887700,
   "window": [
    1794886800,
    1794888000
   ],
   "recent_deploys": [],
   "discriminator": "neo_qpm"
  },
  "evidence": [
   {
    "label": "Cluster charts",
    "href": "/charts?cluster=production&from=1794865200&to=1794888000"
   },
   {
    "label": "Fleet vs normal",
    "href": "/fleet?cluster=production&from=1794865200&to=1794888000"
   }
  ],
  "provenance": {
   "source": "rollup",
   "collected_at": 1794888450,
   "age_seconds": 76,
   "stale": false,
   "shown": null,
   "total": null,
   "truncated": false,
   "coverage_from": null,
   "retain_days": null,
   "note": null
  },
  "impact": 29.5,
  "status": "resolved",
  "first_seen": 1787697056,
  "last_seen": 1794888514,
  "resolved_at": 1794889645,
  "acked_by": null,
  "acked_at": null,
  "snoozed_until": null,
  "occurrences": 1079,
  "narrative": "the whole production cluster's Neo4j load is 96 q/min on production, above a normal of 38 q/min \u00b1 9 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
  "narrative_kind": "template",
  "discriminator": "neo_qpm",
  "level": "low",
  "told_level": "low",
  "episode_at": 1794882200,
  "chat_thread": null,
  "kind_label": "Neo4j load anomaly",
  "group": "Anomalies",
  "age_s": 7192602,
  "stale_s": 1144
 },
 "title": "production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min \u00b1 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)",
 "severity": "info",
 "cluster": "production",
 "kind": "anomaly.neo4j"
}
#in-app,prod-notifications queued
digest-item waiting for the digest
#51974
2026-11-17 03:30:36 UTC
channel test X-ray daily — 10 open (0 critical)
show the card
X-ray daily — 10 open (0 critical)
+0 opened · −76 resolved since the last digest
production
🔵 production (whole cluster) Neo4j load 117 q/min vs a normal 38 q/min ± 10 q/min for Tue 02:00 UTC (4 of the last 4 checks)
🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s)
🔵 docproc: p99 latency (ms) 9.03 s — 4.1× the fleet's usual 2.18 s
🔵 execsvc: Neo4j queries per request 72.7 — 7.2× the fleet's usual 10.1
🔵 dashql: p95 latency (ms) 4.27 s — 9.5× the fleet's usual 451 ms
🔵 virtual-assistant: p95 latency (ms) 1.57 s — 3.5× the fleet's usual 451 ms
🔵 tusker: Neo4j queries per request 24.5 — 4.1× the fleet's usual 6.0
🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s
and 2 more
95 warning-level events were held for this digest · X-ray message #51974
#prod-notifications sent
#in-app delivered
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
digest ok
in-app: 1 inbox row(s) for [email protected]
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
#51953
2026-11-17 01:29:53 UTC
channel test high Production — critical & high: 1 finding firing (1 escalated)
show the card
🟠 HIGH · escalated · execsvc.auto-query-reeval: 20 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
production · Background job failing · high
The background job execsvc.auto-query-reeval on production failed 20 runs recently (211 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The facts
job: execsvc.auto-query-reeval
service: execsvc
failed_runs: 20
error_lines: 211
sample: participant activity 0294d67b-775d-4580-a46a-4b3085691136: auto query cee26d92-2b6c-4e45-a32f-b8758d9ff83a on question 522e2ac3-5aeb-4dd4-ac75-a4fc48384c13: no result from branch evaluation for branch
error_signature: participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, protocolId=? activityRe
live · observed 41m ago · 4× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51953
#prod-notifications sent
#in-app delivered
escalated ok
in-app: 1 inbox row(s) for [email protected]
#51936
2026-11-16 23:34:32 UTC
channel test medium [MEDIUM] execsvc.auto-query-reeval: 16 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
show the card
🟡 MEDIUM · execsvc.auto-query-reeval: 16 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
production · Background job failing · medium
The background job execsvc.auto-query-reeval on production failed 16 runs recently (533 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The facts
job: execsvc.auto-query-reeval
service: execsvc
failed_runs: 16
error_lines: 533
sample: no result from branch evaluation for branch, protocolId=77bcfccb-cc59-4aea-bff8-0991eaa0c788 activityRefId=4b4f0a72-5853-4c33-927d-a45776e920bb
error_signature: no result from branch evaluation for branch, protocolId=? activityRefId=?
live · observed 53m ago · 2× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51936
#prod-notifications sent
#in-app delivered
opened ok
in-app: 1 inbox row(s) for [email protected]
#51934
2026-11-16 23:34:32 UTC
channel test info rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
show the card
✅ resolved: rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
production · Background job failing · medium
Open for 20d, observed 12 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 8h ago · 12× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51934
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#51911
2026-11-16 21:29:12 UTC
channel test info execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
show the card
✅ resolved: execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
production · Error never seen before · high
Open for 4h, observed 3 times. A healthy detector run no longer sees it, or someone resolved it.
live · observed 4h ago · 3× so far · Cloud Logging, last hour, grouped by (service, caller, shape) · X-ray message #51911
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#51845
2026-11-16 17:28:30 UTC
channel test high Production — critical & high: 2 findings firing (1 new, 1 escalated) · 473 events in the last hour
show the card
🟠 Production — critical & high: 2 findings firing (1 new, 1 escalated) · 473 events in the last hour
alert: Production — critical & high · changed
2 firing
🟠 high · production · new · ×473 in the last hour: execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
🟠 high · production · escalated · observed 561× since 70d ago: pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested
message 66 for this group · last one 4h ago · X-ray message #51845
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#51802
2026-11-16 16:16:16 UTC
channel test warning Any error → notify me (master): 5 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in tilda on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in tilda on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794845806,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51802,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51802"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51796
2026-11-16 15:34:10 UTC
channel test medium [MEDIUM] rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
show the card
🟡 MEDIUM · reopened · rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
production · Background job failing · medium
The background job rag-agent.upload on production failed 5 runs recently (53 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The facts
job: rag-agent.upload
service: rag-agent
failed_runs: 5
error_lines: 53
sample: OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open stream: Failed to open stream: Failed to get PDF page count: Failed to open stream: Failed to open st
error_signature: OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open stream: Failed to ope
live · observed 16m ago · 12× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51796
#prod-notifications sent
#in-app delivered
reopened ok
in-app: 1 inbox row(s) for [email protected]
#51783
2026-11-16 15:16:08 UTC
channel test warning Any error → notify me (master): 2 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in chat-agent (tilda) on master; Error in docsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in chat-agent (tilda) on master; Error in docsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794842198,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51783,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51783"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51736
2026-11-16 14:15:55 UTC
channel test warning Any error → notify me (master): 5 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794838585,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51736,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51736"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51730
2026-11-16 13:28:15 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 26× since 18d ago: Neo4j production, database neo4j: 15 Cypher queries failed in the last 15 min
message 65 for this group · last one 4h ago · X-ray message #51730
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#51710
2026-11-16 13:15:10 UTC
channel test warning Any error → notify me (master): 6 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 6 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 6 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794834940,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51710,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51710"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51652
2026-11-16 12:15:02 UTC
channel test warning Any error → notify me (master): 1 more message held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794831332,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51652,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51652"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51565
2026-11-16 09:27:31 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 64 for this group · last one 4h ago · X-ray message #51565
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#51564
2026-11-16 09:26:30 UTC
channel test —
show the payload
{
 "reason": "unchanged",
 "pending": 6
}
#in-app,prod-notifications suppressed
group 6 event(s) changed nothing about the group since the last message
#51529
2026-11-16 08:24:39 UTC
channel test warning Any error → notify me (master): 10 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 10 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 10 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794817510,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51529,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51529"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51500
2026-11-16 07:33:30 UTC
channel test medium [MEDIUM] pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ?
show the card
🟡 MEDIUM · reopened · pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ?
production · Background job failing · medium
The background job pubsub.dashboards-production-dashsub on production failed 5 runs recently (27 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts.
The facts
job: pubsub.dashboards-production-dashsub
service: dashrsvc
failed_runs: 5
error_lines: 27
sample: no data found for label RecommendationByDocument, id 0d2b5f93-69f7-4725-a10b-69b19e08e475
error_signature: no data found for label RecommendationByDocument, id ?
live · observed 6m ago · 553× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51500
#in-app delivered
#prod-notifications sent
reopened in-app: 1 inbox row(s) for [email protected]
ok
#51490
2026-11-16 07:24:28 UTC
channel test warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in docsvc (tilda) on master; Error in docsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in docsvc (tilda) on master; Error in docsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794813898,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51490,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51490"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51435
2026-11-16 05:55:19 UTC
channel test —
show the payload
{
 "reason": "unchanged",
 "pending": 1
}
#in-app,prod-notifications suppressed
group 1 event(s) changed nothing about the group since the last message
#51427
2026-11-16 05:26:13 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 63 for this group · last one 4h ago · X-ray message #51427
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#51426
2026-11-16 05:25:13 UTC
channel test —
show the payload
{
 "reason": "unchanged",
 "pending": 2
}
#in-app,prod-notifications suppressed
group 2 event(s) changed nothing about the group since the last message
#51414
2026-11-16 03:30:03 UTC
channel test X-ray daily — 10 open (0 critical)
show the card
X-ray daily — 10 open (0 critical)
+0 opened · −3 resolved since the last digest
production
🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s)
🔵 execsvc p95 3.88 s vs a normal
🔵 dashql: p99 latency (ms) 3.88 s — 4.2× the fleet's usual 913 ms
🔵 docproc: Neo4j queries per request 74.7 — 16.3× the fleet's usual 4.6
🔵 execsvc: Neo4j queries per request 11.9 — 2.2× the fleet's usual 5.4
🔵 virtual-assistant: Neo4j queries per request 78.2 — 10.7× the fleet's usual 7.3
🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s
🔵 tusker: Neo4j queries per request 109.7 — 6.3× the fleet's usual 17.4
and 2 more
4 warning-level events were held for this digest · X-ray message #51414
#prod-notifications sent
#in-app delivered
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
digest ok
in-app: 1 inbox row(s) for [email protected]
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
#51398
2026-11-16 01:41:21 UTC
channel test warning Any error → notify me (master): 19 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 19 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 19 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in tilda on master; Error in docproc (tilda) on master; Error in ontologysvc (tilda) on master; Error in tilda on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in tilda on master; Error in docproc (tilda) on master; Error in ontologysvc (tilda) on master; Error in tilda on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794793311,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51398,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51398"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51388
2026-11-16 01:24:20 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 62 for this group · last one 4h ago · X-ray message #51388
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#51339
2026-11-16 00:41:18 UTC
channel test warning Any error → notify me (master): 78 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 78 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 78 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794789708,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51339,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51339"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51144
2026-11-15 23:41:15 UTC
channel test warning Any error → notify me (master): 35 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 35 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 35 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794786105,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51144,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51144"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#51044
2026-11-15 22:41:10 UTC
channel test warning Any error → notify me (master): 38 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 38 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 38 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794782500,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 51044,
 "message_url": "https://xray.tilda.pizza/alerts/messages/51044"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50932
2026-11-15 21:41:02 UTC
channel test warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794778892,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 50932,
 "message_url": "https://xray.tilda.pizza/alerts/messages/50932"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50902
2026-11-15 21:24:00 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 61 for this group · last one 4h ago · X-ray message #50902
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50862
2026-11-15 17:23:43 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 18d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 18d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 60 for this group · last one 4h ago · X-ray message #50862
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50846
2026-11-15 14:49:13 UTC
channel test warning Any error → notify me (master): 15 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 15 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 15 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in execsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in execsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794754183,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 50846,
 "message_url": "https://xray.tilda.pizza/alerts/messages/50846"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50820
2026-11-15 13:54:55 UTC
channel test info HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
show the card
✅ resolved: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
production · Autoscaler blind · medium
Open for 43d, observed 177 times. A healthy detector run no longer sees it, or someone resolved it.
x-ray · observed 6h ago · 177× so far · Kubernetes events (logs/events in BigQuery), last 45 min · X-ray message #50820
#prod-notifications sent
#in-app delivered
resolved ok
in-app: 1 inbox row(s) for [email protected]
#50810
2026-11-15 13:48:53 UTC
channel test warning Any error → notify me (master): 7 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 7 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 7 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794750563,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 50810,
 "message_url": "https://xray.tilda.pizza/alerts/messages/50810"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50776
2026-11-15 13:22:50 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 59 for this group · last one 4h ago · X-ray message #50776
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50744
2026-11-15 09:21:57 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 58 for this group · last one 4h ago · X-ray message #50744
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50743
2026-11-15 09:20:57 UTC
channel test —
show the payload
{
 "reason": "unchanged",
 "pending": 2
}
#in-app,prod-notifications suppressed
group 2 event(s) changed nothing about the group since the last message
#50735
2026-11-15 08:42:54 UTC
channel test warning Any error → notify me (master): 23 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 23 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 23 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794732204,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 50735,
 "message_url": "https://xray.tilda.pizza/alerts/messages/50735"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50677
2026-11-15 07:42:12 UTC
channel test warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload
{
 "title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
 "text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in exportsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
 "body": "The alert allows 20 an hour. Most recent held: Error in exportsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
 "url": "https://xray.tilda.pizza/alerts/8",
 "severity": "warning",
 "_scope": {},
 "_job": {
  "finding": null,
  "force": true,
  "retry_at": 1794728562,
  "route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
  "digest_items": []
 },
 "message_id": 50677,
 "message_url": "https://xray.tilda.pizza/alerts/messages/50677"
}
#dev-notifications sent
#in-app delivered
rollup ok
in-app: 1 inbox row(s) for [email protected]
#50613
2026-11-15 05:54:29 UTC
channel test medium [MEDIUM] HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
show the card
🟡 MEDIUM · reopened · HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
production · Autoscaler blind · medium
Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 16, last_message No recommendation.
The facts
namespace: tilda
object: HorizontalPodAutoscaler/ontologysvc-hpa
failures_2h: 16
last_message: No recommendation
discriminator:
x-ray · observed 6m ago · 163× so far · Kubernetes events (logs/events in BigQuery), last 45 min · X-ray message #50613
#prod-notifications sent
#in-app delivered
reopened ok
in-app: 1 inbox row(s) for [email protected]
#50611
2026-11-15 05:20:28 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 57 for this group · last one 4h ago · X-ray message #50611
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50597
2026-11-15 03:30:16 UTC
channel test X-ray daily — 10 open (0 critical)
show the card
X-ray daily — 10 open (0 critical)
+0 opened · −5 resolved since the last digest
production
🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s)
🔵 execsvc p95 3.88 s vs a normal
🔵 dashql: p99 latency (ms) 3.88 s — 4.2× the fleet's usual 913 ms
🔵 docproc: Neo4j queries per request 74.7 — 16.3× the fleet's usual 4.6
🔵 execsvc: Neo4j queries per request 11.9 — 2.2× the fleet's usual 5.4
🔵 virtual-assistant: Neo4j queries per request 78.2 — 10.7× the fleet's usual 7.3
🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s
🔵 tusker: Neo4j queries per request 109.7 — 6.3× the fleet's usual 17.4
and 2 more
6 warning-level events were held for this digest · X-ray message #50597
#prod-notifications sent
#in-app delivered
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
#in-app,prod-notifications sent
digest ok
in-app: 1 inbox row(s) for [email protected]
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
waiting for the digest
#50565
2026-11-15 01:19:48 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 56 for this group · last one 4h ago · X-ray message #50565
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50536
2026-11-14 21:19:15 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 55 for this group · last one 4h ago · X-ray message #50536
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50523
2026-11-14 18:42:46 UTC
channel test medium Production — medium: still open — 4 findings, oldest 17d · 27 events in the last hour
show the card
🔔 Production — medium: still open — 4 findings, oldest 17d · 27 events in the last hour
alert: Production — medium · repeat
4 firing
🟡 medium · production · ×15 in the last hour: execsvc: a new error shape, 15× in the last hour, never seen in 90 days (failed to process db result — Neo4jError: Neo.TransientError.Transacti)
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
🟡 medium · production · ×6 in the last hour: rag-agent: a new error shape, 6× in the last hour, never seen in 90 days (Chunked PDF processing failed: Failed to get PDF page count: Failed to)
🟡 medium · production · ×6 in the last hour: rag-agent: a new error shape, 6× in the last hour, never seen in 90 days (OCR processing failed: Chunked PDF processing failed: Failed to get PD)
message 65 for this group · last one 12h ago · X-ray message #50523
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50508
2026-11-14 17:18:38 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 17d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 17d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 54 for this group · last one 4h ago · X-ray message #50508
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]
#50465
2026-11-14 13:18:14 UTC
channel test medium Production — critical & high: still open — 1 finding, oldest 16d
show the card
🔔 Production — critical & high: still open — 1 finding, oldest 16d
alert: Production — critical & high · repeat
1 firing
🟡 medium · production · observed 25× since 16d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min
message 53 for this group · last one 4h ago · X-ray message #50465
#prod-notifications sent
#in-app delivered
group ok
in-app: 1 inbox row(s) for [email protected]