Deliveries on production 65 what went out for production · all time · only production, last 1 hour · for you (0 unread)
Every message X-ray produced for an alert — delivered in X-ray, sent to a channel, rendered in dry-run, held or dropped — and why. One row per message: a group of a hundred findings is one row. Every row carries its message number (#N), the same one printed on the card and the bell row — click it to see the message on its alert's page, with what it was about. external channels dry-run
| When | Alert | Message | Delivered to | Event | Why |
|---|---|---|---|---|---|
| #52134 2026-11-17 10:41:25 UTC |
channel test | info dashrsvc Neo4j load 61 q/min vs a normal 0 q/min ± 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)
show the payload{
"event": "reopened",
"finding": {
"id": 1088,
"fingerprint": "86ecf8a5e876f925b7e2",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "dashrsvc",
"title": "dashrsvc Neo4j load 61 q/min vs a normal 0 q/min \u00b1 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 61.0,
"expected_median": 0.2,
"expected_mad": 0.29652,
"z": 205.04519088088492,
"rel": 303.99999999999994,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 10:00 UTC",
"bucket": 1794910500,
"window": [
1794909600,
1794910800
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/dashrsvc?cluster=production&from=1794888000&to=1794910800"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794888000&to=1794910800"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794911420,
"age_seconds": 639,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 7.0,
"status": "open",
"first_seen": 1788850771,
"last_seen": 1794912041,
"resolved_at": null,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 95,
"narrative": "dashrsvc's Neo4j load is 50 q/min on production, above a normal of 1 q/min \u00b1 1 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 33.2, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794912041,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 6061288,
"stale_s": 18
},
"title": "dashrsvc Neo4j load 61 q/min vs a normal 0 q/min \u00b1 0 q/min for Tue 10:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52126 2026-11-17 10:19:19 UTC |
channel test | warning Any error → notify me (master): 2 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in schedsvc (tilda) on master; Error in tilda on master",
"body": "The alert allows 20 an hour. Most recent held: Error in schedsvc (tilda) on master; Error in tilda on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794910789,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 52126,
"message_url": "https://xray.tilda.pizza/alerts/messages/52126"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #52113 2026-11-17 10:00:20 UTC |
channel test | info virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min ± 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 5498,
"fingerprint": "9e46498162c3f35b85bf",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "virtual-assistant",
"title": "virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min \u00b1 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 300.4,
"expected_median": 2.2,
"expected_mad": 1.4826,
"z": 201.1331444759207,
"rel": 135.54545454545453,
"buckets_hit": 4,
"n": 48,
"weeks": 4,
"slot": "Tue 09:00 UTC",
"bucket": 1794906900,
"window": [
1794906000,
1794907200
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/virtual-assistant?cluster=production&from=1794884400&to=1794907200"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794884400&to=1794907200"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794907808,
"age_seconds": 598,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 12.3,
"status": "resolved",
"first_seen": 1792763357,
"last_seen": 1794908390,
"resolved_at": 1794909577,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 259,
"narrative": "virtual-assistant's Neo4j load is 300 q/min on production, above a normal of 2 q/min \u00b1 1 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 201.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794908390,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 2146236,
"stale_s": 1203
},
"title": "virtual-assistant Neo4j load 300 q/min vs a normal 2 q/min \u00b1 1 q/min for Tue 09:00 UTC (4 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52108 2026-11-17 09:23:11 UTC |
channel test | info execsvc Neo4j load 83 q/min vs a normal 0 q/min ± 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 153,
"fingerprint": "2bd5ad0e4658eb2d6436",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "execsvc",
"title": "execsvc Neo4j load 83 q/min vs a normal 0 q/min \u00b1 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 82.8,
"expected_median": 0.0,
"expected_mad": 20.0,
"z": 4.14,
"rel": Infinity,
"buckets_hit": 4,
"n": 48,
"weeks": 4,
"slot": "Tue 08:00 UTC",
"bucket": 1794903900,
"window": [
1794903000,
1794904200
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/execsvc?cluster=production&from=1794881400&to=1794904200"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794881400&to=1794904200"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794904797,
"age_seconds": 973,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 12.4,
"status": "resolved",
"first_seen": 1787759210,
"last_seen": 1794905754,
"resolved_at": 1794907325,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 371,
"narrative": "execsvc's Neo4j load is 83 q/min on production, above a normal of 0 q/min \u00b1 20 q/min for this hour of the week (4 of the last 4 five-minute buckets, z = 4.1, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794905754,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 7148131,
"stale_s": 1587
},
"title": "execsvc Neo4j load 83 q/min vs a normal 0 q/min \u00b1 20 q/min for Tue 08:00 UTC (4 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52107 2026-11-17 09:23:11 UTC |
channel test | info docsvc Neo4j load 471 q/min vs a normal 48 q/min ± 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 649,
"fingerprint": "5f1819d07c793bfe2483",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "docsvc",
"title": "docsvc Neo4j load 471 q/min vs a normal 48 q/min \u00b1 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 470.6,
"expected_median": 47.9,
"expected_mad": 70.72002,
"z": 5.977091069827186,
"rel": 8.82463465553236,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 08:00 UTC",
"bucket": 1794903900,
"window": [
1794903000,
1794904200
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/docsvc?cluster=production&from=1794881400&to=1794904200"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794881400&to=1794904200"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794904797,
"age_seconds": 973,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 21.6,
"status": "resolved",
"first_seen": 1788412396,
"last_seen": 1794905754,
"resolved_at": 1794907325,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 1029,
"narrative": "docsvc's Neo4j load is 471 q/min on production, above a normal of 48 q/min \u00b1 71 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794902649,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 6494945,
"stale_s": 1587
},
"title": "docsvc Neo4j load 471 q/min vs a normal 48 q/min \u00b1 71 q/min for Tue 08:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52105 2026-11-17 09:19:08 UTC |
channel test | warning Any error → notify me (master): 1 more message held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794907178,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 52105,
"message_url": "https://xray.tilda.pizza/alerts/messages/52105"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #52092 2026-11-17 08:57:00 UTC |
channel test | info production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min ± 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 155,
"fingerprint": "34db0855651c5c451b99",
"kind": "anomaly.traffic",
"severity": "info",
"cluster": "production",
"subject": "*",
"title": "production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min \u00b1 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "api_rpm",
"unit": "req/min",
"value": 46.0,
"expected_median": 4.2,
"expected_mad": 6.22692,
"z": 6.712788987171828,
"rel": 9.95238095238095,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 07:00 UTC",
"bucket": 1794902100,
"window": [
1794901500,
1794902700
],
"recent_deploys": [],
"discriminator": "api_rpm"
},
"evidence": [
{
"label": "Cluster charts",
"href": "/charts?cluster=production&from=1794879900&to=1794902700"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794879900&to=1794902700"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794903342,
"age_seconds": 677,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 40.0,
"status": "resolved",
"first_seen": 1787759909,
"last_seen": 1794904001,
"resolved_at": 1794905754,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 799,
"narrative": "the whole production cluster's request rate is 46.0 req/min on production, above a normal of 4.2 req/min \u00b1 6.2 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.7, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "api_rpm",
"level": "low",
"told_level": "low",
"episode_at": 1794902649,
"chat_thread": null,
"kind_label": "Traffic anomaly",
"group": "Anomalies",
"age_s": 7145861,
"stale_s": 1769
},
"title": "production (whole cluster) request rate 46.0 req/min vs a normal 4.2 req/min \u00b1 6.2 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.traffic"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52091 2026-11-17 08:57:00 UTC |
channel test | info docproc Neo4j load 118 q/min vs a normal 14 q/min ± 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 648,
"fingerprint": "e44edcffd892b971d1a5",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "docproc",
"title": "docproc Neo4j load 118 q/min vs a normal 14 q/min \u00b1 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 118.0,
"expected_median": 13.5,
"expected_mad": 4.89258,
"z": 21.358874050092183,
"rel": 7.7407407407407405,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 07:00 UTC",
"bucket": 1794902100,
"window": [
1794901500,
1794902700
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/docproc?cluster=production&from=1794879900&to=1794902700"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794879900&to=1794902700"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794903342,
"age_seconds": 676,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 13.6,
"status": "resolved",
"first_seen": 1788412396,
"last_seen": 1794904001,
"resolved_at": 1794905754,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 1235,
"narrative": "docproc's Neo4j load is 118 q/min on production, above a normal of 14 q/min \u00b1 5 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 21.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794904001,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 6493374,
"stale_s": 1769
},
"title": "docproc Neo4j load 118 q/min vs a normal 14 q/min \u00b1 5 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52090 2026-11-17 08:57:00 UTC |
channel test | info production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min ± 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 49,
"fingerprint": "7a1fc9a9706dadb35d51",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "*",
"title": "production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min \u00b1 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 302.0,
"expected_median": 57.2,
"expected_mad": 36.17544,
"z": 6.767022045896332,
"rel": 4.27972027972028,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 07:00 UTC",
"bucket": 1794902100,
"window": [
1794901500,
1794902700
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Cluster charts",
"href": "/charts?cluster=production&from=1794879900&to=1794902700"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794879900&to=1794902700"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794903342,
"age_seconds": 677,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 29.5,
"status": "resolved",
"first_seen": 1787697056,
"last_seen": 1794904001,
"resolved_at": 1794905754,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 1081,
"narrative": "the whole production cluster's Neo4j load is 302 q/min on production, above a normal of 57 q/min \u00b1 36 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.8, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794904001,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 7208714,
"stale_s": 1769
},
"title": "production (whole cluster) Neo4j load 302 q/min vs a normal 57 q/min \u00b1 36 q/min for Tue 07:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52089 2026-11-17 08:57:00 UTC |
channel test | info docsvc request rate 20.2 req/min vs a normal 1.0 req/min ± 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 223,
"fingerprint": "ee625f6aa4ec4f81cd50",
"kind": "anomaly.traffic",
"severity": "info",
"cluster": "production",
"subject": "docsvc",
"title": "docsvc request rate 20.2 req/min vs a normal 1.0 req/min \u00b1 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "api_rpm",
"unit": "req/min",
"value": 20.2,
"expected_median": 1.0,
"expected_mad": 1.4826,
"z": 12.950222581950628,
"rel": 19.2,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 07:00 UTC",
"bucket": 1794902100,
"window": [
1794901500,
1794902700
],
"recent_deploys": [],
"discriminator": "api_rpm"
},
"evidence": [
{
"label": "Service",
"href": "/service/docsvc?cluster=production&from=1794879900&to=1794902700"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794879900&to=1794902700"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794903342,
"age_seconds": 676,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 26.8,
"status": "resolved",
"first_seen": 1787816487,
"last_seen": 1794904001,
"resolved_at": 1794905754,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 574,
"narrative": "docsvc's request rate is 20.2 req/min on production, above a normal of 1.0 req/min \u00b1 1.5 req/min for this hour of the week (3 of the last 4 five-minute buckets, z = 13.0, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "api_rpm",
"level": "low",
"told_level": "low",
"episode_at": 1794902649,
"chat_thread": null,
"kind_label": "Traffic anomaly",
"group": "Anomalies",
"age_s": 7089283,
"stale_s": 1769
},
"title": "docsvc request rate 20.2 req/min vs a normal 1.0 req/min \u00b1 1.5 req/min for Tue 07:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.traffic"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52049 2026-11-17 07:35:33 UTC |
channel test | medium [MEDIUM] execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
show the card🟡 MEDIUM · execsvc.auto-query-reeval-sweep: 11 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=? production · Background job failing · medium The background job execsvc.auto-query-reeval-sweep on production failed 11 runs recently (358 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts. The facts job: execsvc.auto-query-reeval-sweep service: execsvc failed_runs: 11 error_lines: 358 sample: no result from branch evaluation for branch, protocolId=77bcfccb-cc59-4aea-bff8-0991eaa0c788 activityRefId=76d84261-cdb3-4d29-aba0-234a24c8652d error_signature: no result from branch evaluation for branch, protocolId=? activityRefId=? live · observed 11m ago · 5× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52049 |
#in-app delivered#prod-notifications sent |
opened | in-app: 1 inbox row(s) for [email protected] ok |
| #52047 2026-11-17 07:35:31 UTC |
channel test | info execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
show the card✅ resolved: execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro production · Background job failing · medium Open for 9h, observed 7 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 3h ago · 7× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52047 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #52045 2026-11-17 07:35:31 UTC |
channel test | info pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
show the card✅ resolved: pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested production · Background job failing · medium Open for 70d, observed 570 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 4h ago · 570× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52045 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #52034 2026-11-17 06:43:15 UTC |
channel test | warning onboardingsvc is burning its error budget at 0.0× (1 h) / 13.3× (6 h) against a 99.50% objective
show the payload{
"event": "resolved",
"finding": {
"id": 1530,
"fingerprint": "9e19b05a63df8239d9bc",
"kind": "budget.burning",
"severity": "warning",
"cluster": "production",
"subject": "onboardingsvc",
"title": "onboardingsvc is burning its error budget at 0.0\u00d7 (1 h) / 13.3\u00d7 (6 h) against a 99.50% objective",
"facts": {
"objective": 99.5,
"objective_source": "derived",
"achieved": 93.33333333333333,
"calls": 225.0,
"bad": 15.0,
"c5xx": 15.0,
"budget": 1.125,
"spent": 13.333333333333334,
"remaining_pct": 0.0,
"burn_1h": 0.0,
"burn_6h": 13.333333333333332,
"verdict": "burning",
"p95_ms": 165.0,
"window": [
1794874505,
1794896105
],
"recent_deploys": [],
"discriminator": ""
},
"evidence": [
{
"label": "Error budgets",
"href": "/budgets?cluster=production&from=1794874505&to=1794896105"
},
{
"label": "onboardingsvc service page",
"href": "/service/onboardingsvc?cluster=production&from=1794874505&to=1794896105"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794895685,
"age_seconds": 424,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 51.6,
"status": "resolved",
"first_seen": 1789163502,
"last_seen": 1794896105,
"resolved_at": 1794897755,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 199,
"narrative": "onboardingsvc is spending its error budget 0.0\u00d7 faster than the objective allows over the last hour (13.3\u00d7 over six hours). The objective is 99.5%, derived \u2014 a baseline, not a promise. 15 bad events against 225 calls in the window.",
"narrative_kind": "template",
"discriminator": "",
"level": "low",
"told_level": "medium",
"episode_at": 1794892665,
"chat_thread": "xray-alert-11-748fd5a928-9e19b05a63df8239d9bc-1793224828",
"kind_label": "Error budget burning",
"group": "Reliability",
"age_s": 5734258,
"stale_s": 1655
},
"title": "onboardingsvc is burning its error budget at 0.0\u00d7 (1 h) / 13.3\u00d7 (6 h) against a 99.50% objective",
"severity": "warning",
"cluster": "production",
"kind": "budget.burning"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52020 2026-11-17 05:35:00 UTC |
channel test | info Tilda namespace findings (master): all clear — 1 resolved
show the card🔵 Tilda namespace findings (master): all clear — 1 resolved alert: Tilda namespace findings (master) · changed 1 resolved ✅ execsvc server-error rate 89.19% vs a normal 0.00% ± 1.00% for Tue 04:00 UTC (3 of the last 4 checks) — mostly POST /api/trialexec/v2/study/:id/activity/status (787× 5xx) message 2 for this group · last one 48m ago · X-ray message #52020 |
#dev-notifications sent#in-app delivered |
group | ok in-app: 52 inbox row(s) for 52 people on X-ray |
| #52018 2026-11-17 05:30:00 UTC |
channel test | info execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro
show the card✅ resolved: execsvc.auto-query-reeval: 6 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro production · Background job failing · medium Open for 7h, observed 7 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 1h ago · 7× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52018 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #52016 2026-11-17 05:30:00 UTC |
channel test | info pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested
show the card✅ resolved: pubsub.dashboards-production-dashsub: 7 failed runs in the last 6 h — message type retired, no longer ingested production · Background job failing · medium Open for 70d, observed 570 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 2h ago · 570× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #52016 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #52011 2026-11-17 05:17:58 UTC |
channel test | info production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min ± 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 49,
"fingerprint": "7a1fc9a9706dadb35d51",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "*",
"title": "production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min \u00b1 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 570.2,
"expected_median": 39.4,
"expected_mad": 10.08168,
"z": 52.64995516620246,
"rel": 13.472081218274115,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 04:00 UTC",
"bucket": 1794890400,
"window": [
1794889500,
1794890700
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Cluster charts",
"href": "/charts?cluster=production&from=1794867900&to=1794890700"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794867900&to=1794890700"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794891309,
"age_seconds": 352,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 29.5,
"status": "resolved",
"first_seen": 1787697056,
"last_seen": 1794891647,
"resolved_at": 1794892665,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 1080,
"narrative": "the whole production cluster's Neo4j load is 570 q/min on production, above a normal of 39 q/min \u00b1 10 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 52.6, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794891647,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 7195621,
"stale_s": 1030
},
"title": "production (whole cluster) Neo4j load 570 q/min vs a normal 39 q/min \u00b1 10 q/min for Tue 04:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #52000 2026-11-17 04:27:47 UTC |
channel test | info production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min ± 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)
show the payload{
"event": "resolved",
"finding": {
"id": 49,
"fingerprint": "7a1fc9a9706dadb35d51",
"kind": "anomaly.neo4j",
"severity": "info",
"cluster": "production",
"subject": "*",
"title": "production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min \u00b1 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)",
"facts": {
"measure": "neo_qpm",
"unit": "q/min",
"value": 96.2,
"expected_median": 38.2,
"expected_mad": 9.04386,
"z": 6.413190827810249,
"rel": 1.5183246073298429,
"buckets_hit": 3,
"n": 48,
"weeks": 4,
"slot": "Tue 03:00 UTC",
"bucket": 1794887700,
"window": [
1794886800,
1794888000
],
"recent_deploys": [],
"discriminator": "neo_qpm"
},
"evidence": [
{
"label": "Cluster charts",
"href": "/charts?cluster=production&from=1794865200&to=1794888000"
},
{
"label": "Fleet vs normal",
"href": "/fleet?cluster=production&from=1794865200&to=1794888000"
}
],
"provenance": {
"source": "rollup",
"collected_at": 1794888450,
"age_seconds": 76,
"stale": false,
"shown": null,
"total": null,
"truncated": false,
"coverage_from": null,
"retain_days": null,
"note": null
},
"impact": 29.5,
"status": "resolved",
"first_seen": 1787697056,
"last_seen": 1794888514,
"resolved_at": 1794889645,
"acked_by": null,
"acked_at": null,
"snoozed_until": null,
"occurrences": 1079,
"narrative": "the whole production cluster's Neo4j load is 96 q/min on production, above a normal of 38 q/min \u00b1 9 q/min for this hour of the week (3 of the last 4 five-minute buckets, z = 6.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it \u2014 the service page shows the split.",
"narrative_kind": "template",
"discriminator": "neo_qpm",
"level": "low",
"told_level": "low",
"episode_at": 1794882200,
"chat_thread": null,
"kind_label": "Neo4j load anomaly",
"group": "Anomalies",
"age_s": 7192602,
"stale_s": 1144
},
"title": "production (whole cluster) Neo4j load 96 q/min vs a normal 38 q/min \u00b1 9 q/min for Tue 03:00 UTC (3 of the last 4 checks)",
"severity": "info",
"cluster": "production",
"kind": "anomaly.neo4j"
} |
#in-app,prod-notifications queued |
digest-item | waiting for the digest |
| #51974 2026-11-17 03:30:36 UTC |
channel test | X-ray daily — 10 open (0 critical)
show the cardX-ray daily — 10 open (0 critical) +0 opened · −76 resolved since the last digest production 🔵 production (whole cluster) Neo4j load 117 q/min vs a normal 38 q/min ± 10 q/min for Tue 02:00 UTC (4 of the last 4 checks) 🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s) 🔵 docproc: p99 latency (ms) 9.03 s — 4.1× the fleet's usual 2.18 s 🔵 execsvc: Neo4j queries per request 72.7 — 7.2× the fleet's usual 10.1 🔵 dashql: p95 latency (ms) 4.27 s — 9.5× the fleet's usual 451 ms 🔵 virtual-assistant: p95 latency (ms) 1.57 s — 3.5× the fleet's usual 451 ms 🔵 tusker: Neo4j queries per request 24.5 — 4.1× the fleet's usual 6.0 🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s and 2 more 95 warning-level events were held for this digest · X-ray message #51974 |
#prod-notifications sent#in-app delivered#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent |
digest | ok in-app: 1 inbox row(s) for [email protected] waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest |
| #51953 2026-11-17 01:29:53 UTC |
channel test | high Production — critical & high: 1 finding firing (1 escalated)
show the card🟠 HIGH · escalated · execsvc.auto-query-reeval: 20 failed runs in the last 6 h — participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, pro production · Background job failing · high The background job execsvc.auto-query-reeval on production failed 20 runs recently (211 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts. The facts job: execsvc.auto-query-reeval service: execsvc failed_runs: 20 error_lines: 211 sample: participant activity 0294d67b-775d-4580-a46a-4b3085691136: auto query cee26d92-2b6c-4e45-a32f-b8758d9ff83a on question 522e2ac3-5aeb-4dd4-ac75-a4fc48384c13: no result from branch evaluation for branch error_signature: participant activity ?: auto query ? on question ?: no result from branch evaluation for branch, protocolId=? activityRe live · observed 41m ago · 4× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51953 |
#prod-notifications sent#in-app delivered |
escalated | ok in-app: 1 inbox row(s) for [email protected] |
| #51936 2026-11-16 23:34:32 UTC |
channel test | medium [MEDIUM] execsvc.auto-query-reeval: 16 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=?
show the card🟡 MEDIUM · execsvc.auto-query-reeval: 16 failed runs in the last 6 h — no result from branch evaluation for branch, protocolId=? activityRefId=? production · Background job failing · medium The background job execsvc.auto-query-reeval on production failed 16 runs recently (533 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts. The facts job: execsvc.auto-query-reeval service: execsvc failed_runs: 16 error_lines: 533 sample: no result from branch evaluation for branch, protocolId=77bcfccb-cc59-4aea-bff8-0991eaa0c788 activityRefId=4b4f0a72-5853-4c33-927d-a45776e920bb error_signature: no result from branch evaluation for branch, protocolId=? activityRefId=? live · observed 53m ago · 2× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51936 |
#prod-notifications sent#in-app delivered |
opened | ok in-app: 1 inbox row(s) for [email protected] |
| #51934 2026-11-16 23:34:32 UTC |
channel test | info rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
show the card✅ resolved: rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s production · Background job failing · medium Open for 20d, observed 12 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 8h ago · 12× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51934 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #51911 2026-11-16 21:29:12 UTC |
channel test | info execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no )
show the card✅ resolved: execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no ) production · Error never seen before · high Open for 4h, observed 3 times. A healthy detector run no longer sees it, or someone resolved it. live · observed 4h ago · 3× so far · Cloud Logging, last hour, grouped by (service, caller, shape) · X-ray message #51911 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #51845 2026-11-16 17:28:30 UTC |
channel test | high Production — critical & high: 2 findings firing (1 new, 1 escalated) · 473 events in the last hour
show the card🟠 Production — critical & high: 2 findings firing (1 new, 1 escalated) · 473 events in the last hour alert: Production — critical & high · changed 2 firing 🟠 high · production · new · ×473 in the last hour: execsvc: a new error shape, 473× in the last hour, never seen in 90 days (failed to evaluate and raise auto query for participant question — no ) 🟠 high · production · escalated · observed 561× since 70d ago: pubsub.dashboards-production-dashsub: 22 failed runs in the last 6 h — message type retired, no longer ingested message 66 for this group · last one 4h ago · X-ray message #51845 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #51802 2026-11-16 16:16:16 UTC |
channel test | warning Any error → notify me (master): 5 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in tilda on master",
"body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in tilda on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794845806,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51802,
"message_url": "https://xray.tilda.pizza/alerts/messages/51802"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51796 2026-11-16 15:34:10 UTC |
channel test | medium [MEDIUM] rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s
show the card🟡 MEDIUM · reopened · rag-agent.upload: 5 failed runs in the last 6 h — OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open s production · Background job failing · medium The background job rag-agent.upload on production failed 5 runs recently (53 failure lines; lines within one run count once), detected via its ERROR-severity log lines. A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts. The facts job: rag-agent.upload service: rag-agent failed_runs: 5 error_lines: 53 sample: OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open stream: Failed to open stream: Failed to get PDF page count: Failed to open stream: Failed to open st error_signature: OCR processing failed: Chunked PDF processing failed: Failed to get PDF page count: Failed to open stream: Failed to ope live · observed 16m ago · 12× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51796 |
#prod-notifications sent#in-app delivered |
reopened | ok in-app: 1 inbox row(s) for [email protected] |
| #51783 2026-11-16 15:16:08 UTC |
channel test | warning Any error → notify me (master): 2 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 2 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in chat-agent (tilda) on master; Error in docsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in chat-agent (tilda) on master; Error in docsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794842198,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51783,
"message_url": "https://xray.tilda.pizza/alerts/messages/51783"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51736 2026-11-16 14:15:55 UTC |
channel test | warning Any error → notify me (master): 5 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 5 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794838585,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51736,
"message_url": "https://xray.tilda.pizza/alerts/messages/51736"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51730 2026-11-16 13:28:15 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 26× since 18d ago: Neo4j production, database neo4j: 15 Cypher queries failed in the last 15 min message 65 for this group · last one 4h ago · X-ray message #51730 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #51710 2026-11-16 13:15:10 UTC |
channel test | warning Any error → notify me (master): 6 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 6 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 6 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794834940,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51710,
"message_url": "https://xray.tilda.pizza/alerts/messages/51710"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51652 2026-11-16 12:15:02 UTC |
channel test | warning Any error → notify me (master): 1 more message held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 1 more message held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794831332,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51652,
"message_url": "https://xray.tilda.pizza/alerts/messages/51652"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51565 2026-11-16 09:27:31 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 64 for this group · last one 4h ago · X-ray message #51565 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #51564 2026-11-16 09:26:30 UTC |
channel test | —
show the payload{
"reason": "unchanged",
"pending": 6
} |
#in-app,prod-notifications suppressed |
group | 6 event(s) changed nothing about the group since the last message |
| #51529 2026-11-16 08:24:39 UTC |
channel test | warning Any error → notify me (master): 10 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 10 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 10 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in studiosvc (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794817510,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51529,
"message_url": "https://xray.tilda.pizza/alerts/messages/51529"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51500 2026-11-16 07:33:30 UTC |
channel test | medium [MEDIUM] pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ?
show the card🟡 MEDIUM · reopened · pubsub.dashboards-production-dashsub: 5 failed runs in the last 6 h — no data found for label RecommendationByDocument, id ? production · Background job failing · medium The background job pubsub.dashboards-production-dashsub on production failed 5 runs recently (27 failure lines; lines within one run count once), detected via its structured error/exception field (below ERROR severity). A job that logs its failures below ERROR severity — a WARNING carrying an error field — is invisible to the novel-shape detector and to the page's own failed count; this detector reads the error field itself, which is why the finding now appears. The sample failure is in the facts. The facts job: pubsub.dashboards-production-dashsub service: dashrsvc failed_runs: 5 error_lines: 27 sample: no data found for label RecommendationByDocument, id 0d2b5f93-69f7-4725-a10b-69b19e08e475 error_signature: no data found for label RecommendationByDocument, id ? live · observed 6m ago · 553× so far · error-severity job lines from BigQuery, last hour, burst-grouped per job · X-ray message #51500 |
#in-app delivered#prod-notifications sent |
reopened | in-app: 1 inbox row(s) for [email protected] ok |
| #51490 2026-11-16 07:24:28 UTC |
channel test | warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in docsvc (tilda) on master; Error in docsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in docsvc (tilda) on master; Error in docsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794813898,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51490,
"message_url": "https://xray.tilda.pizza/alerts/messages/51490"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51435 2026-11-16 05:55:19 UTC |
channel test | —
show the payload{
"reason": "unchanged",
"pending": 1
} |
#in-app,prod-notifications suppressed |
group | 1 event(s) changed nothing about the group since the last message |
| #51427 2026-11-16 05:26:13 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 63 for this group · last one 4h ago · X-ray message #51427 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #51426 2026-11-16 05:25:13 UTC |
channel test | —
show the payload{
"reason": "unchanged",
"pending": 2
} |
#in-app,prod-notifications suppressed |
group | 2 event(s) changed nothing about the group since the last message |
| #51414 2026-11-16 03:30:03 UTC |
channel test | X-ray daily — 10 open (0 critical)
show the cardX-ray daily — 10 open (0 critical) +0 opened · −3 resolved since the last digest production 🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s) 🔵 execsvc p95 3.88 s vs a normal 🔵 dashql: p99 latency (ms) 3.88 s — 4.2× the fleet's usual 913 ms 🔵 docproc: Neo4j queries per request 74.7 — 16.3× the fleet's usual 4.6 🔵 execsvc: Neo4j queries per request 11.9 — 2.2× the fleet's usual 5.4 🔵 virtual-assistant: Neo4j queries per request 78.2 — 10.7× the fleet's usual 7.3 🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s 🔵 tusker: Neo4j queries per request 109.7 — 6.3× the fleet's usual 17.4 and 2 more 4 warning-level events were held for this digest · X-ray message #51414 |
#prod-notifications sent#in-app delivered#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent |
digest | ok in-app: 1 inbox row(s) for [email protected] waiting for the digest waiting for the digest waiting for the digest waiting for the digest |
| #51398 2026-11-16 01:41:21 UTC |
channel test | warning Any error → notify me (master): 19 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 19 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 19 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in tilda on master; Error in docproc (tilda) on master; Error in ontologysvc (tilda) on master; Error in tilda on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in tilda on master; Error in docproc (tilda) on master; Error in ontologysvc (tilda) on master; Error in tilda on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794793311,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51398,
"message_url": "https://xray.tilda.pizza/alerts/messages/51398"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51388 2026-11-16 01:24:20 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 62 for this group · last one 4h ago · X-ray message #51388 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #51339 2026-11-16 00:41:18 UTC |
channel test | warning Any error → notify me (master): 78 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 78 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 78 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794789708,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51339,
"message_url": "https://xray.tilda.pizza/alerts/messages/51339"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51144 2026-11-15 23:41:15 UTC |
channel test | warning Any error → notify me (master): 35 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 35 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 35 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794786105,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51144,
"message_url": "https://xray.tilda.pizza/alerts/messages/51144"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #51044 2026-11-15 22:41:10 UTC |
channel test | warning Any error → notify me (master): 38 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 38 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 38 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794782500,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 51044,
"message_url": "https://xray.tilda.pizza/alerts/messages/51044"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50932 2026-11-15 21:41:02 UTC |
channel test | warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in docproc (tilda) on master; Error in docproc (tilda) on master; Error in docproc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794778892,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 50932,
"message_url": "https://xray.tilda.pizza/alerts/messages/50932"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50902 2026-11-15 21:24:00 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 61 for this group · last one 4h ago · X-ray message #50902 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50862 2026-11-15 17:23:43 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 18d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 18d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 18d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 60 for this group · last one 4h ago · X-ray message #50862 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50846 2026-11-15 14:49:13 UTC |
channel test | warning Any error → notify me (master): 15 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 15 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 15 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in execsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in tilda on master; Error in virtual-assistant (tilda) on master; Error in execsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794754183,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 50846,
"message_url": "https://xray.tilda.pizza/alerts/messages/50846"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50820 2026-11-15 13:54:55 UTC |
channel test | info HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
show the card✅ resolved: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h production · Autoscaler blind · medium Open for 43d, observed 177 times. A healthy detector run no longer sees it, or someone resolved it. x-ray · observed 6h ago · 177× so far · Kubernetes events (logs/events in BigQuery), last 45 min · X-ray message #50820 |
#prod-notifications sent#in-app delivered |
resolved | ok in-app: 1 inbox row(s) for [email protected] |
| #50810 2026-11-15 13:48:53 UTC |
channel test | warning Any error → notify me (master): 7 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 7 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 7 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master; Error in virtual-assistant (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794750563,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 50810,
"message_url": "https://xray.tilda.pizza/alerts/messages/50810"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50776 2026-11-15 13:22:50 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 59 for this group · last one 4h ago · X-ray message #50776 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50744 2026-11-15 09:21:57 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 58 for this group · last one 4h ago · X-ray message #50744 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50743 2026-11-15 09:20:57 UTC |
channel test | —
show the payload{
"reason": "unchanged",
"pending": 2
} |
#in-app,prod-notifications suppressed |
group | 2 event(s) changed nothing about the group since the last message |
| #50735 2026-11-15 08:42:54 UTC |
channel test | warning Any error → notify me (master): 23 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 23 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 23 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794732204,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 50735,
"message_url": "https://xray.tilda.pizza/alerts/messages/50735"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50677 2026-11-15 07:42:12 UTC |
channel test | warning Any error → notify me (master): 3 more messages held by the rate limit in the last hour
show the payload{
"title": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour",
"text": "Any error \u2192 notify me (master): 3 more messages held by the rate limit in the last hour\nThe alert allows 20 an hour. Most recent held: Error in exportsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
"body": "The alert allows 20 an hour. Most recent held: Error in exportsvc (tilda) on master; Error in execsvc (tilda) on master; Error in execsvc (tilda) on master",
"url": "https://xray.tilda.pizza/alerts/8",
"severity": "warning",
"_scope": {},
"_job": {
"finding": null,
"force": true,
"retry_at": 1794728562,
"route_signature": "{\"clusters\": [\"master\"], \"kind\": \"log\", \"kinds\": [], \"min_severity\": \"warning\", \"mode\": \"immediate\", \"services\": \"*\", \"spec\": {\"namespaces\": [\"tilda\"]}}",
"digest_items": []
},
"message_id": 50677,
"message_url": "https://xray.tilda.pizza/alerts/messages/50677"
} |
#dev-notifications sent#in-app delivered |
rollup | ok in-app: 1 inbox row(s) for [email protected] |
| #50613 2026-11-15 05:54:29 UTC |
channel test | medium [MEDIUM] HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h
show the card🟡 MEDIUM · reopened · HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h production · Autoscaler blind · medium Autoscaler blind: HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 16× in 2 h. Facts: namespace tilda, object HorizontalPodAutoscaler/ontologysvc-hpa, failures_2h 16, last_message No recommendation. The facts namespace: tilda object: HorizontalPodAutoscaler/ontologysvc-hpa failures_2h: 16 last_message: No recommendation discriminator: x-ray · observed 6m ago · 163× so far · Kubernetes events (logs/events in BigQuery), last 45 min · X-ray message #50613 |
#prod-notifications sent#in-app delivered |
reopened | ok in-app: 1 inbox row(s) for [email protected] |
| #50611 2026-11-15 05:20:28 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 57 for this group · last one 4h ago · X-ray message #50611 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50597 2026-11-15 03:30:16 UTC |
channel test | X-ray daily — 10 open (0 critical)
show the cardX-ray daily — 10 open (0 critical) +0 opened · −5 resolved since the last digest production 🔵 docsvc: Neo4j queries per request 90.0 — 19.6× the fleet's usual 4.6 (also p99 latency (ms) 4.62 s) 🔵 execsvc p95 3.88 s vs a normal 🔵 dashql: p99 latency (ms) 3.88 s — 4.2× the fleet's usual 913 ms 🔵 docproc: Neo4j queries per request 74.7 — 16.3× the fleet's usual 4.6 🔵 execsvc: Neo4j queries per request 11.9 — 2.2× the fleet's usual 5.4 🔵 virtual-assistant: Neo4j queries per request 78.2 — 10.7× the fleet's usual 7.3 🔵 enrollsvc: p99 latency (ms) 4.24 s — 4.1× the fleet's usual 1.04 s 🔵 tusker: Neo4j queries per request 109.7 — 6.3× the fleet's usual 17.4 and 2 more 6 warning-level events were held for this digest · X-ray message #50597 |
#prod-notifications sent#in-app delivered#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent#in-app,prod-notifications sent |
digest | ok in-app: 1 inbox row(s) for [email protected] waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest waiting for the digest |
| #50565 2026-11-15 01:19:48 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 56 for this group · last one 4h ago · X-ray message #50565 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50536 2026-11-14 21:19:15 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 55 for this group · last one 4h ago · X-ray message #50536 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50523 2026-11-14 18:42:46 UTC |
channel test | medium Production — medium: still open — 4 findings, oldest 17d · 27 events in the last hour
show the card🔔 Production — medium: still open — 4 findings, oldest 17d · 27 events in the last hour alert: Production — medium · repeat 4 firing 🟡 medium · production · ×15 in the last hour: execsvc: a new error shape, 15× in the last hour, never seen in 90 days (failed to process db result — Neo4jError: Neo.TransientError.Transacti) 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min 🟡 medium · production · ×6 in the last hour: rag-agent: a new error shape, 6× in the last hour, never seen in 90 days (Chunked PDF processing failed: Failed to get PDF page count: Failed to) 🟡 medium · production · ×6 in the last hour: rag-agent: a new error shape, 6× in the last hour, never seen in 90 days (OCR processing failed: Chunked PDF processing failed: Failed to get PD) message 65 for this group · last one 12h ago · X-ray message #50523 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50508 2026-11-14 17:18:38 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 17d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 17d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 17d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 54 for this group · last one 4h ago · X-ray message #50508 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |
| #50465 2026-11-14 13:18:14 UTC |
channel test | medium Production — critical & high: still open — 1 finding, oldest 16d
show the card🔔 Production — critical & high: still open — 1 finding, oldest 16d alert: Production — critical & high · repeat 1 firing 🟡 medium · production · observed 25× since 16d ago: Neo4j production, database neo4j: 6 Cypher queries failed in the last 15 min message 53 for this group · last one 4h ago · X-ray message #50465 |
#prod-notifications sent#in-app delivered |
group | ok in-app: 1 inbox row(s) for [email protected] |