dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)
warning
Query shape degrading
test26
resolved
impact 30
Owner: Vishal Karedla (backup: Taylor Futral)
opened 34d ago · last observed 3d ago · observed 8× · #4863 · 2ecab535f846e759d22a
What happened template
A dashrsvc query shape on test26 has p95 1.7 s in the second half of the window against 19 ms in the first (89.8×, 272 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | dashrsvc |
| plan_hash | 00000000 |
| queries | 272 |
| floor_ms | 1,000 |
| p95_early | 18.92 |
| p95_late | 1,700 |
| ratio | 89.83 |
| avg_ms | 44.12 |
| sample | MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ? |
| rw | write |
| severity_capped | critical → warning: test26 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-15 00:07:50 UTC → 2026-11-15 06:07:50 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-10-15 20:18:38 UTCthis finding dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)
- 2026-10-26 07:19:27 UTCfinding · info execsvc runs 37 foreground Neo4j queries per request (12,049 queries / 327 calls, last hour)
- 2026-10-26 07:28:31 UTCfinding · info enrollsvc Neo4j load 43 q/min vs a normal 3 q/min ± 4 q/min for Tue 06:00 UTC (3 of the last 4 checks)
- 2026-10-27 19:38:41 UTCfinding · info feedsvc Neo4j load 93 q/min vs a normal 0 q/min ± 20 q/min for Tue 00:00 UTC (3 of the last 4 checks)
- 2026-10-28 02:02:34 UTCfinding · warning execsvc (tilda) image pull failing for at least 13 min — 1 pod
- 2026-10-28 03:47:44 UTCfinding · info feedsvc: Neo4j queries per request 162.4 — 22.3× the fleet's usual 7.3
- 2026-10-28 04:49:11 UTCfinding · info feedsvc request rate 25.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Tue 01:00 UTC (4 of the last 4 checks)
- 2026-10-28 18:05:55 UTCfinding · warning feedsvc shape 00000000: p95 4.85 s now vs 28 ms earlier (172.7×, 3,394 runs)
- 2026-10-29 01:01:22 UTCfinding · info docsvc request rate 28.8 req/min vs a normal 0.0 req/min ± 5.0 req/min for Mon 06:00 UTC (3 of the last 4 checks)
- 2026-11-14 10:16:00 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 10:51:22 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 11:00:00 UTClatency cluster p95 peaked at ≥ 2.0 min
- 2026-11-14 11:14:54 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:16:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 11:52:45 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:16:14 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 14:27:08 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:27:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:30:14 UTCdeploy [email protected] deployed studiosvc-infra-dev
- 2026-11-14 14:45:39 UTCdeploy automated deployed studiosvc-infra-dev
- 2026-11-14 18:24:25 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/onboardingsvc by hand (kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350,gzip(gfe))
- 2026-11-14 20:00:17 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:00:18 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-14 20:17:22 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 20:17:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 23:49:52 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/feedsvc by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:58 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/virtual-assistant by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:06:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 02:24:26 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/execsvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 02:25:02 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-15 05:02:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 05:16:48 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:11:31 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
- 2026-11-15 06:32:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/chat-agent by hand (kubectl/v1.36.1 (darwin/arm64) kubernetes/7569396,gzip(gfe))
History
- 2026-11-15 06:07:50 UTCreopened x-ray · severity=warning was=warning title=dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs) level=low was_level=low finding={'id': 4863, 'fingerprint': '2ecab535f846e759d22a', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'dashrsvc', 'title': 'dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)', 'facts': {'service': 'dashrsvc', 'plan_hash': '00000000', 'queries': 272, 'floor_ms': 1000, 'p95_early': 18.924999999999997, 'p95_late': 1699.9999999999993, 'ratio': 89.82826948480843, 'avg_ms': 44.12132352941177, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794701270, 1794722870], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794701270&to=1794722870'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794701270&to=1794722870&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794722870, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.5, 'status': 'open', 'first_seen': 1792095518, 'last_seen': 1794722870, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': 'A dashrsvc query shape on test26 has p95 1.3 s in the second half of the window against 19 ms in the first (67.0×, 322 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794722870, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 2627357, 'stale_s': 5}
- 2026-11-15 06:28:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 4863, 'fingerprint': '2ecab535f846e759d22a', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test26', 'subject': 'dashrsvc', 'title': 'dashrsvc shape 00000000: p95 1.70 s now vs 19 ms earlier (89.8×, 272 runs)', 'facts': {'service': 'dashrsvc', 'plan_hash': '00000000', 'queries': 272, 'floor_ms': 1000, 'p95_early': 18.924999999999997, 'p95_late': 1699.9999999999993, 'ratio': 89.82826948480843, 'avg_ms': 44.12132352941177, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794701270, 1794722870], 'recent_deploys': [], 'severity_capped': 'critical → warning: test26 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test26&from=1794701270&to=1794722870'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test26&from=1794701270&to=1794722870&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794722870, 'age_seconds': 5, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.5, 'status': 'resolved', 'first_seen': 1792095518, 'last_seen': 1794722870, 'resolved_at': 1794724138, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': 'A dashrsvc query shape on test26 has p95 1.7 s in the second half of the window against 19 ms in the first (89.8×, 272 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794722870, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 2628625, 'stale_s': 1273}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794722870 |
| age_seconds | 5 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |