docsvc shape 00000000: p95 1.65 s now vs 69 ms earlier (23.9×, 1,858 runs)
warning
Query shape degrading
test27
resolved
impact 30
Owner: Prasad Limaye (backup: Harish Kumar)
opened 70d ago · last observed 1d ago · observed 141× · #1179 · 27f45e142170efbfa3ac
What happened template
A docsvc query shape on test27 has p95 1.6 s in the second half of the window against 69 ms in the first (23.9×, 1,858 executions). The shape reads: “MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | docsvc |
| plan_hash | 00000000 |
| queries | 1,858 |
| floor_ms | 1,000 |
| p95_early | 69.13 |
| p95_late | 1,650 |
| ratio | 23.87 |
| avg_ms | 41.48 |
| sample | MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a |
| rw | write |
| severity_capped | critical → warning: test27 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-16 19:31:55 UTC → 2026-11-17 01:31:55 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 19:31:55 UTC → 2026-11-17 01:31:55 UTC) there were 4; every line links to the grouped errors behind it.
PUT /api/documents/promote |
4 bad of 6 calls · 100% of the service's 5xx |
The error shapes behind them
| 4× | Request completed — 500 · document is already in Staging collection, no promotion needed since 2026-11-03 16:40:25 UTC |
| 2× | failed to process db result — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transactionId=1… since 2026-11-03 12:36:16 UTC |
| 1× | failed to write relationship cypher — Neo4jError: Neo.TransientError.Transaction.DeadlockDetected (ForsetiClient[transac… since 2026-11-03 12:36:16 UTC |
| 1× | Failed to create EventPlan for Study — failed to link event plan to study: Neo4jError: Neo.TransientError.Transaction.De… since 2026-11-15 16:39:56 UTC |
| 1× | could not update document metadata for doc id fde96cb4-ea75-492a-85f3-e5b447ff3612: conflict: document fde96cb4-ea75-492… since 2026-11-17 06:53:42 UTC |
The story — No docsvc deploy or hand change is recorded in the hour before this opened. Everything below is docsvc only.
- 2026-09-08 23:07:35 UTCthis finding docsvc shape 00000000: p95 1.65 s now vs 69 ms earlier (23.9×, 1,858 runs)
- 2026-09-11 16:28:43 UTCfinding · info docsvc p95 4.47 s vs a normal 91 ms ± 63 ms for Sat 18:00 UTC (3 of the last 4 checks)
- 2026-10-23 13:49:17 UTCfinding · info docsvc: Neo4j queries per request 107.5 — 18.6× the fleet's usual 5.8 (also p95 latency (ms) 2.77 s)
- 2026-10-23 13:49:17 UTCfinding · info docsvc Neo4j load 7,364 q/min vs a normal 1 q/min ± 1 q/min for Tue 06:00 UTC (3 of the last 4 checks)
- 2026-10-23 13:49:17 UTCfinding · info docsvc request rate 6.4 req/min vs a normal 0.2 req/min ± 0.3 req/min for Mon 20:00 UTC (3 of the last 4 checks)
- 2026-11-15 00:00:00 UTClatency docsvc p95 peaked at 4.05 s
- 2026-11-16 00:00:00 UTCerrors docsvc 5xx peaked at 9 in one five-minute bucket (median 0 in this window)
- 2026-11-16 04:26:06 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 07:10:55 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 07:15:20 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 10:18:07 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.35.1 (darwin/arm64) kubernetes/8fea90b,gzip(gfe))
- 2026-11-16 16:46:16 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:30:32 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:30:33 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:23 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 18:54:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.30.5 (darwin/arm64) kubernetes/633f2e9,gzip(gfe))
- 2026-11-16 21:30:24 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-16 21:30:27 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/docsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
History
- 2026-11-17 01:15:27 UTCreopened x-ray · severity=warning was=warning title=docsvc shape 00000000: p95 1.90 s now vs 68 ms earlier (27.9×, 1,883 runs) level=low was_level=low finding={'id': 1179, 'fingerprint': '27f45e142170efbfa3ac', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.90 s now vs 68 ms earlier (27.9×, 1,883 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 1883, 'floor_ms': 1000, 'p95_early': 68.1249999999998, 'p95_late': 1899.9999999999986, 'ratio': 27.889908256880794, 'avg_ms': 41.02920870950611, 'sample': '// First part for StudySiteUser scope match (:StudySiteDuty)-[:STUDYSITEDUTY_TO_STUDYSITEACTOR]->(scopeUser:StudySiteActor{delpending:false}) with distinct scopeUser match (scopeUser)-[:STUDYSITEACTOR_TO_PLAN_DEFINITION]->(sitePd:DslPlanDefinition{type:?}) match (sitePd)-[:PLAN_DEFINITION_TO_ADOPTED_WORKFLOWS]->(w:WorkflowV2)<-[:TRIGGER_TO_WORKFLOW]-(t:Trigger{scope:?}) match (sitePd)-[:PLAN_DEFIN', 'rw': 'write', 'window': [1794856527, 1794878127], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794856527&to=1794878127'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794856527&to=1794878127&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794877920, 'age_seconds': 214, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 25.1, 'status': 'open', 'first_seen': 1788908855, 'last_seen': 1794878127, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 140, 'narrative': 'A docsvc query shape on test27 has p95 1.1 s in the second half of the window against 365 ms in the first (3.1×, 353 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794878127, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 5969278, 'stale_s': 6}
- 2026-11-17 01:53:30 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 1179, 'fingerprint': '27f45e142170efbfa3ac', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'docsvc', 'title': 'docsvc shape 00000000: p95 1.65 s now vs 69 ms earlier (23.9×, 1,858 runs)', 'facts': {'service': 'docsvc', 'plan_hash': '00000000', 'queries': 1858, 'floor_ms': 1000, 'p95_early': 69.12790697674397, 'p95_late': 1649.9999999999986, 'ratio': 23.868797308662796, 'avg_ms': 41.48277717976319, 'sample': 'MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a', 'rw': 'write', 'window': [1794857515, 1794879115], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794857515&to=1794879115'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794857515&to=1794879115&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794878907, 'age_seconds': 216, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 30.1, 'status': 'resolved', 'first_seen': 1788908855, 'last_seen': 1794879115, 'resolved_at': 1794880410, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 141, 'narrative': 'A docsvc query shape on test27 has p95 1.6 s in the second half of the window against 69 ms in the first (23.9×, 1,858 executions). The shape reads: “MATCH(a:Organization{deltimestamp:$a_deltimestamp,alias:$a_alias}) RETURN a…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794878127, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 5971562, 'stale_s': 1302}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794878907 |
| age_seconds | 216 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |