studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)
warning
Query shape degrading
test27
resolved
impact 30
Owner: Suresh Boddapati (backup: Madhava Mukkamala)
opened 12d ago · last observed 3d ago · observed 8× · #16285 · 4a6d5a9ad3ac4020bcc5
What happened template
A studiosvc query shape on test27 has p95 2.2 s in the second half of the window against 128 ms in the first (16.8×, 865 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | studiosvc |
| plan_hash | 00000000 |
| queries | 865 |
| floor_ms | 1,000 |
| p95_early | 127.71 |
| p95_late | 2,150 |
| ratio | 16.83 |
| avg_ms | 44.40 |
| sample | MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ? |
| rw | write |
| severity_capped | critical → warning: test27 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-14 10:49:02 UTC → 2026-11-14 16:49:02 UTC |
What is failing in studiosvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from studiosvc. In this finding's window (2026-11-14 10:49:02 UTC → 2026-11-14 16:49:02 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 13006× | error on pubsub receive — rpc error: code = NotFound desc = Resource not found (resource=study-site-binding-test27-studi… since 2026-11-14 14:23:11 UTC |
| 17× | http error response returned to client since 2026-10-06 01:20:44 UTC |
| 12× | failed to fetch form summary — form 76909f97-e9ff-48dd-b4ff-65597c5c1ee1 not found since 2026-11-05 13:41:09 UTC |
| 9× | failed to fetch question detail — question not found since 2026-09-16 05:04:44 UTC |
| 8× | Request completed — 500 · graph action pre failed for FormV2:object is being used for checking conditions in one or more… since 2026-10-18 02:42:24 UTC |
The story — No studiosvc deploy or hand change is recorded in the hour before this opened. Everything below is studiosvc only.
- 2026-11-05 16:33:42 UTCthis finding studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)
- 2026-11-05 22:27:19 UTCfinding · info studiosvc p95 7.35 s vs a normal 199 ms ± 157 ms for Tue 07:00 UTC (4 of the last 4 checks)
- 2026-11-14 12:00:44 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
- 2026-11-14 14:00:00 UTClatency studiosvc p95 peaked at ≥ 2.0 min
- 2026-11-14 14:25:45 UTCfinding · warning studiosvc: a new error shape, ≥13006× in the last hour, never seen in 90 days (error on pubsub receive — rpc error: code = NotFound desc = Resource n)
- 2026-11-14 14:26:39 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/studiosvc by hand (kubectl/v1.35.9 (darwin/arm64) kubernetes/d834de3,gzip(gfe))
History
- 2026-11-14 16:08:09 UTCreopened x-ray · severity=warning was=warning title=studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs) level=low was_level=low finding={'id': 16285, 'fingerprint': '4a6d5a9ad3ac4020bcc5', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)', 'facts': {'service': 'studiosvc', 'plan_hash': '00000000', 'queries': 865, 'floor_ms': 1000, 'p95_early': 127.71428571428552, 'p95_late': 2150.0000000000023, 'ratio': 16.834451901566037, 'avg_ms': 44.39537572254335, 'sample': '// First part for StudySiteUser scope match (:StudySiteDuty)-[:STUDYSITEDUTY_TO_STUDYSITEACTOR]->(scopeUser:StudySiteActor{delpending:false}) with distinct scopeUser match (scopeUser)-[:STUDYSITEACTOR_TO_PLAN_DEFINITION]->(sitePd:DslPlanDefinition{type:?}) match (sitePd)-[:PLAN_DEFINITION_TO_ADOPTED_WORKFLOWS]->(w:WorkflowV2)<-[:TRIGGER_TO_WORKFLOW]-(t:Trigger{scope:?}) match (sitePd)-[:PLAN_DEFIN', 'rw': 'write', 'window': [1794650889, 1794672489], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794650889&to=1794672489'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794650889&to=1794672489&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794672487, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.9, 'status': 'open', 'first_seen': 1793896422, 'last_seen': 1794672489, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 6, 'narrative': 'A studiosvc query shape on test27 has p95 2.6 s in the second half of the window against 11 ms in the first (231.6×, 3,642 executions). The shape reads: “MATCH (c:TMFSyncConfig) RETURN c {.*} AS c ORDER BY c.created_at DESC SKIP $offset LIMIT $limit…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794672489, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 776071, 'stale_s': 4}
- 2026-11-14 22:06:36 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16285, 'fingerprint': '4a6d5a9ad3ac4020bcc5', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'studiosvc', 'title': 'studiosvc shape 00000000: p95 2.15 s now vs 128 ms earlier (16.8×, 865 runs)', 'facts': {'service': 'studiosvc', 'plan_hash': '00000000', 'queries': 865, 'floor_ms': 1000, 'p95_early': 127.71428571428552, 'p95_late': 2150.0000000000023, 'ratio': 16.834451901566037, 'avg_ms': 44.39537572254335, 'sample': 'MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organization) } RETURN task.tildaid LIMIT ?', 'rw': 'write', 'window': [1794653342, 1794674942], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794653342&to=1794674942'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794653342&to=1794674942&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794674940, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.9, 'status': 'resolved', 'first_seen': 1793896422, 'last_seen': 1794674942, 'resolved_at': 1794693996, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': 'A studiosvc query shape on test27 has p95 2.2 s in the second half of the window against 128 ms in the first (16.8×, 865 executions). The shape reads: “MATCH (task:Task) WHERE task.task_number = ? AND EXISTS { (task)-[:BELONGS_TO]->(:Inbox{type:?})-[:MANAGED_BY]->(:Organi…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794672489, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 797579, 'stale_s': 19059}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794674940 |
| age_seconds | 7 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |