enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)
warning
Query shape degrading
test27
resolved
impact 30
Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli)
opened 5d ago · last observed 1d ago · observed 9× · #17200 · 67b0fcdccd582d4ad4b7
What happened template
A enrollsvc query shape on test27 has p95 3.0 s in the second half of the window against 296 ms in the first (10.1×, 696 executions). The shape reads: “MATCH (d:Document {tildaid: $document_id}) MATCH (d)-[:DOC_TO_ORG]-(o:Organization) MATCH (d)-[:DOCUMENT_VERSION]->(dv:D…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.
A Cypher shape whose p95 is worsening across the window and is at least 1 s now; spikes under a second are ignored.
The facts
| service | enrollsvc |
| plan_hash | 00000000 |
| queries | 696 |
| floor_ms | 1,000 |
| p95_early | 295.91 |
| p95_late | 2,975 |
| ratio | 10.05 |
| avg_ms | 45.97 |
| sample | MATCH (d:Document {tildaid: $document_id}) MATCH (d)-[:DOC_TO_ORG]-(o:Organization) MATCH (d)-[:DOCUMENT_VERSION]->(dv:DocumentVersion) WHERE dv.tildaid = COALESCE($version_id, d.primaryVersionId) AND (dv.uploadMethod IS NULL OR dv.uploadMethod <> ?) AND d.delpending = false AND d.primaryVersionId IS NOT NULL AND d.primaryVersionId <> ? // Admission covers only the basic workflow (linearize + page |
| rw | write |
| severity_capped | critical → warning: test27 is a test-tier cluster |
| discriminator | 00000000 |
| window | 2026-11-16 18:37:22 UTC → 2026-11-17 00:37:22 UTC |
The story — No enrollsvc deploy or hand change is recorded in the hour before this opened. Everything below is enrollsvc only.
- 2026-11-13 08:19:38 UTCthis finding enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)
- 2026-11-15 15:15:12 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/enrollsvc by hand (kubectl/v1.32.2 (darwin/amd64) kubernetes/67a30c0,gzip(gfe))
- 2026-11-17 00:00:00 UTClatency enrollsvc p95 peaked at 4.85 s
History
- 2026-11-15 19:16:14 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17200, 'fingerprint': '67b0fcdccd582d4ad4b7', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'enrollsvc', 'title': 'enrollsvc shape 00000000: p95 1.90 s now vs 10 ms earlier (190.7×, 222 runs)', 'facts': {'service': 'enrollsvc', 'plan_hash': '00000000', 'queries': 222, 'floor_ms': 1000, 'p95_early': 9.965811965811966, 'p95_late': 1899.9999999999998, 'ratio': 190.6518010291595, 'avg_ms': 46.648648648648646, 'sample': '// Same AIRun.status seeding and collect/UNWIND barrier as // count_in_progress_and_aborted_ai_runs; see the note there. MATCH (ai:AIRun) WHERE ai.status IN [?, ?] AND ( (ai.status = ? AND ($cutoff_ms <= ? OR COALESCE(ai.updatedat, ai.createdat) < $cutoff_ms)) OR (ai.status = ? AND ($aborted_cutoff_ms <= ? OR COALESCE(ai.updatedat, ai.createdat) < $aborted_cutoff_ms)) ) WITH collect(ai) AS adoptab', 'rw': 'write', 'window': [1794597162, 1794618762], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794597162&to=1794618762'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794597162&to=1794618762&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794618760, 'age_seconds': 7, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.6, 'status': 'resolved', 'first_seen': 1794557978, 'last_seen': 1794618762, 'resolved_at': 1794770174, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 7, 'narrative': 'A enrollsvc query shape on test27 has p95 1.9 s in the second half of the window against 10 ms in the first (190.7×, 222 executions). The shape reads: “// Same AIRun.status seeding and collect/UNWIND barrier as // count_in_progress_and_aborted_ai_runs; see the note there.…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794614877, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 212200, 'stale_s': 151416}
- 2026-11-17 00:18:51 UTCreopened x-ray · severity=warning was=warning title=enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs) level=low was_level=low finding={'id': 17200, 'fingerprint': '67b0fcdccd582d4ad4b7', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'enrollsvc', 'title': 'enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)', 'facts': {'service': 'enrollsvc', 'plan_hash': '00000000', 'queries': 696, 'floor_ms': 1000, 'p95_early': 295.9090909090905, 'p95_late': 2974.9999999999977, 'ratio': 10.053763440860221, 'avg_ms': 45.9683908045977, 'sample': 'MATCH (currentUser:User{deltimestamp:?,deltimestamp:?,email:$email}) WHERE (currentUser.isDummy IS NULL OR currentUser.isDummy <> true) OPTIONAL MATCH (supportUser:User{email:$supportUserEmail}) OPTIONAL MATCH (targetOrg:Organization{delpending:false, alias:$orgAlias}) // A deactivated (Inactive) org actor is not a membership (ENG-?). OPTIONAL MATCH (currentUser)-[:FOR_USER]-(orgActor:Actor{delpen', 'rw': 'write', 'window': [1794853131, 1794874731], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794853131&to=1794874731'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794853131&to=1794874731&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794874729, 'age_seconds': 9, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 24.5, 'status': 'open', 'first_seen': 1794557978, 'last_seen': 1794874731, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 8, 'narrative': 'A enrollsvc query shape on test27 has p95 1.9 s in the second half of the window against 10 ms in the first (190.7×, 222 executions). The shape reads: “// Same AIRun.status seeding and collect/UNWIND barrier as // count_in_progress_and_aborted_ai_runs; see the note there.…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794874731, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 316760, 'stale_s': 7}
- 2026-11-17 06:42:35 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17200, 'fingerprint': '67b0fcdccd582d4ad4b7', 'kind': 'shape.degrading', 'severity': 'warning', 'cluster': 'test27', 'subject': 'enrollsvc', 'title': 'enrollsvc shape 00000000: p95 2.97 s now vs 296 ms earlier (10.1×, 696 runs)', 'facts': {'service': 'enrollsvc', 'plan_hash': '00000000', 'queries': 696, 'floor_ms': 1000, 'p95_early': 295.9090909090905, 'p95_late': 2974.9999999999977, 'ratio': 10.053763440860221, 'avg_ms': 45.9683908045977, 'sample': 'MATCH (d:Document {tildaid: $document_id}) MATCH (d)-[:DOC_TO_ORG]-(o:Organization) MATCH (d)-[:DOCUMENT_VERSION]->(dv:DocumentVersion) WHERE dv.tildaid = COALESCE($version_id, d.primaryVersionId) AND (dv.uploadMethod IS NULL OR dv.uploadMethod <> ?) AND d.delpending = false AND d.primaryVersionId IS NOT NULL AND d.primaryVersionId <> ? // Admission covers only the basic workflow (linearize + page', 'rw': 'write', 'window': [1794854242, 1794875842], 'recent_deploys': [], 'severity_capped': 'critical → warning: test27 is a test-tier cluster', 'discriminator': '00000000'}, 'evidence': [{'label': 'Query shape', 'href': '/neo/shape/00000000?cluster=test27&from=1794854242&to=1794875842'}, {'label': 'Degrading shapes', 'href': '/neo/queries?cluster=test27&from=1794854242&to=1794875842&order=degrading'}], 'provenance': {'source': 'rollup', 'collected_at': 1794875753, 'age_seconds': 97, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 29.5, 'status': 'resolved', 'first_seen': 1794557978, 'last_seen': 1794875842, 'resolved_at': 1794897755, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': 'A enrollsvc query shape on test27 has p95 3.0 s in the second half of the window against 296 ms in the first (10.1×, 696 executions). The shape reads: “MATCH (d:Document {tildaid: $document_id}) MATCH (d)-[:DOC_TO_ORG]-(o:Organization) MATCH (d)-[:DOCUMENT_VERSION]->(dv:D…”. Slower can mean more data, a lost index, or lock contention — the shape page has the CPU/wait split. Shapes whose p95 stays under a second are never reported: below that nobody notices, whatever the ratio.', 'narrative_kind': 'template', 'discriminator': '00000000', 'level': 'low', 'told_level': None, 'episode_at': 1794874731, 'chat_thread': None, 'kind_label': 'Query shape degrading', 'group': 'Neo4j', 'age_s': 339784, 'stale_s': 21920}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794875753 |
| age_seconds | 97 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | shapes |