docsvc runs 34 foreground Neo4j queries per request (23,246 queries / 693 calls, last hour)
info
N+1 fan-out
production
resolved
impact 22
Owner: Prasad Limaye (backup: Harish Kumar)
opened 26d ago · last observed 2d ago · observed 218× · #5502 · 7fe9dc0ecc5f4fcd8f13
What happened template
docsvc on production averaged 33.5 foreground Neo4j queries per API request over the last hour (23,246 queries for 693 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.
One request that runs a large number of Neo4j queries.
The facts
| service | docsvc |
| qpr | 33.50 |
| calls | 693.00 |
| fg_queries | 23,246 |
| bg_queries | 62.00 |
| discriminator | |
| window | 2026-11-16 15:14:38 UTC → 2026-11-16 16:14:38 UTC |
What is failing in docsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 15:14:38 UTC → 2026-11-16 16:14:38 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 30× | failed to fetch document, err: VersionId 10b81676-b9d2-4a9d-89ea-320ebf61c3c7 not found in document 693f6fbc-cec0-448c-9… since 2026-11-09 14:40:22 UTC |
| 19× | failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(16108973) already exis… since 2026-11-10 08:37:50 UTC |
| 19× | failed to bulk-write document nodes — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(16108973) alre… since 2026-11-10 08:37:50 UTC |
| 3× | access denied for this study since 2026-11-11 13:51:29 UTC |
| 3× | failed to get document: document fc8c6283-c350-44c9-87b0-a9f91e79761c, user [email protected], document not found: no… since 2026-11-09 17:11:09 UTC |
History
- 2026-11-14 17:10:50 UTCreopened x-ray · severity=info was=info title=docsvc runs 266 foreground Neo4j queries per request (16,759 queries / 63 calls, last hour) level=low was_level=low finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 266 foreground Neo4j queries per request (16,759 queries / 63 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 266.0, 'calls': 63.0, 'fg_queries': 16759.0, 'bg_queries': 58.0, 'window': [1794672650, 1794676250], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794672650&to=1794676250'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794672650&to=1794676250'}], 'provenance': {'source': 'rollup', 'collected_at': 1794676208, 'age_seconds': 49, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 18.0, 'status': 'open', 'first_seen': 1792764270, 'last_seen': 1794676250, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 213, 'narrative': 'docsvc on production averaged 40.7 foreground Neo4j queries per API request over the last hour (53,715 queries for 1,319 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794676250, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1911987, 'stale_s': 7}
- 2026-11-14 18:37:55 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 216 foreground Neo4j queries per request (17,895 queries / 83 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 215.6, 'calls': 83.0, 'fg_queries': 17895.0, 'bg_queries': 65.0, 'window': [1794674009, 1794677609], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794674009&to=1794677609'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794674009&to=1794677609'}], 'provenance': {'source': 'rollup', 'collected_at': 1794677591, 'age_seconds': 25, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 23.1, 'status': 'resolved', 'first_seen': 1792764270, 'last_seen': 1794677609, 'resolved_at': 1794681475, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 214, 'narrative': 'docsvc on production averaged 215.6 foreground Neo4j queries per API request over the last hour (17,895 queries for 83 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794676250, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1917210, 'stale_s': 3871}
- 2026-11-16 15:28:22 UTCreopened x-ray · severity=info was=info title=docsvc runs 82 foreground Neo4j queries per request (35,377 queries / 432 calls, last hour) level=low was_level=low finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 82 foreground Neo4j queries per request (35,377 queries / 432 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 81.9, 'calls': 432.0, 'fg_queries': 35377.0, 'bg_queries': 63.0, 'window': [1794839302, 1794842902], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794839302&to=1794842902'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794839302&to=1794842902'}], 'provenance': {'source': 'rollup', 'collected_at': 1794842820, 'age_seconds': 91, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.6, 'status': 'open', 'first_seen': 1792764270, 'last_seen': 1794842902, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 215, 'narrative': 'docsvc on production averaged 215.6 foreground Neo4j queries per API request over the last hour (17,895 queries for 83 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794842902, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 2078641, 'stale_s': 9}
- 2026-11-16 16:32:01 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 34 foreground Neo4j queries per request (23,246 queries / 693 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 33.5, 'calls': 693.0, 'fg_queries': 23246.0, 'bg_queries': 62.0, 'window': [1794842078, 1794845678], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794842078&to=1794845678'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794842078&to=1794845678'}], 'provenance': {'source': 'rollup', 'collected_at': 1794845636, 'age_seconds': 52, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 21.9, 'status': 'resolved', 'first_seen': 1792764270, 'last_seen': 1794845678, 'resolved_at': 1794846721, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 218, 'narrative': 'docsvc on production averaged 33.5 foreground Neo4j queries per API request over the last hour (23,246 queries for 693 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794842902, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 2082460, 'stale_s': 1052}
Notifications
- 2026-11-16 16:32:19 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-16 15:29:10 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-14 18:38:46 UTCdigest-itemin-app,prod-notifications · sent
- 2026-11-14 17:11:37 UTCdigest-itemin-app,prod-notifications · sent
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794845636 |
| age_seconds | 52 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | fanout |