Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 23:24:36.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docsvc runs 34 foreground Neo4j queries per request (23,246 queries / 693 calls, last hour)

info N+1 fan-out production resolved impact 22 Owner: Prasad Limaye (backup: Harish Kumar) opened 26d ago · last observed 2d ago · observed 218× · #5502 · 7fe9dc0ecc5f4fcd8f13

What happened template

docsvc on production averaged 33.5 foreground Neo4j queries per API request over the last hour (23,246 queries for 693 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

One request that runs a large number of Neo4j queries.

The facts

servicedocsvc
qpr33.50
calls693.00
fg_queries23,246
bg_queries62.00
discriminator
window2026-11-16 15:14:38 UTC → 2026-11-16 16:14:38 UTC

What is failing in docsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-16 15:14:38 UTC → 2026-11-16 16:14:38 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

30× failed to fetch document, err: VersionId 10b81676-b9d2-4a9d-89ea-320ebf61c3c7 not found in document 693f6fbc-cec0-448c-9… since 2026-11-09 14:40:22 UTC
19× failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(16108973) already exis… since 2026-11-10 08:37:50 UTC
19× failed to bulk-write document nodes — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(16108973) alre… since 2026-11-10 08:37:50 UTC
3× access denied for this study since 2026-11-11 13:51:29 UTC
3× failed to get document: document fc8c6283-c350-44c9-87b0-a9f91e79761c, user [email protected], document not found: no… since 2026-11-09 17:11:09 UTC

All docsvc errors, grouped → · service page · its logs

History

  • 2026-11-14 17:10:50 UTCreopened x-ray · severity=info was=info title=docsvc runs 266 foreground Neo4j queries per request (16,759 queries / 63 calls, last hour) level=low was_level=low finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 266 foreground Neo4j queries per request (16,759 queries / 63 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 266.0, 'calls': 63.0, 'fg_queries': 16759.0, 'bg_queries': 58.0, 'window': [1794672650, 1794676250], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794672650&to=1794676250'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794672650&to=1794676250'}], 'provenance': {'source': 'rollup', 'collected_at': 1794676208, 'age_seconds': 49, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 18.0, 'status': 'open', 'first_seen': 1792764270, 'last_seen': 1794676250, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 213, 'narrative': 'docsvc on production averaged 40.7 foreground Neo4j queries per API request over the last hour (53,715 queries for 1,319 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794676250, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1911987, 'stale_s': 7}
  • 2026-11-14 18:37:55 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 216 foreground Neo4j queries per request (17,895 queries / 83 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 215.6, 'calls': 83.0, 'fg_queries': 17895.0, 'bg_queries': 65.0, 'window': [1794674009, 1794677609], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794674009&to=1794677609'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794674009&to=1794677609'}], 'provenance': {'source': 'rollup', 'collected_at': 1794677591, 'age_seconds': 25, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 23.1, 'status': 'resolved', 'first_seen': 1792764270, 'last_seen': 1794677609, 'resolved_at': 1794681475, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 214, 'narrative': 'docsvc on production averaged 215.6 foreground Neo4j queries per API request over the last hour (17,895 queries for 83 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794676250, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1917210, 'stale_s': 3871}
  • 2026-11-16 15:28:22 UTCreopened x-ray · severity=info was=info title=docsvc runs 82 foreground Neo4j queries per request (35,377 queries / 432 calls, last hour) level=low was_level=low finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 82 foreground Neo4j queries per request (35,377 queries / 432 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 81.9, 'calls': 432.0, 'fg_queries': 35377.0, 'bg_queries': 63.0, 'window': [1794839302, 1794842902], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794839302&to=1794842902'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794839302&to=1794842902'}], 'provenance': {'source': 'rollup', 'collected_at': 1794842820, 'age_seconds': 91, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.6, 'status': 'open', 'first_seen': 1792764270, 'last_seen': 1794842902, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 215, 'narrative': 'docsvc on production averaged 215.6 foreground Neo4j queries per API request over the last hour (17,895 queries for 83 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794842902, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 2078641, 'stale_s': 9}
  • 2026-11-16 16:32:01 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 5502, 'fingerprint': '7fe9dc0ecc5f4fcd8f13', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'production', 'subject': 'docsvc', 'title': 'docsvc runs 34 foreground Neo4j queries per request (23,246 queries / 693 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 33.5, 'calls': 693.0, 'fg_queries': 23246.0, 'bg_queries': 62.0, 'window': [1794842078, 1794845678], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=production&from=1794842078&to=1794845678'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=production&from=1794842078&to=1794845678'}], 'provenance': {'source': 'rollup', 'collected_at': 1794845636, 'age_seconds': 52, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 21.9, 'status': 'resolved', 'first_seen': 1792764270, 'last_seen': 1794845678, 'resolved_at': 1794846721, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 218, 'narrative': 'docsvc on production averaged 33.5 foreground Neo4j queries per API request over the last hour (23,246 queries for 693 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': 'low', 'episode_at': 1794842902, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 2082460, 'stale_s': 1052}

Notifications

  • 2026-11-16 16:32:19 UTCdigest-itemin-app,prod-notifications · sent
  • 2026-11-16 15:29:10 UTCdigest-itemin-app,prod-notifications · sent
  • 2026-11-14 18:38:46 UTCdigest-itemin-app,prod-notifications · sent
  • 2026-11-14 17:11:37 UTCdigest-itemin-app,prod-notifications · sent

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794845636
age_seconds52
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorfanout