Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 3 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 12:49:52.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

docsvc runs 45 foreground Neo4j queries per request (17,242 queries / 384 calls, last hour)

info N+1 fan-out master resolved impact 15 Owner: Prasad Limaye (backup: Harish Kumar) opened 83d ago · last observed 1d ago · observed 177× · #242 · 9fafd3fb04cc02e67473

What happened template

docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (17,242 queries for 384 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

One request that runs a large number of Neo4j queries.

The facts

servicedocsvc
qpr44.90
calls384.00
fg_queries17,242
bg_queries67.00
discriminator
window2026-11-17 07:04:09 UTC → 2026-11-17 08:04:09 UTC

What is failing in docsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from docsvc. In this finding's window (2026-11-17 07:04:09 UTC → 2026-11-17 08:04:09 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

23× Request completed — 500 · could not update document metadata for doc id 0211da3b-b92d-4f58-a742-62624434f054: could not … since 2026-09-30 16:47:17 UTC
7× Request completed — 500 · could not update document metadata for doc id 5e8b6a30-080e-4d35-9878-e57e75108c3d: could not … since 2026-10-08 19:24:13 UTC
7× could not update document metadata for doc id b59d0e97-516c-41b7-a96b-2fc113fc1bbe: could not update the document: Eithe… since 2026-11-13 04:59:59 UTC
3× could not update document metadata for doc id 3b6cc256-1e7a-4e3d-af36-34264f3800fc: could not update document for catego… since 2026-11-13 04:59:57 UTC
1× message was never published because transaction was never committed since 2026-11-17 08:36:31 UTC

All docsvc errors, grouped → · service page · its logs

History

  • 2026-11-16 06:01:45 UTCreopened x-ray · severity=info was=info title=docsvc runs 52 foreground Neo4j queries per request (4,447 queries / 86 calls, last hour) level=low was_level=low finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 52 foreground Neo4j queries per request (4,447 queries / 86 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 51.7, 'calls': 86.0, 'fg_queries': 4447.0, 'bg_queries': 39.0, 'window': [1794805305, 1794808905], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794805305&to=1794808905'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794805305&to=1794808905'}], 'provenance': {'source': 'rollup', 'collected_at': 1794808448, 'age_seconds': 465, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 10.8, 'status': 'open', 'first_seen': 1787820848, 'last_seen': 1794808905, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 172, 'narrative': 'docsvc on master averaged 167.7 foreground Neo4j queries per API request over the last hour (68,092 queries for 406 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 6988065, 'stale_s': 8}
  • 2026-11-16 06:42:08 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 41 foreground Neo4j queries per request (4,497 queries / 110 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 40.9, 'calls': 110.0, 'fg_queries': 4497.0, 'bg_queries': 39.0, 'window': [1794806547, 1794810147], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794806547&to=1794810147'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794806547&to=1794810147'}], 'provenance': {'source': 'rollup', 'collected_at': 1794809991, 'age_seconds': 167, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.1, 'status': 'resolved', 'first_seen': 1787820848, 'last_seen': 1794810147, 'resolved_at': 1794811328, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 173, 'narrative': 'docsvc on master averaged 40.9 foreground Neo4j queries per API request over the last hour (4,497 queries for 110 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794808905, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 6990487, 'stale_s': 1188}
  • 2026-11-16 06:57:06 UTCreopened x-ray · severity=info was=info title=docsvc runs 45 foreground Neo4j queries per request (5,069 queries / 113 calls, last hour) level=low was_level=low finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 45 foreground Neo4j queries per request (5,069 queries / 113 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 44.9, 'calls': 113.0, 'fg_queries': 5069.0, 'bg_queries': 42.0, 'window': [1794808626, 1794812226], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794808626&to=1794812226'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794808626&to=1794812226'}], 'provenance': {'source': 'rollup', 'collected_at': 1794812153, 'age_seconds': 80, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 9.0, 'status': 'open', 'first_seen': 1787820848, 'last_seen': 1794812226, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc on master averaged 40.9 foreground Neo4j queries per API request over the last hour (4,497 queries for 110 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794812226, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 6991385, 'stale_s': 7}
  • 2026-11-16 07:17:55 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 45 foreground Neo4j queries per request (5,069 queries / 113 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 44.9, 'calls': 113.0, 'fg_queries': 5069.0, 'bg_queries': 42.0, 'window': [1794808626, 1794812226], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794808626&to=1794812226'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794808626&to=1794812226'}], 'provenance': {'source': 'rollup', 'collected_at': 1794812153, 'age_seconds': 80, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 14.0, 'status': 'resolved', 'first_seen': 1787820848, 'last_seen': 1794812226, 'resolved_at': 1794813475, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 174, 'narrative': 'docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (5,069 queries for 113 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794812226, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 6992634, 'stale_s': 1256}
  • 2026-11-16 12:29:31 UTCreopened x-ray · severity=info was=info title=docsvc runs 127 foreground Neo4j queries per request (36,521 queries / 287 calls, last hour) level=low was_level=low finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 127 foreground Neo4j queries per request (36,521 queries / 287 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 127.3, 'calls': 287.0, 'fg_queries': 36521.0, 'bg_queries': 36.0, 'window': [1794828571, 1794832171], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794828571&to=1794832171'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794828571&to=1794832171'}], 'provenance': {'source': 'rollup', 'collected_at': 1794831305, 'age_seconds': 876, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 8.5, 'status': 'open', 'first_seen': 1787820848, 'last_seen': 1794832171, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 175, 'narrative': 'docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (5,069 queries for 113 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794832171, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 7011333, 'stale_s': 10}
  • 2026-11-16 12:49:12 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 127 foreground Neo4j queries per request (36,521 queries / 287 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 127.3, 'calls': 287.0, 'fg_queries': 36521.0, 'bg_queries': 36.0, 'window': [1794828571, 1794832171], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794828571&to=1794832171'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794828571&to=1794832171'}], 'provenance': {'source': 'rollup', 'collected_at': 1794831305, 'age_seconds': 876, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 13.5, 'status': 'resolved', 'first_seen': 1787820848, 'last_seen': 1794832171, 'resolved_at': 1794833352, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 175, 'narrative': 'docsvc on master averaged 127.3 foreground Neo4j queries per API request over the last hour (36,521 queries for 287 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794832171, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 7012515, 'stale_s': 1192}
  • 2026-11-17 07:42:20 UTCreopened x-ray · severity=info was=info title=docsvc runs 51 foreground Neo4j queries per request (17,120 queries / 337 calls, last hour) level=low was_level=low finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 51 foreground Neo4j queries per request (17,120 queries / 337 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 50.8, 'calls': 337.0, 'fg_queries': 17120.0, 'bg_queries': 64.0, 'window': [1794897740, 1794901340], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794897740&to=1794901340'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794897740&to=1794901340'}], 'provenance': {'source': 'rollup', 'collected_at': 1794900767, 'age_seconds': 583, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 10.0, 'status': 'open', 'first_seen': 1787820848, 'last_seen': 1794901340, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 176, 'narrative': 'docsvc on master averaged 127.3 foreground Neo4j queries per API request over the last hour (36,521 queries for 287 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794901340, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 7080502, 'stale_s': 10}
  • 2026-11-17 08:26:41 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 242, 'fingerprint': '9fafd3fb04cc02e67473', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'master', 'subject': 'docsvc', 'title': 'docsvc runs 45 foreground Neo4j queries per request (17,242 queries / 384 calls, last hour)', 'facts': {'service': 'docsvc', 'qpr': 44.9, 'calls': 384.0, 'fg_queries': 17242.0, 'bg_queries': 67.0, 'window': [1794899049, 1794902649], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=master&from=1794899049&to=1794902649'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=master&from=1794899049&to=1794902649'}], 'provenance': {'source': 'rollup', 'collected_at': 1794901996, 'age_seconds': 663, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 15.3, 'status': 'resolved', 'first_seen': 1787820848, 'last_seen': 1794902649, 'resolved_at': 1794904001, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 177, 'narrative': 'docsvc on master averaged 44.9 foreground Neo4j queries per API request over the last hour (17,242 queries for 384 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794901340, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 7083164, 'stale_s': 1363}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794901996
age_seconds663
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorfanout