Xray
daily rollups 1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 14 days▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 02:03:00.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

execsvc runs 37 foreground Neo4j queries per request (12,049 queries / 327 calls, last hour)

info N+1 fan-out test26 resolved impact 12 Owner: Madhava Mukkamala (backup: Shashank Agrawal) opened 23d ago · last observed 2d ago · observed 22× · #6291 · 157cabde901ffb8486f9

What happened template

execsvc on test26 averaged 36.8 foreground Neo4j queries per API request over the last hour (12,049 queries for 327 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.

One request that runs a large number of Neo4j queries.

The facts

serviceexecsvc
qpr36.80
calls327.00
fg_queries12,049
bg_queries497.00
discriminator
window2026-11-16 05:57:06 UTC → 2026-11-16 06:57:06 UTC

What is failing in execsvc — the bad events, decomposed

“Bad events” are HTTP 5xx answers from execsvc. In this finding's window (2026-11-16 05:57:06 UTC → 2026-11-16 06:57:06 UTC) there were 0; every line links to the grouped errors behind it.

The error shapes behind them

21× error anchor visit index has not been set. Does this study have an anchor visit? continuing without setting day/date ran… since 2026-11-06 00:02:19 UTC
12× error anchor visit index has not been set. Does this study have an anchor visit? continuing without setting day/date ran… since 2026-11-12 15:37:28 UTC
3× could not process study site bundle import — source and target study are the same (d4ccf900-31a7-47fc-83bf-37afc98f7217)… since 2026-11-16 05:52:58 UTC

All execsvc errors, grouped → · service page · its logs

History

  • 2026-11-16 05:33:15 UTCreopened x-ray · severity=info was=info title=execsvc runs 26 foreground Neo4j queries per request (1,538 queries / 60 calls, last hour) level=low was_level=low finding={'id': 6291, 'fingerprint': '157cabde901ffb8486f9', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'test26', 'subject': 'execsvc', 'title': 'execsvc runs 26 foreground Neo4j queries per request (1,538 queries / 60 calls, last hour)', 'facts': {'service': 'execsvc', 'qpr': 25.6, 'calls': 60.0, 'fg_queries': 1538.0, 'bg_queries': 198.0, 'window': [1794803595, 1794807195], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=test26&from=1794803595&to=1794807195'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=test26&from=1794803595&to=1794807195'}], 'provenance': {'source': 'rollup', 'collected_at': 1794807195, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 5.2, 'status': 'open', 'first_seen': 1792999167, 'last_seen': 1794807195, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 18, 'narrative': 'execsvc on test26 averaged 59.8 foreground Neo4j queries per API request over the last hour (5,984 queries for 100 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794807195, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1808036, 'stale_s': 8}
  • 2026-11-16 18:07:03 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 6291, 'fingerprint': '157cabde901ffb8486f9', 'kind': 'n1.fanout', 'severity': 'info', 'cluster': 'test26', 'subject': 'execsvc', 'title': 'execsvc runs 37 foreground Neo4j queries per request (12,049 queries / 327 calls, last hour)', 'facts': {'service': 'execsvc', 'qpr': 36.8, 'calls': 327.0, 'fg_queries': 12049.0, 'bg_queries': 497.0, 'window': [1794808626, 1794812226], 'discriminator': ''}, 'evidence': [{'label': 'Neo4j by service', 'href': '/overview?cluster=test26&from=1794808626&to=1794812226'}, {'label': 'Journey (per-request Cypher)', 'href': '/journey?cluster=test26&from=1794808626&to=1794812226'}], 'provenance': {'source': 'rollup', 'collected_at': 1794812226, 'age_seconds': 8, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.5, 'status': 'resolved', 'first_seen': 1792999167, 'last_seen': 1794812226, 'resolved_at': 1794852423, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 22, 'narrative': 'execsvc on test26 averaged 36.8 foreground Neo4j queries per API request over the last hour (12,049 queries for 327 calls). Dozens of queries per request is the N+1 signature: a round-trip per row instead of one query. The Journey page shows which screens pay for it, request by request.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794807195, 'chat_thread': None, 'kind_label': 'N+1 fan-out', 'group': 'Journey', 'age_s': 1853265, 'stale_s': 40206}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcerollup
collected_at1794812226
age_seconds8
staleFalse
shownNone
totalNone
truncatedFalse
coverage_fromNone
retain_daysNone
noteNone
detectorfanout