enrollsvc: Neo4j queries per request 50.0 — 6.9× the fleet's usual 7.3
info
Service outlier
test26
open
impact 10
Owner: Suresh Boddapati (backup: Sambasiva Bandarupalli)
opened 74d ago · last observed 22h ago · observed 372× · #801 · 23ce938a23655761d4ea
What happened template
enrollsvc's Neo4j queries per request is 50.0 against the 5 other services' median of 7.3 (8.6 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.
A service unlike the rest of the fleet on one measure (robust z against its peers).
The facts
| measures | {'measure': 'Neo4j queries per request', 'value': 49.98924731182796, 'value_fmt': '50.0', 'median': 7.284507042253521, 'median_fmt': '7.3', 'mad': 4.954817037396794, 'z': 8.61883293515335, 'direction': 'above', 'peers': 5} |
| measure | Neo4j queries per request |
| value | 49.99 |
| value_fmt | 50.0 |
| median | 7.28 |
| median_fmt | 7.3 |
| z | 8.62 |
| direction | above |
| peers | 5 |
| rule | {"min_score": 3.5, "min_relative_pct": 35, "warn_on": []} |
| discriminator | |
| window | 2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC |
What is failing in enrollsvc — the bad events, decomposed
“Bad events” are HTTP 5xx answers from enrollsvc. In this finding's window (2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 1566× | could not add chart review study node while binding study to org — Neo4jError: Neo.ClientError.Schema.ConstraintValidati… since 2026-10-09 11:44:30 UTC |
| 19× | failed to create node — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12181421) already exists wit… since 2026-10-30 19:23:45 UTC |
| 19× | failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12181421) already exis… since 2026-10-30 19:23:45 UTC |
| 18× | failed to create node — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exists wit… since 2026-11-02 07:19:50 UTC |
| 18× | failed to process db result — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed (Node(12619656) already exis… since 2026-11-02 07:19:50 UTC |
History
- 2026-11-16 01:43:50 UTCreopened x-ray · severity=info was=info title=enrollsvc: p95 latency (ms) 1.45 s — 7.3× the fleet's usual 198 ms level=low was_level=low finding={'id': 801, 'fingerprint': '23ce938a23655761d4ea', 'kind': 'outlier.service', 'severity': 'info', 'cluster': 'test26', 'subject': 'enrollsvc', 'title': "enrollsvc: p95 latency (ms) 1.45 s — 7.3× the fleet's usual 198 ms", 'facts': {'measures': [{'measure': 'p95 latency (ms)', 'value': 1449.9999999999957, 'value_fmt': '1.45 s', 'median': 198.12, 'median_fmt': '198 ms', 'mad': 220.9926495, 'z': 5.664803797014957, 'direction': 'above', 'peers': 5}], 'measure': 'p95 latency (ms)', 'value': 1449.9999999999957, 'value_fmt': '1.45 s', 'median': 198.12, 'median_fmt': '198 ms', 'z': 5.664803797014957, 'direction': 'above', 'peers': 5, 'window': [1794771830, 1794793430], 'recent_deploys': [], 'rule': {'min_score': 3.5, 'min_relative_pct': 35, 'warn_on': []}, 'discriminator': ''}, 'evidence': [{'label': 'Outliers', 'href': '/outliers?cluster=test26&from=1794771830&to=1794793430'}, {'label': 'enrollsvc service page', 'href': '/service/enrollsvc?cluster=test26&from=1794771830&to=1794793430'}, {'label': 'Tune this rule', 'href': '/alerts/system/outliers'}], 'provenance': {'source': 'rollup', 'collected_at': 1794793430, 'age_seconds': 1, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 5.5, 'status': 'open', 'first_seen': 1788533064, 'last_seen': 1794793430, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 282, 'narrative': "enrollsvc's Neo4j queries per request is 5.5 against the 5 other services' median of 1.5 (9.3 robust MADs above, six-hour window). This compares the service with its peers, not with its own history — the seasonal detector does that.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794793430, 'chat_thread': None, 'kind_label': 'Service outlier', 'group': 'Anomalies', 'age_s': 6260367, 'stale_s': 1}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794912040 |
| age_seconds | 2 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | outliers |