← findings
neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors
warning
Log rule matched
production
resolved
impact 42
opened 50d ago · last observed 2d ago · observed 665× · #2907 · 3c4e3dd1a99e1b634c52
What happened template
neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors
Error lines a log rule watches for (a service, a namespace, a pattern), grouped per error shape with the count in the rule's window — one incident, not one message per line. Its level is the rule's.
The facts
| rule_id | 15 |
| rule_name | Production logs — Neo4j errors |
| level | medium |
| count | 16 |
| count_is_floor | no |
| window_min | 30 |
| shape | (empty) |
| sample | |
| trace | 7ae72ff1062d68f80e8531fa50373286 |
| caller | (no caller) |
| namespace | neo4j-system |
| log_severity | ERROR |
| first_ts | 1,794,828,955 |
| last_ts | 1,794,828,960 |
| pattern | |
| discriminator | 15/48b2f0579d65 |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-29 18:12:37 UTCthis finding neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors
- 2026-09-29 23:04:32 UTCfinding · warning ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h
- 2026-10-03 04:31:35 UTCfinding · warning HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
- 2026-10-13 15:54:07 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
- 2026-10-18 18:16:26 UTCfinding · critical docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
- 2026-10-19 06:46:26 UTCfinding · critical notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
- 2026-10-23 13:03:15 UTCfinding · info virtual-assistant: Neo4j queries per request 83.8 — 14.0× the fleet's usual 6.0
- 2026-10-23 13:42:32 UTCfinding · info dashql: p95 latency (ms) 3.77 s — 5.2× the fleet's usual 726 ms
- 2026-10-23 13:49:17 UTCfinding · info virtual-assistant Neo4j load 413 q/min vs a normal 4 q/min ± 4 q/min for Tue 10:00 UTC (3 of the last 4 checks)
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
History
- 2026-11-16 11:36:45 UTCreopened x-ray · severity=warning was=warning title=neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors level=medium was_level=medium finding={'id': 2907, 'fingerprint': '3c4e3dd1a99e1b634c52', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'neo4j', 'title': 'neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors', 'facts': {'rule_id': 15, 'rule_name': 'Production logs — Neo4j errors', 'level': 'medium', 'count': 16, 'count_is_floor': False, 'window_min': 30, 'shape': '(empty)', 'sample': '', 'trace': '7ae72ff1062d68f80e8531fa50373286', 'caller': '(no caller)', 'namespace': 'neo4j-system', 'log_severity': 'ERROR', 'first_ts': 1794828955, 'last_ts': 1794828960, 'pattern': '', 'discriminator': '15/48b2f0579d65'}, 'evidence': [{'label': 'neo4j error logs', 'href': '/logs?cluster=production&from=1794827205&to=1794829005&service=neo4j&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/7ae72ff1062d68f80e8531fa50373286?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #15 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790705557, 'last_seen': 1794829005, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 636, 'narrative': None, 'narrative_kind': None, 'discriminator': '15/48b2f0579d65', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794829005, 'chat_thread': 'xray-alert-11-748fd5a928-g-1793196027', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4123450, 'stale_s': 2}
- 2026-11-16 12:06:55 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2907, 'fingerprint': '3c4e3dd1a99e1b634c52', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'neo4j', 'title': 'neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors', 'facts': {'rule_id': 15, 'rule_name': 'Production logs — Neo4j errors', 'level': 'medium', 'count': 16, 'count_is_floor': False, 'window_min': 30, 'shape': '(empty)', 'sample': '', 'trace': '7ae72ff1062d68f80e8531fa50373286', 'caller': '(no caller)', 'namespace': 'neo4j-system', 'log_severity': 'ERROR', 'first_ts': 1794828955, 'last_ts': 1794828960, 'pattern': '', 'discriminator': '15/48b2f0579d65'}, 'evidence': [{'label': 'neo4j error logs', 'href': '/logs?cluster=production&from=1794828954&to=1794830754&service=neo4j&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/7ae72ff1062d68f80e8531fa50373286?cluster=production'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #15 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790705557, 'last_seen': 1794830754, 'resolved_at': 1794830815, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 665, 'narrative': None, 'narrative_kind': None, 'discriminator': '15/48b2f0579d65', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794829005, 'chat_thread': 'xray-alert-11-748fd5a928-g-1793196027', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4125261, 'stale_s': 64}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | log rule #15 over error_lines (BigQuery) |
| detector | — |