neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2
warning
Pod keeps restarting
delta
open
impact 31
opened 12d ago · last observed 2d ago · observed 9× · #16838 · 59daeba4073ae34926d4
What happened template
neo4j-differential-backup-delta in neo4j-system on delta has restarted in place 4 times since 2026-11-17 10:15 UTC (4 in the last 30 minutes, 4 in 24 hours), most recently at 2026-11-17 10:25 UTC. Kubernetes' account of it: 104 sched events, 3 crash events. Last event: BackOff — “Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe31”. It is running image 5.26.28-diff-backup-2. Containers restarting: neo4j-differential-backup ×4. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.
A workload's containers restart in place over and over — crash loop, OOM kills, liveness kills or image pulls failing.
The facts
| workload | neo4j-differential-backup-delta |
| namespace | neo4j-system |
| restarts_30m | 4 |
| restarts_1h | 4 |
| restarts_24h | 4 |
| restarts_this_run | 4 |
| run_minutes | 10 |
| since | 1,794,910,500 |
| last_restart_at | 1,794,911,100 |
| pods | 1 |
| containers | {"neo4j-differential-backup": 4} |
| reasons | {"sched": 104, "crash": 3} |
| dominant_reason | unschedulable |
| image | us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/neo4j-differential-backup:5.26.28-diff-backup-2 |
| last_event | {"reason": "BackOff", "message": "Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe3173d241)", "at": 1794910904} |
| share_of_api_calls_24h | 0.00 |
| discriminator | |
| window | 2026-11-17 10:15:00 UTC → 2026-11-17 10:40:41 UTC |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-07 13:16:46 UTCthis finding neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2
- 2026-11-14 09:31:36 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-04366d86-rcf6 (1× in 30 min)
- 2026-11-14 09:54:26 UTCfinding · critical onboardingsvc GET /api/organizations/teammatesDemoSandbox: permission denied — 32 of 32 calls (100 %; 401×32) in the last 15 min — 164× its usual 0.2
- 2026-11-14 10:08:52 UTCfinding · info 1,211 new Cypher shapes today on delta vs a median of 239 a day — literals inlined somewhere
- 2026-11-14 12:05:42 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-04366d86-4t4b (1× in 30 min)
- 2026-11-14 13:24:15 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/v2/study/:id: permission denied — 29 of 29 calls (100 %; 401×29) in the last 15 min — it usually has none
- 2026-11-14 15:10:15 UTCdeploy automated deployed studiosvc-infra-delta
- 2026-11-15 00:58:00 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-htzz (1× in 30 min)
- 2026-11-15 06:01:24 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-ktl6 (1× in 30 min)
- 2026-11-15 06:01:24 UTCfinding · info ReadOnlyFileSystemDetected: gk3-delta-nap-d9uprtrn-c9db01cf-adpl (1× in 30 min)
- 2026-11-15 06:25:00 UTCerrors cluster 5xx peaked at 35 in one five-minute bucket (median 0 in this window)
- 2026-11-16 05:11:39 UTCdeploy [email protected] deployed asm
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:11:41 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:11:42 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:11:45 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:11:46 UTCdeploy [email protected] deployed litellm
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:11:47 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:11:48 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:11:49 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:11:50 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:11:51 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:11:53 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:11:54 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:11:55 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:11:56 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:11:57 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 16:55:00 UTClatency cluster p95 peaked at 7.14 s
History
- 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2 level=low was_level=low finding={'id': 16838, 'fingerprint': '59daeba4073ae34926d4', 'kind': 'pod.unstable', 'severity': 'warning', 'cluster': 'delta', 'subject': 'neo4j-system/neo4j-differential-backup-delta', 'title': 'neo4j-differential-backup-delta on delta has restarted 4 times in 10 min — unschedulable, image 5.26.28-diff-backup-2', 'facts': {'workload': 'neo4j-differential-backup-delta', 'namespace': 'neo4j-system', 'restarts_30m': 4, 'restarts_1h': 4, 'restarts_24h': 4, 'restarts_this_run': 4, 'run_minutes': 10, 'since': 1794910500, 'last_restart_at': 1794911100, 'pods': 1, 'containers': {'neo4j-differential-backup': 4}, 'reasons': {'sched': 104, 'crash': 3}, 'dominant_reason': 'unschedulable', 'image': 'us-central1-docker.pkg.dev/tilda-dev-325721/artifacts/services/neo4j-differential-backup:5.26.28-diff-backup-2', 'last_event': {'reason': 'BackOff', 'message': 'Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29915180-wmvtw_neo4j-system(3b547b5e-8f5d-4404-91e7-1dbe3173d241)', 'at': 1794910904}, 'share_of_api_calls_24h': 0.0, 'recent_deploys': [], 'window': [1794910500, 1794912041], 'discriminator': ''}, 'evidence': [{'label': 'Reliability', 'href': '/reliability?cluster=delta&range=24h'}, {'label': 'Pods', 'href': '/pods?cluster=delta'}, {'label': 'neo4j-differential-backup-delta error logs', 'href': '/logs?cluster=delta&service=neo4j-differential-backup-delta&severity=ERROR&range=24h'}], 'provenance': {'source': 'x-ray', 'note': 'pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes'}, 'impact': 26.0, 'status': 'open', 'first_seen': 1794057406, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 9, 'narrative': "neo4j-differential-backup-delta in neo4j-system on delta has restarted in place 4 times since 2026-11-14 06:35 UTC (4 in the last 30 minutes, 4 in 24 hours), most recently at 2026-11-14 06:40 UTC. Kubernetes' account of it: 3 sched events, 3 crash events. Last event: BackOff — “Back-off restarting failed container neo4j-differential-backup in pod neo4j-differential-backup-delta-29910640-9lc2h_neo4j-system(df9b7269-8ef9-4867-ad4e-a2dec6”. It is running image 5.26.28-diff-backup-2. Containers restarting: neo4j-differential-backup ×4. The workload carried 0.0% of the cluster's API calls in the last day. Restarts are counted from the container restart counter, so a rollout or a spot preemption does not count; a crash does. It resolves on its own after thirty quiet minutes.", 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794912041, 'chat_thread': None, 'kind_label': 'Pod keeps restarting', 'group': 'Reliability', 'age_s': 854749, 'stale_s': 114}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | pod_rollup: Monitoring restart_count deltas + Kubernetes events, per five minutes |
| detector | pod_unstable |