tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective
warning
Error budget burning
delta
open
impact 60
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 12d ago · last observed 1d ago · observed 16× · #16346 · 001f7f6c52a4cfea9392
What happened template
tmfsync is spending its error budget 8.4× faster than the objective allows over the last hour (0.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 26 bad events against 6,583 calls in the window.
Bad events are consuming the error budget faster than the objective allows.
The facts
| objective | 99.50 |
| objective_source | derived |
| achieved | 99.61 |
| calls | 6,583 |
| bad | 26.00 |
| c5xx | 26.00 |
| budget | 32.91 |
| spent | 0.79 |
| remaining_pct | 21.01 |
| burn_1h | 8.40 |
| burn_6h | 0.79 |
| verdict | burning |
| p95_ms | 48.76 |
| discriminator | |
| window | 2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC |
What is failing in tmfsync — the bad events, decomposed
“Bad events” are HTTP 5xx answers from tmfsync. In this finding's window (2026-11-17 04:40:41 UTC → 2026-11-17 10:40:41 UTC) there were 26; every line links to the grouped errors behind it.
GET /health/ready |
26 bad of 6583 calls · 100% of the service's 5xx |
The error shapes behind them
| 35× | Request completed — 503 since 2026-11-15 06:27:32 UTC |
| 32× | Neo4j health check failed — Cannot resolve address neo4j-neo4j-delta.neo4j-system.svc.cluster.local:7687 since 2026-11-15 06:27:35 UTC |
| 16× | Read query failed — Cannot resolve address neo4j-neo4j-delta.neo4j-system.svc.cluster.local:7687 since 2026-11-15 06:27:45 UTC |
| 15× | Unable to retrieve routing information since 2026-11-15 06:29:07 UTC |
| 11× | NEO4J EXECUTE_READ [FAILED] 0.00s — Cannot resolve address neo4j-neo4j-delta.neo4j-system.svc.cluster.local:7687 since 2026-11-15 06:27:45 UTC |
The story — No tmfsync deploy or hand change is recorded in the hour before this opened. Everything below is tmfsync only.
- 2026-11-06 06:16:40 UTCthis finding tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective
- 2026-11-15 00:00:00 UTCerrors tmfsync 5xx peaked at 35 in one five-minute bucket (median 0 in this window)
- 2026-11-15 06:28:58 UTCfinding · critical tmfsync: a new error shape, 35× in the last hour, never seen in 90 days (Request completed — ?)
- 2026-11-15 06:28:58 UTCfinding · warning tmfsync: a new error shape, 32× in the last hour, never seen in 90 days (Neo4j health check failed — Cannot resolve address neo4j-neo4j-delta.n)
- 2026-11-15 06:28:58 UTCfinding · warning tmfsync: a new error shape, 16× in the last hour, never seen in 90 days (Read query failed — Cannot resolve address neo4j-neo4j-delta.neo4j-sys)
- 2026-11-15 06:28:58 UTCfinding · warning tmfsync: a new error shape, 11× in the last hour, never seen in 90 days (NEO4J EXECUTE_READ [FAILED] ?00s — Cannot resolve address neo4j-neo4j-)
- 2026-11-15 06:28:58 UTCfinding · warning tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (Reconciliation sweep failed — Cannot resolve address neo4j-neo4j-delta)
- 2026-11-15 06:28:58 UTCfinding · warning tmfsync: a new error shape, 5× in the last hour, never seen in 90 days (NEO4J EXECUTE_READ [FAILED] ?01s — Cannot resolve address neo4j-neo4j-)
- 2026-11-16 05:11:58 UTCdeploy [email protected] deployed tmfsync
History
- 2026-11-15 06:36:59 UTCreopened x-ray · severity=warning was=warning title=tmfsync is burning its error budget at 6.5× (1 h) / 1.0× (6 h) against a 99.50% objective level=low was_level=low finding={'id': 16346, 'fingerprint': '001f7f6c52a4cfea9392', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tmfsync', 'title': 'tmfsync is burning its error budget at 6.5× (1 h) / 1.0× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 99.50551003108222, 'calls': 7078.0, 'bad': 35.0, 'c5xx': 35.0, 'budget': 35.39, 'spent': 0.9889799378355467, 'remaining_pct': 1.1020062164453281, 'burn_1h': 6.493506493506493, 'burn_6h': 0.9889799378355468, 'verdict': 'burning', 'p95_ms': 48.75185290763968, 'window': [1794703019, 1794724619], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=delta&from=1794703019&to=1794724619'}, {'label': 'tmfsync service page', 'href': '/service/tmfsync?cluster=delta&from=1794703019&to=1794724619'}], 'provenance': {'source': 'rollup', 'collected_at': 1794724587, 'age_seconds': 35, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 58.5, 'status': 'open', 'first_seen': 1793945800, 'last_seen': 1794724619, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': 'tmfsync is spending its error budget 14.0× faster than the objective allows over the last hour (2.8× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 989 bad events against 71,177 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794724619, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 778822, 'stale_s': 3}
- 2026-11-15 07:38:58 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16346, 'fingerprint': '001f7f6c52a4cfea9392', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tmfsync', 'title': 'tmfsync is burning its error budget at 7.0× (1 h) / 1.0× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 99.49842361708225, 'calls': 6978.0, 'bad': 35.0, 'c5xx': 35.0, 'budget': 34.89, 'spent': 1.0031527658354829, 'remaining_pct': 0.0, 'burn_1h': 6.993006993006993, 'burn_6h': 1.0031527658354829, 'verdict': 'burning', 'p95_ms': 48.71812274368231, 'window': [1794705433, 1794727033], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=delta&from=1794705433&to=1794727033'}, {'label': 'tmfsync service page', 'href': '/service/tmfsync?cluster=delta&from=1794705433&to=1794727033'}], 'provenance': {'source': 'rollup', 'collected_at': 1794726856, 'age_seconds': 180, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 63.5, 'status': 'resolved', 'first_seen': 1793945800, 'last_seen': 1794727033, 'resolved_at': 1794728338, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 15, 'narrative': 'tmfsync is spending its error budget 7.0× faster than the objective allows over the last hour (1.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 35 bad events against 6,978 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794724619, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 782541, 'stale_s': 1308}
- 2026-11-17 10:40:41 UTCreopened x-ray · severity=warning was=warning title=tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective level=low was_level=low finding={'id': 16346, 'fingerprint': '001f7f6c52a4cfea9392', 'kind': 'budget.burning', 'severity': 'warning', 'cluster': 'delta', 'subject': 'tmfsync', 'title': 'tmfsync is burning its error budget at 8.4× (1 h) / 0.8× (6 h) against a 99.50% objective', 'facts': {'objective': 99.5, 'objective_source': 'derived', 'achieved': 99.6050432933313, 'calls': 6583.0, 'bad': 26.0, 'c5xx': 26.0, 'budget': 32.915, 'spent': 0.7899134133373842, 'remaining_pct': 21.00865866626158, 'burn_1h': 8.40064620355412, 'burn_6h': 0.7899134133373842, 'verdict': 'burning', 'p95_ms': 48.76214931779856, 'window': [1794890441, 1794912041], 'recent_deploys': [], 'discriminator': ''}, 'evidence': [{'label': 'Error budgets', 'href': '/budgets?cluster=delta&from=1794890441&to=1794912041'}, {'label': 'tmfsync service page', 'href': '/service/tmfsync?cluster=delta&from=1794890441&to=1794912041'}], 'provenance': {'source': 'rollup', 'collected_at': 1794911456, 'age_seconds': 592, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 54.8, 'status': 'open', 'first_seen': 1793945800, 'last_seen': 1794912041, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 16, 'narrative': 'tmfsync is spending its error budget 7.0× faster than the objective allows over the last hour (1.0× over six hours). The objective is 99.5%, derived — a baseline, not a promise. 35 bad events against 6,978 calls in the window.', 'narrative_kind': 'template', 'discriminator': '', 'level': 'low', 'told_level': None, 'episode_at': 1794912041, 'chat_thread': None, 'kind_label': 'Error budget burning', 'group': 'Reliability', 'age_s': 966247, 'stale_s': 6}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794911456 |
| age_seconds | 592 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | budgets |