tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held)
warning
Error never seen before
test27
resolved
impact 26
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 2d ago · last observed 2d ago · observed 2× · #17485 · 82806bc56dc259782acc
What happened template
tmfsync logged “identical sync: document held” 6 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 19:10 UTC, raised at identical/pair_sync.py:326). Example trace: 32cac73727c15b60a6dff6e266c3f6cb.
An error shape appearing for the first time in 90 days, five or more times in an hour.
The facts
| service | tmfsync |
| caller | identical/pair_sync.py:326 |
| shape | identical sync: document held |
| count | 6 |
| error_class | — |
| first_seen | 2026-11-16 19:10:40 UTC (2d ago) |
| sample | identical sync: document held |
| trace | 32cac73727c15b60a6dff6e266c3f6cb |
| shape_hash | 4f91e2652e031ed2 |
| count_is_floor | no |
| discriminator | 4f91e2652e031ed2 |
| window | 2026-11-16 18:57:55 UTC → 2026-11-16 19:57:55 UTC |
What is failing in tmfsync — the bad events, decomposed
“Bad events” are HTTP 5xx answers from tmfsync. In this finding's window (2026-11-16 18:57:55 UTC → 2026-11-16 19:57:55 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 15× | Egress jobs OUT of safety-net re-drive budget with documents still NOT in eTMF — human action required (fix cause, then … since 2026-09-30 21:15:51 UTC |
| 14× | Ingress jobs OUT of safety-net re-drive budget with documents still NOT in Tilda — human action required (fix cause, the… since 2026-10-09 15:14:26 UTC |
| 14× | Egress dead-letter backlog — documents NOT in eTMF, need attention (fix cause, then replay via POST /configs/{id}/export… since 2026-09-30 21:15:50 UTC |
| 6× | identical sync: document held since 2026-11-16 19:10:40 UTC |
| 3× | QC issue DEAD-LETTERED — finding not annotated in eTMF (parent doc IS placed); fix cause, then re-export the parent doc … since 2026-11-16 20:39:36 UTC |
The story — 3 tmfsync changes landed in the hour before this opened — adjacency, not proof. Everything below is tmfsync only.
- 2026-11-16 18:40:42 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:53:59 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 18:58:41 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
- 2026-11-16 19:33:28 UTCthis finding tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held)
- 2026-11-16 20:24:14 UTChand change [email protected] changed apps/v1/namespaces/tilda/deployments/tmfsync by hand (kubectl/v1.34.1 (darwin/arm64) kubernetes/93248f9,gzip(gfe))
History
- 2026-11-16 19:33:28 UTCopened x-ray · severity=warning title=tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held) finding={'id': 17485, 'fingerprint': '82806bc56dc259782acc', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tmfsync', 'title': 'tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held)', 'facts': {'service': 'tmfsync', 'caller': 'identical/pair_sync.py:326', 'shape': 'identical sync: document held', 'count': 6, 'error_class': None, 'first_seen': 1794856240, 'sample': 'identical sync: document held', 'trace': '32cac73727c15b60a6dff6e266c3f6cb', 'shape_hash': '4f91e2652e031ed2', 'window': [1794854008, 1794857608], 'count_is_floor': False, 'discriminator': '4f91e2652e031ed2'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=test27&from=1794854008&to=1794857608&service=tmfsync'}, {'label': 'Logs', 'href': '/logs?cluster=test27&from=1794854008&to=1794857608&service=tmfsync&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/32cac73727c15b60a6dff6e266c3f6cb?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 22.6, 'status': 'open', 'first_seen': 1794857608, 'last_seen': 1794857608, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '4f91e2652e031ed2', 'level': 'medium', 'told_level': None, 'episode_at': 1794857608, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 40, 'stale_s': 40}
- 2026-11-16 20:12:56 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 17485, 'fingerprint': '82806bc56dc259782acc', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'test27', 'subject': 'tmfsync', 'title': 'tmfsync: a new error shape, 6× in the last hour, never seen in 90 days (identical sync: document held)', 'facts': {'service': 'tmfsync', 'caller': 'identical/pair_sync.py:326', 'shape': 'identical sync: document held', 'count': 6, 'error_class': None, 'first_seen': 1794856240, 'sample': 'identical sync: document held', 'trace': '32cac73727c15b60a6dff6e266c3f6cb', 'shape_hash': '4f91e2652e031ed2', 'window': [1794855475, 1794859075], 'count_is_floor': False, 'discriminator': '4f91e2652e031ed2'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=test27&from=1794855475&to=1794859075&service=tmfsync'}, {'label': 'Logs', 'href': '/logs?cluster=test27&from=1794855475&to=1794859075&service=tmfsync&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/32cac73727c15b60a6dff6e266c3f6cb?cluster=test27'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 25.6, 'status': 'resolved', 'first_seen': 1794857608, 'last_seen': 1794859075, 'resolved_at': 1794859976, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2, 'narrative': 'tmfsync logged “identical sync: document held” 6 times in the last hour and never before in 90 days on test27 (first seen 2026-11-16 19:10 UTC, raised at identical/pair_sync.py:326). Example trace: 32cac73727c15b60a6dff6e266c3f6cb.', 'narrative_kind': 'template', 'discriminator': '4f91e2652e031ed2', 'level': 'medium', 'told_level': None, 'episode_at': 1794857608, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 2456, 'stale_s': 989}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |