virtual-assistant: a new error shape, 23× in the last hour, never seen in 90 days ([PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to)
warning
Error never seen before
automation
open
impact 27
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 2d ago · last observed 2d ago · observed 2× · #17385 · 6f47f4332cb858f1a59e
What happened template
virtual-assistant logged “[PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to record match via docsvc API for doc ?” 23 times in the last hour and never before in 90 days on automation (first seen 2026-11-16 09:05 UTC, raised at workflow/placeholder_match.py:426). Example trace: a917e57ad37a8de9cb3d56c80f0176de.
An error shape appearing for the first time in 90 days, five or more times in an hour.
The facts
| service | virtual-assistant |
| caller | workflow/placeholder_match.py:426 |
| shape | [PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to record match via docsvc API for doc ? |
| count | 23 |
| error_class | — |
| first_seen | 2026-11-16 09:05:16 UTC (2d ago) |
| sample | [PlaceholderMatch][placeholder_id=b7e43a75-f27f-4650-ae8c-efecb6cfce75] Failed to record match: Failed to record match via docsvc API for doc b00d531c-ed60-43b9-a80b-c8bcb29a6982 |
| trace | a917e57ad37a8de9cb3d56c80f0176de |
| shape_hash | d0157ca47a458a97 |
| count_is_floor | no |
| discriminator | d0157ca47a458a97 |
| window | 2026-11-16 08:50:12 UTC → 2026-11-16 09:50:12 UTC |
What is failing in virtual-assistant — the bad events, decomposed
“Bad events” are HTTP 5xx answers from virtual-assistant. In this finding's window (2026-11-16 08:50:12 UTC → 2026-11-16 09:50:12 UTC) there were 0; every line links to the grouped errors behind it.
The error shapes behind them
| 6824× | Failed to fetch document info: 403 - {"message":"user with email not found"} since 2026-10-13 00:17:51 UTC |
| 190× | [DocumentSubscriber] event handler permanent failure — acking to drop (no redelivery loop): event not found: 0c9b356b-27… since 2026-10-19 00:36:33 UTC |
| 116× | [DocumentSubscriber] event handler permanent failure — acking to drop (no redelivery loop): bulk_create failed after 3 a… since 2026-10-13 00:41:42 UTC |
| 28× | GET request failed for http://docsvc-automation.tilda.svc.cluster.local:80/api/documents/76630ee6-7970-4325-a614-e3b0d7d… since 2026-11-01 01:12:16 UTC |
| 24× | HTTP error occurred: Client error '403 Forbidden' for url 'http://docsvc-automation.tilda.svc.cluster.local/api/document… since 2026-11-01 01:12:16 UTC |
All virtual-assistant errors, grouped → · service page · its logs
The story — 1 virtual-assistant change landed in the hour before this opened — adjacency, not proof. Everything below is virtual-assistant only.
- 2026-11-16 08:21:53 UTChand change [email protected] changed apps/v1/namespaces/matrix/deployments/virtual-assistant by hand (kubectl/v1.33.3 (darwin/arm64) kubernetes/80779bd,gzip(gfe))
- 2026-11-16 09:06:49 UTCthis finding virtual-assistant: a new error shape, 23× in the last hour, never seen in 90 days ([PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to)
- 2026-11-16 10:00:00 UTClatency virtual-assistant p95 peaked at 4.81 s
History
- 2026-11-16 09:06:49 UTCopened x-ray · severity=warning title=virtual-assistant: a new error shape, 18× in the last hour, never seen in 90 days ([PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to) finding={'id': 17385, 'fingerprint': '6f47f4332cb858f1a59e', 'kind': 'error.novel', 'severity': 'warning', 'cluster': 'automation', 'subject': 'virtual-assistant', 'title': 'virtual-assistant: a new error shape, 18× in the last hour, never seen in 90 days ([PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to)', 'facts': {'service': 'virtual-assistant', 'caller': 'workflow/placeholder_match.py:426', 'shape': '[PlaceholderMatch][placeholder_id=?] Failed to record match: Failed to record match via docsvc API for doc ?', 'count': 18, 'error_class': None, 'first_seen': 1794819916, 'sample': '[PlaceholderMatch][placeholder_id=a9b69089-8131-4a89-bcd3-d4dcd2c2b0c6] Failed to record match: Failed to record match via docsvc API for doc fce7a0a7-40ac-4001-b94b-901a4b786685', 'trace': 'a917e570ff1d8339cb3d56ca2cd8f653', 'shape_hash': 'd0157ca47a458a97', 'window': [1794816409, 1794820009], 'count_is_floor': False, 'discriminator': 'd0157ca47a458a97'}, 'evidence': [{'label': 'Errors, grouped', 'href': '/errors?cluster=automation&from=1794816409&to=1794820009&service=virtual-assistant'}, {'label': 'Logs', 'href': '/logs?cluster=automation&from=1794816409&to=1794820009&service=virtual-assistant&severity=ERROR'}, {'label': 'Example trace', 'href': '/trace/a917e570ff1d8339cb3d56ca2cd8f653?cluster=automation'}], 'provenance': {'source': 'live', 'collected_at': None, 'age_seconds': None, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': 'Cloud Logging, last hour, grouped by (service, caller, shape)'}, 'impact': 26.1, 'status': 'open', 'first_seen': 1794820009, 'last_seen': 1794820009, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': 'd0157ca47a458a97', 'level': 'medium', 'told_level': None, 'episode_at': 1794820009, 'chat_thread': None, 'kind_label': 'Error never seen before', 'group': 'Errors', 'age_s': 53, 'stale_s': 53}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | live |
| collected_at | None |
| age_seconds | None |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | Cloud Logging, last hour, grouped by (service, caller, shape) |
| detector | errors_novel |