ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal
critical
Log rule matched
production
resolved
impact 72
Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed)
opened 3d ago · last observed 2d ago · observed 19× · #17292 · b3b32d5f3e6abfd5d7f5
What happened template
ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal
Error lines a log rule watches for (a service, a namespace, a pattern), grouped per error shape with the count in the rule's window — one incident, not one message per line. Its level is the rule's.
First look — investigated before anyone looked
Cause not clear from logs and recent changes; ontologysvc logged 1 traceback at 05:42 UTC on 2026-11-15, inside uvicorn's lifespan receive (lifespan/on.py:134, waiting on receive_queue.get()), with no app frames in the sample.
Checked: no ontologysvc deploys in the 24 h before (running build deployed 2026-11-07 13:56 UTC by [email protected]), and no matching code in ontologysvc/.
This error shape has been seen since 2026-08-26, and a similar lifespan traceback (finding 3732) was resolved in October.
The only nearby finding, a cert-manager-cainjector warning 4 min earlier, has no link to this service.
Next step: check whether an ontologysvc pod was stopped or restarted at 05:42 UTC; if so, this is a one-off shutdown traceback, not a service failure.
low confidence
claude-opus-5-5 · 3d ago · $0.03 — every link above is evidence it read; nothing else was claimed.
The facts
| rule_id | 13 |
| rule_name | Production logs — panic / fatal |
| level | high |
| count | 1 |
| count_is_floor | no |
| window_min | 10 |
| shape | Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw |
| sample | Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan await receive() File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get |
| trace | — |
| caller | lifespan/on.py:134 |
| namespace | tilda |
| log_severity | ERROR |
| first_ts | 1,794,805,925 |
| last_ts | 1,794,805,925 |
| pattern | (?i)\b(panic|fatal)\b|traceback \(most recent call last\) |
| discriminator | 13/3ad81281e690 |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-11-15 05:43:29 UTCthis finding ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
History
- 2026-11-15 05:43:29 UTCopened x-ray · severity=critical title=ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794721370, 'last_ts': 1794721370, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794720809&to=1794721409&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794721409, 'last_seen': 1794721409, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': None, 'episode_at': 1794721409, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 1, 'stale_s': 1}
- 2026-11-15 05:53:29 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794721370, 'last_ts': 1794721370, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794721349&to=1794721949&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'resolved', 'first_seen': 1794721409, 'last_seen': 1794721949, 'resolved_at': 1794722009, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794721409, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 601, 'stale_s': 61}
- 2026-11-16 05:13:11 UTCreopened x-ray · severity=critical was=critical title=ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal level=high was_level=high finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794805925, 'last_ts': 1794805925, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794805391&to=1794805991&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794721409, 'last_seen': 1794805991, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794805991, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 84583, 'stale_s': 1}
- 2026-11-16 05:22:13 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794805925, 'last_ts': 1794805925, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794805873&to=1794806473&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'resolved', 'first_seen': 1794721409, 'last_seen': 1794806473, 'resolved_at': 1794806533, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 19, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794805991, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 85125, 'stale_s': 61}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | log rule #13 over error_lines (BigQuery) |
| detector | — |