Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 03:44:31.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal

critical Log rule matched production resolved impact 72 Owner: Sambasiva Bandarupalli (backup: Ahmed Aburaed) opened 3d ago · last observed 2d ago · observed 19× · #17292 · b3b32d5f3e6abfd5d7f5

What happened template

ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal

Error lines a log rule watches for (a service, a namespace, a pattern), grouped per error shape with the count in the rule's window — one incident, not one message per line. Its level is the rule's.

First look — investigated before anyone looked

Cause not clear from logs and recent changes; ontologysvc logged 1 traceback at 05:42 UTC on 2026-11-15, inside uvicorn's lifespan receive (lifespan/on.py:134, waiting on receive_queue.get()), with no app frames in the sample.
Checked: no ontologysvc deploys in the 24 h before (running build deployed 2026-11-07 13:56 UTC by [email protected]), and no matching code in ontologysvc/.
This error shape has been seen since 2026-08-26, and a similar lifespan traceback (finding 3732) was resolved in October.
The only nearby finding, a cert-manager-cainjector warning 4 min earlier, has no link to this service.
Next step: check whether an ontologysvc pod was stopped or restarted at 05:42 UTC; if so, this is a one-off shutdown traceback, not a service failure.

low confidence

claude-opus-5-5 · 3d ago · $0.03 — every link above is evidence it read; nothing else was claimed.

The facts

rule_id13
rule_nameProduction logs — panic / fatal
levelhigh
count1
count_is_floorno
window_min10
shapeTraceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw
sampleTraceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan await receive() File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get
trace—
callerlifespan/on.py:134
namespacetilda
log_severityERROR
first_ts1,794,805,925
last_ts1,794,805,925
pattern(?i)\b(panic|fatal)\b|traceback \(most recent call last\)
discriminator13/3ad81281e690

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-15 05:43:29 UTCopened x-ray · severity=critical title=ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794721370, 'last_ts': 1794721370, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794720809&to=1794721409&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794721409, 'last_seen': 1794721409, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 1, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': None, 'episode_at': 1794721409, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 1, 'stale_s': 1}
  • 2026-11-15 05:53:29 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794721370, 'last_ts': 1794721370, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794721349&to=1794721949&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'resolved', 'first_seen': 1794721409, 'last_seen': 1794721949, 'resolved_at': 1794722009, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 10, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794721409, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 601, 'stale_s': 61}
  • 2026-11-16 05:13:11 UTCreopened x-ray · severity=critical was=critical title=ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal level=high was_level=high finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794805925, 'last_ts': 1794805925, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794805391&to=1794805991&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'open', 'first_seen': 1794721409, 'last_seen': 1794805991, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 11, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794805991, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 84583, 'stale_s': 1}
  • 2026-11-16 05:22:13 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 17292, 'fingerprint': 'b3b32d5f3e6abfd5d7f5', 'kind': 'log.pattern', 'severity': 'critical', 'cluster': 'production', 'subject': 'ontologysvc', 'title': 'ontologysvc: 1× “Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.ge” in 10 min — Production logs — panic / fatal', 'facts': {'rule_id': 13, 'rule_name': 'Production logs — panic / fatal', 'level': 'high', 'count': 1, 'count_is_floor': False, 'window_min': 10, 'shape': 'Traceback (most recent call last): File ?, line ?, in lifespan await receive() File ?, line ?, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ?, line ?, in get aw', 'sample': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 645, in lifespan\n await receive()\n File "/usr/local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.12/asyncio/queues.py", line 158, in get\n ', 'trace': None, 'caller': 'lifespan/on.py:134', 'namespace': 'tilda', 'log_severity': 'ERROR', 'first_ts': 1794805925, 'last_ts': 1794805925, 'pattern': '(?i)\\b(panic|fatal)\\b|traceback \\(most recent call last\\)', 'discriminator': '13/3ad81281e690'}, 'evidence': [{'label': 'ontologysvc error logs', 'href': '/logs?cluster=production&from=1794805873&to=1794806473&service=ontologysvc&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #13 over error_lines (BigQuery)'}, 'impact': 72.5, 'status': 'resolved', 'first_seen': 1794721409, 'last_seen': 1794806473, 'resolved_at': 1794806533, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 19, 'narrative': None, 'narrative_kind': None, 'discriminator': '13/3ad81281e690', 'level': 'high', 'told_level': 'high', 'episode_at': 1794805991, 'chat_thread': None, 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 85125, 'stale_s': 61}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
notelog rule #13 over error_lines (BigQuery)
detector—