← findings
cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors
warning
Log rule matched
production
resolved
impact 42
opened 50d ago · last observed 1d ago · observed 2705× · #2894 · d81a0ff47f93f4127ede
What happened template
cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors
Error lines a log rule watches for (a service, a namespace, a pattern), grouped per error shape with the count in the rule's window — one incident, not one message per line. Its level is the rule's.
The facts
| rule_id | 14 |
| rule_name | Production logs — platform errors |
| level | medium |
| count | 8 |
| count_is_floor | no |
| window_min | 30 |
| shape | ? err=?api\? logger=? kind=? type=? secret=? certificate=? |
| sample | "unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \"api\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api" |
| trace | — |
| caller | (no caller) |
| namespace | cert-manager |
| log_severity | ERROR |
| first_ts | 1,794,896,280 |
| last_ts | 1,794,896,280 |
| pattern | |
| discriminator | 14/a4b8a6f9db46 |
The story — No deploy or hand change is recorded in the hour before this opened.
- 2026-09-29 17:03:20 UTCthis finding cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors
- 2026-09-29 18:12:37 UTCfinding · warning neo4j: 16× “(empty)” in 30 min — Production logs — Neo4j errors
- 2026-09-29 23:04:32 UTCfinding · warning ScaledJob/rag-agent-worker (tilda) cannot read its scaling metric — 2× in 2 h
- 2026-10-03 04:31:35 UTCfinding · warning HorizontalPodAutoscaler/ontologysvc-hpa (tilda) cannot read its scaling metric — 4× in 2 h
- 2026-10-13 15:54:07 UTCfinding · critical onboardingsvc GET /api/myprofile/visibility/study/:id/blindedScopes: 15 of 15 calls failed (100 %) in the last 15 min
- 2026-10-18 18:16:26 UTCfinding · critical docproc GET /api/v1/docproc/ai-runs/:id: permission denied — 28 of 53 calls (53 %; 403×28) in the last 15 min — it usually has none
- 2026-10-19 06:46:26 UTCfinding · critical notifsvc GET /api/v1/notif/uievents: permission denied — 28 of 39 calls (72 %; 401×28) in the last 15 min — 11× its usual 2.6
- 2026-10-23 13:03:15 UTCfinding · info virtual-assistant: Neo4j queries per request 83.8 — 14.0× the fleet's usual 6.0
- 2026-10-23 13:42:32 UTCfinding · info dashql: p95 latency (ms) 3.77 s — 5.2× the fleet's usual 726 ms
- 2026-11-16 00:15:00 UTClatency cluster p95 peaked at 4.55 s
- 2026-11-16 05:10:11 UTCdeploy [email protected] deployed data-monitor
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed chat-agent
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed dashrsvc
- 2026-11-16 05:10:12 UTCdeploy [email protected] deployed workplansvc-infra-production
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed execsvc
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed dashql
- 2026-11-16 05:10:13 UTCdeploy [email protected] deployed enrollsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed exportsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed docsvc
- 2026-11-16 05:10:14 UTCdeploy [email protected] deployed finsvc
- 2026-11-16 05:10:15 UTCdeploy [email protected] deployed notifsvc
- 2026-11-16 05:10:16 UTCdeploy [email protected] deployed onboardingsvc
- 2026-11-16 05:10:17 UTCdeploy [email protected] deployed patientsvc
- 2026-11-16 05:10:19 UTCdeploy [email protected] deployed docproc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed ontologysvc
- 2026-11-16 05:10:23 UTCdeploy [email protected] deployed schedsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed studiosvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed regsvc
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tusker
- 2026-11-16 05:10:24 UTCdeploy [email protected] deployed tmfsync
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed studioaisvc
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed rag-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplan-agent
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed virtual-assistant
- 2026-11-16 05:10:25 UTCdeploy [email protected] deployed workplansvc
- 2026-11-16 05:10:51 UTCdeploy [email protected] deployed asm
- 2026-11-17 00:20:00 UTCerrors cluster 5xx peaked at 15 in one five-minute bucket (median 0 in this window)
History
- 2026-11-14 19:54:57 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794686069, 'last_ts': 1794686069, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794684297&to=1794686097&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794686097, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2497, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794686097, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 3984698, 'stale_s': 1}
- 2026-11-14 20:25:05 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794686069, 'last_ts': 1794686069, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794686045&to=1794687845&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794687845, 'resolved_at': 1794687905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2526, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794686097, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 3986507, 'stale_s': 62}
- 2026-11-15 05:38:28 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794721104, 'last_ts': 1794721104, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794719308&to=1794721108&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794721108, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2527, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721108, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4019709, 'stale_s': 1}
- 2026-11-15 06:08:30 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794721104, 'last_ts': 1794721104, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794721050&to=1794722850&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794722850, 'resolved_at': 1794722910, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2556, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721108, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4021511, 'stale_s': 61}
- 2026-11-15 15:23:13 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794756139, 'last_ts': 1794756139, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794754393&to=1794756193&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794756193, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2557, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794756193, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4054794, 'stale_s': 1}
- 2026-11-15 15:53:15 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794756139, 'last_ts': 1794756139, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794756135&to=1794757935&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794757935, 'resolved_at': 1794757995, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2586, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794756193, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4056596, 'stale_s': 61}
- 2026-11-16 01:06:19 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="apiservice" type="apiservice" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794791174, 'last_ts': 1794791174, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794789379&to=1794791179&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794791179, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2587, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794791179, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4089780, 'stale_s': 1}
- 2026-11-16 01:36:21 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="apiservice" type="apiservice" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794791174, 'last_ts': 1794791174, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794791120&to=1794792920&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794792920, 'resolved_at': 1794792981, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2616, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794791179, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4091582, 'stale_s': 62}
- 2026-11-16 10:50:14 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794826210, 'last_ts': 1794826210, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794824414&to=1794826214&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794826214, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2617, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794826214, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4124816, 'stale_s': 2}
- 2026-11-16 11:20:37 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794826210, 'last_ts': 1794826210, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794826177&to=1794827977&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794827977, 'resolved_at': 1794828037, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2646, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794826214, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4126640, 'stale_s': 63}
- 2026-11-16 20:35:04 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794861245, 'last_ts': 1794861245, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794859504&to=1794861304&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794861304, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2647, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794861304, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4159906, 'stale_s': 2}
- 2026-11-16 21:04:08 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794861245, 'last_ts': 1794861245, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794861188&to=1794862988&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794862988, 'resolved_at': 1794863048, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2675, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794861304, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4161651, 'stale_s': 63}
- 2026-11-17 06:18:09 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="customresourcedefinition" type="customresourcedefinition" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794896280, 'last_ts': 1794896280, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794894489&to=1794896289&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794896289, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2676, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794896289, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4194891, 'stale_s': 2}
- 2026-11-17 06:48:16 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794896280, 'last_ts': 1794896280, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794896236&to=1794898036&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794898036, 'resolved_at': 1794898096, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2705, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794896289, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4196697, 'stale_s': 61}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | x-ray |
| note | log rule #14 over error_lines (BigQuery) |
| detector | — |