Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-20 06:10:43.
› →|
updated 1d ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.
← findings

cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors

warning Log rule matched production resolved impact 42 opened 50d ago · last observed 1d ago · observed 2705× · #2894 · d81a0ff47f93f4127ede

What happened template

cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors

Error lines a log rule watches for (a service, a namespace, a pattern), grouped per error shape with the count in the rule's window — one incident, not one message per line. Its level is the rule's.

The facts

rule_id14
rule_nameProduction logs — platform errors
levelmedium
count8
count_is_floorno
window_min30
shape? err=?api\? logger=? kind=? type=? secret=? certificate=?
sample"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \"api\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"
trace—
caller(no caller)
namespacecert-manager
log_severityERROR
first_ts1,794,896,280
last_ts1,794,896,280
pattern
discriminator14/a4b8a6f9db46

The story — No deploy or hand change is recorded in the hour before this opened.

History

  • 2026-11-14 19:54:57 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794686069, 'last_ts': 1794686069, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794684297&to=1794686097&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794686097, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2497, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794686097, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 3984698, 'stale_s': 1}
  • 2026-11-14 20:25:05 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794686069, 'last_ts': 1794686069, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794686045&to=1794687845&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794687845, 'resolved_at': 1794687905, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2526, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794686097, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 3986507, 'stale_s': 62}
  • 2026-11-15 05:38:28 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794721104, 'last_ts': 1794721104, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794719308&to=1794721108&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794721108, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2527, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721108, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4019709, 'stale_s': 1}
  • 2026-11-15 06:08:30 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794721104, 'last_ts': 1794721104, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794721050&to=1794722850&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794722850, 'resolved_at': 1794722910, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2556, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794721108, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4021511, 'stale_s': 61}
  • 2026-11-15 15:23:13 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794756139, 'last_ts': 1794756139, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794754393&to=1794756193&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794756193, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2557, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794756193, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4054794, 'stale_s': 1}
  • 2026-11-15 15:53:15 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794756139, 'last_ts': 1794756139, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794756135&to=1794757935&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794757935, 'resolved_at': 1794757995, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2586, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794756193, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4056596, 'stale_s': 61}
  • 2026-11-16 01:06:19 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="apiservice" type="apiservice" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794791174, 'last_ts': 1794791174, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794789379&to=1794791179&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794791179, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2587, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794791179, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4089780, 'stale_s': 1}
  • 2026-11-16 01:36:21 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="apiservice" type="apiservice" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794791174, 'last_ts': 1794791174, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794791120&to=1794792920&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794792920, 'resolved_at': 1794792981, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2616, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794791179, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4091582, 'stale_s': 62}
  • 2026-11-16 10:50:14 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794826210, 'last_ts': 1794826210, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794824414&to=1794826214&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794826214, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2617, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794826214, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4124816, 'stale_s': 2}
  • 2026-11-16 11:20:37 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" type="mutatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794826210, 'last_ts': 1794826210, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794826177&to=1794827977&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794827977, 'resolved_at': 1794828037, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2646, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794826214, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4126640, 'stale_s': 63}
  • 2026-11-16 20:35:04 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794861245, 'last_ts': 1794861245, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794859504&to=1794861304&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794861304, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2647, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794861304, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4159906, 'stale_s': 2}
  • 2026-11-16 21:04:08 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794861245, 'last_ts': 1794861245, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794861188&to=1794862988&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794862988, 'resolved_at': 1794863048, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2675, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794861304, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4161651, 'stale_s': 63}
  • 2026-11-17 06:18:09 UTCreopened x-ray · severity=warning was=warning title=cert-manager-cainjector: 8× “? err=?api\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors level=medium was_level=medium finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="customresourcedefinition" type="customresourcedefinition" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794896280, 'last_ts': 1794896280, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794894489&to=1794896289&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'open', 'first_seen': 1790701400, 'last_seen': 1794896289, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2676, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794896289, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4194891, 'stale_s': 2}
  • 2026-11-17 06:48:16 UTCresolved x-ray · reason=not observed grace_s=0 finding={'id': 2894, 'fingerprint': 'd81a0ff47f93f4127ede', 'kind': 'log.pattern', 'severity': 'warning', 'cluster': 'production', 'subject': 'cert-manager-cainjector', 'title': 'cert-manager-cainjector: 8× “? err=?api\\? logger=? kind=? type=? secret=? certificate=?” in 30 min — Production logs — platform errors', 'facts': {'rule_id': 14, 'rule_name': 'Production logs — platform errors', 'level': 'medium', 'count': 8, 'count_is_floor': False, 'window_min': 30, 'shape': '? err=?api\\? logger=? kind=? type=? secret=? certificate=?', 'sample': '"unable to fetch certificate that owns the secret" err="Certificate.cert-manager.io \\"api\\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" type="validatingwebhookconfiguration" secret="tilda-gateway/tls-certificate-api" certificate="tilda-gateway/api"', 'trace': None, 'caller': '(no caller)', 'namespace': 'cert-manager', 'log_severity': 'ERROR', 'first_ts': 1794896280, 'last_ts': 1794896280, 'pattern': '', 'discriminator': '14/a4b8a6f9db46'}, 'evidence': [{'label': 'cert-manager-cainjector error logs', 'href': '/logs?cluster=production&from=1794896236&to=1794898036&service=cert-manager-cainjector&severity=ERROR'}], 'provenance': {'source': 'x-ray', 'note': 'log rule #14 over error_lines (BigQuery)'}, 'impact': 42.5, 'status': 'resolved', 'first_seen': 1790701400, 'last_seen': 1794898036, 'resolved_at': 1794898096, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 2705, 'narrative': None, 'narrative_kind': None, 'discriminator': '14/a4b8a6f9db46', 'level': 'medium', 'told_level': 'medium', 'episode_at': 1794896289, 'chat_thread': 'xray-alert-11-748fd5a928-d81a0ff47f93f4127ede-1792882177', 'kind_label': 'Log rule matched', 'group': 'Errors', 'age_s': 4196697, 'stale_s': 61}

Check it yourself

Every link carries the finding's own time window, so it shows tomorrow what it shows now.

Actions

Sign in (IAP) to acknowledge, snooze or resolve.

Where the numbers come from

sourcex-ray
notelog rule #14 over error_lines (BigQuery)
detector—