从 1.18.6 升级到 1.19.1 时，不允许使用 hubble-relay，operation not permitted

本文永久链接: https://www.xtplayer.cn/rke2/hubble-relay-operation-not-permitted-on-upgrade-from-1-18-6-to-1-19-1/

环境

运行 rke2-cilium v1.18.6 的 Rancher 配置的 RKE2 集群正在升级到 rke2-cilium 1.19.1。

受影响的 RKE2 版本包括：

v1.32.13+rke2r1
v1.33.9+rke2r1
v1.34.5+rke2r1
v1.35.2+rke2r1

问题背景

在 Rancher 配置的 RKE2 集群中，运行 rke2-cilium v1.18.6，并采用类似的 cilium 配置：

rke2-cilium:
  bandwidthManager:
    bbr: true
    enabled: true
  bpf:
    hostLegacyRouting: false
    masquerade: true
  hubble:
    enabled: true
    relay:
      enabled: true
  k8sServiceHost: 127.0.0.1
  k8sServicePort: 6443
  kubeProxyReplacement: true
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 25%

hubble-relay pod 可能无法在您的某个节点上启动，并显示以下 pod 日志：

time=2026-03-19T20:35:03.505629254Z level=info msg="Starting gRPC server..." subsys=hubble-relay options="{peerTarget:hubble-peer.kube-system.svc.cluster.local.:443 retryTimeout:30000000000 listenAddress::4245 healthListenAddress::4222 metricsListenAddress::9966 log:0xc0003dace0 serverTLSConfig:<nil> insecureServer:true clientTLSConfig:0xc0003db770 clusterName:default insecureClient:false observerOptions:[0x21b70c0 0x21b71a0] grpcMetrics:0xc0007220e0 grpcUnaryInterceptors:[0x226ea00 0x226dde0] grpcStreamInterceptors:[0x226eac0 0x226e020]}"
time=2026-03-19T20:35:03.50708986Z level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" subsys=hubble-relay error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.43.232.112:443: connect: operation not permitted\"" connectionTimeout=30s
time=2026-03-19T20:35:33.508934741Z level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" subsys=hubble-relay error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.43.232.112:443: connect: operation not permitted\"" connectionTimeout=30s
time=2026-03-19T20:36:03.510216143Z level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" subsys=hubble-relay error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.43.232.112:443: connect: operation not permitted\"" connectionTimeout=30s
time=2026-03-19T20:36:10.332066602Z level=info msg="Stopping server..." subsys=hubble-relay
time=2026-03-19T20:36:10.332232119Z level=info msg="Server stopped" subsys=hubble-relay

解决方法

Rancher 配置集群或非 Rancher 配置集群:

升级后重启受影响的节点，这将清除节点上的 BPF 状态。或者在升级到 1.19.1 版本之前，请在 Cilium Helm Values 中将cleanBPFState设置为 true。
注意！根据 Cilium 文档，升级到 1.19.1 版本后应删除此键值。

Rancher 配置的 rke2 集群

通过 Rancher UI 修改 Cilium Chart：

在 Rancher UI 中导航至集群管理
选择受影响的集群，编辑集群配置
在“集群配置”下，点击 Add-on: Cilium
找到 cleanBPFState 并将其从 false 更改为 true
最后点击保存

正常进行升级，然后在升级后还原更改

非 Rancher 配置的 rke2 集群

通过 HelmChartConfig 自定义 Cilium Chart 参数，添加 cleanBPFState: true
按正常流程完成升级，然后在升级后撤销更改。

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    bandwidthManager:
      bbr: true
      enabled: true
    bpf:
      hostLegacyRouting: false
      masquerade: true
    cleanBPFState: true #####This is Only Change
    hubble:
      enabled: true
      relay:
        enabled: true
    k8sServiceHost: 127.0.0.1
    k8sServicePort: 6443
    kubeProxyReplacement: true
    updateStrategy:
      rollingUpdate:
      maxUnavailable: 25%

问题原因

问题已上报给 Cilium 公司，相关issue https://github.com/cilium/cilium/issues/44891，正在等待官方答复。