Kubernetes 集群尝试基于 Calico 启用 eBPF(Kubernetes cluster tries to enable ebpf based on calico)

之前基于 Kubernetes 1.23.5 + calico 3.22 尝试过，但没有成功，遇到的问题详见 K8s 网络切换到 Calico eBPF 后 service node port 与 ingress nginx 不能正常工作。

现在基于 Kubernetes 1.24 + calico 3.23 再次尝试启用 eBPF，官方帮助文档 Enable the eBPF dataplane。

前提条件：Ubuntu 版本至少 20.04，确认 BPF 文件系统是否已挂载

# mount | grep "/sys/fs/bpf"
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)

创建 ConfigMap，配置 api-server 的主机名与端口号，calico 通过这个 ConfinMap 获取 api-server 的地址

kubectl create cm kubernetes-services-endpoint -n kube-system

kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: kube-system
data:
  KUBERNETES_SERVICE_HOST: "k8s-api"
  KUBERNETES_SERVICE_PORT: "6443"

如果不知道 api-server 的地址，可以通过下面的命令获取

kubectl get configmap -n kube-system kube-proxy -o yaml | grep server

重新部署 calico pod

kubectl delete pod -n kube-system -l k8s-app=calico-node
kubectl delete pod -n kube-system -l k8s-app=calico-kube-controllers

检查 calico pod 是否运行正常

kubectl get pods -n kube-system | grep calico

禁用 kube-proxy

kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}'

启用 eBPF

calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": true}}'

确认 eBPF 是否已启用

calicoctl get felixConfiguration -o yaml | grep bpfEnabled
    bpfEnabled: true

在 node 服务器上用 tc 命令检查 eBPF 是否正常启用

tc -s qdisc show dev eth0 | grep clsact 
qdisc `clsact` ffff: parent ffff:fff1

出现 说明已经启用。

clsact

但问题依旧，NodePort 依然无法正常工作。

算了，不折腾了，改用 cilium。

————————

Previous attempts based on kubernetes 1.23.5 + calico 3.22 failed. See the problems encountered in k8s network. After switching to calico ebpf, service node port and ingress nginx cannot work normally.

Now try to enable ebpf again based on kubernetes 1.24 + calico 3.23. The official help document enables the ebpf dataplane.

Prerequisite: Ubuntu version is at least 20.04, and confirm whether BPF file system is mounted

# mount | grep "/sys/fs/bpf"
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)

Create configmap and configure the host name and port number of API server. Calico obtains the address of API server through this confinmap

kubectl create cm kubernetes-services-endpoint -n kube-system

kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: kube-system
data:
  KUBERNETES_SERVICE_HOST: "k8s-api"
  KUBERNETES_SERVICE_PORT: "6443"

If you don’t know the address of API server, you can get it through the following command

kubectl get configmap -n kube-system kube-proxy -o yaml | grep server

Redeploy calico pod

kubectl delete pod -n kube-system -l k8s-app=calico-node
kubectl delete pod -n kube-system -l k8s-app=calico-kube-controllers

Check whether calico pod works normally

kubectl get pods -n kube-system | grep calico

禁用 kube-proxy

kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}'

Enable ebpf

calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": true}}'

Verify that ebpf is enabled

calicoctl get felixConfiguration -o yaml | grep bpfEnabled
    bpfEnabled: true

Use the TC command on the node server to check whether ebpf is enabled normally

tc -s qdisc show dev eth0 | grep clsact 
qdisc `clsact` ffff: parent ffff:fff1

The display indicates that it is enabled.

clsact

But the problem remains. Nodeport still doesn’t work properly.

Forget it, don’t bother, use cilium instead.