kube-vip搭建kubernetes高可用

1. 概览

  • OS:KVM(Ubuntu20.04)
  • Docker:Docker version 19.03.15
  • Kubernetes:v1.20.0
  • Helm:v3.7.2
节点 地址
kube-vip 192.168.2.100
master01 192.168.2.89
master02 192.168.2.90
master03 192.168.2.91
slave-01 192.168.2.92

2. 简介

kube-vip 可以在你的控制平面节点上提供一个 Kubernetes 原生的 HA 负载均衡,我们不需要再在外部设置 HAProxy 和 Keepalived 来实现集群的高可用了。

kube-vip 是一个为 Kubernetes 集群内部和外部提供高可用和负载均衡的开源项目,在 Vmware 的 Tanzu 项目中已经使用 kube-vip 替换了用于 vSphere 部署的 HAProxy 负载均衡器,本文我们将先来了解 kube-vip 如何用于 Kubernetes 控制平面的高可用和负载均衡功能。

2.1 特点

Kube-Vip 最初是为 Kubernetes 控制平面提供 HA 解决方案而创建的,随着时间的推移,它已经发展为将相同的功能合并到 Kubernetes 的 LoadBalancer 类型的 Service 中了。

  • VIP 地址可以是 IPv4 或 IPv6
  • 带有 ARP(第2层)或 BGP(第3层)的控制平面
  • 使用领导选举或 raft 控制平面
  • 带有 kubeadm(静态 Pod)的控制平面 HA
  • 带有 K3s/和其他(DaemonSets)的控制平面 HA
  • 使用 ARP 领导者选举的 Service LoadBalancer(第 2 层)
  • 通过 BGP 使用多个节点的 Service LoadBalancer
  • 每个命名空间或全局的 Service LoadBalancer 地址池
  • Service LoadBalancer 地址通过 UPNP 暴露给网关

2.2 HAProxy 和 kube-vip 的 HA 集群

在以前我们在私有环境下创建 Kubernetes 集群时,我们需要准备一个硬件/软件的负载均衡器来创建多控制面集群,更多的情况下我们会选择使用 HAProxy + Keepalived 来实现这个功能。一般情况下我们会创建两个Pod(HAProxy+Keepalive),通过 VIP 将流量重定向到后端的某个 Kubernetes 控制器平面节点上。

kube-vip 可以通过静态 pod 运行在控制平面节点上,这些 pod 通过ARP 对话来识别每个节点上的其他主机,所以需要在 hosts 文件中设置每个节点的 IP 地址,我们可以选择 BGP 或 ARP 来设置负载平衡器,这与 Metal LB 比较类似。

2.3 kube-vip 架构

  • Cluster

kube-vip 建立了一个多节点或多模块的集群来提供高可用性。在 ARP 模式下,会选出一个领导者,这个节点将继承虚拟 IP 并成为集群内负载均衡的领导者,而在 BGP 模式下,所有节点都会通知 VIP 地址。

当使用 ARP 或 layer2 时,它将使用领导者选举,当然也可以使用 raft 集群技术,但这种方法在很大程度上已经被领导者选举所取代,特别是在集群中运行时。

  • 虚拟IP

集群中的领导者将分配 vip,并将其绑定到配置中声明的选定接口上。当领导者改变时,它将首先撤销 vip,或者在失败的情况下,vip 将直接由下一个当选的领导者分配。

当 vip 从一个主机移动到另一个主机时,任何使用 vip 的主机将保留以前的 vip <-> MAC 地址映射,直到 ARP 过期(通常是30秒)并检索到一个新的 vip <-> MAC 映射,这可以通过使用无偿的 ARP 广播来优化。

  • ARP

kube-vip可以被配置为广播一个无偿的 arp(可选),通常会立即通知所有本地主机 vip <-> MAC 地址映射已经改变。

3. 使用kube-vip

3.1 系统准备

  • 原生Ubuntu20.04
  • 替换国内源
sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo sh -c "echo 'deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
' > /etc/apt/sources.list"
sudo apt-get update
sudo apt-get install -y vim curl lrzsz htop gcc make tree mariadb-client-10.3 iotop redis-tools apt-transport-https ca-certificates software-properties-common

 

  • Docker下载

kubernetes 官方建议 docker 驱动采用 systemd,当然可以不修改,只是kubeadm init时会有warning提示

sudo apt-get remove docker docker-engine docker.io containerd runc -y
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt-get install docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal -y
sudo usermod -aG docker kyfq
cat <<EOF | sudo tee /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "insecure-registries": ["192.168.2.221:5003","192.168.2.223:5004"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-opts": {
	        "max-size": "100m"
    },
    "storage-driver": "overlay2"
}
EOF
sudo systemctl restart docker
  • Docker查看使用驱动
kyfq@master-01:~$ docker info | grep "Cgroup"
 Cgroup Driver: systemd
  • 关闭swap及防火墙
sudo ufw disable
sudo swapoff -a
sudo sed -i 's/\/swapfile/#&/' /etc/fstab
  • 内核设置
sudo vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
sudo sysctl -p /etc/sysctl.d/k8s.conf
#载入ipvs内核模块
modprobe ip_vs 
modprobe ip_vs_rr 

 

  • 下载kubernetes
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt install kubelet=1.20.0-00 kubeadm=1.20.0-00 kubectl=1.20.0-00 -y

 

  • 生成kube-vip.yaml

每台主中都需要生成kube-vip.yaml,使用加入集群的方法变成控制节点需要后续加入/etc/kubernetes/manifests/,不然报错/etc/kubernetes/manifests/下不为空

export VIP=192.168.2.100
export INTERFACE=ens3
docker run --network host --rm plndr/kube-vip:0.3.1 manifest pod --interface $INTERFACE --vip $VIP --controlplane --services --bgp --localAS 65000 --bgpRouterID 192.168.2.89 --bgppeers 192.168.2.90:65000::false,192.168.2.91:65000::false | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

3.2 初始化kubernetes

    • 初始化

    在第一个主节点进行初始化即可

    sudo kubeadm init --control-plane-endpoint 192.168.2.100:6443 --apiserver-advertise-address 192.168.2.89 --apiserver-bind-port 6443 --pod-network-cidr 172.16.0.0/16 --upload-certs --image-repository registry.aliyuncs.com/google_containers
    

    完成后会出现一下内容,其他主节点使用对应的命令进行加入节点即可。

    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    Alternatively, if you are the root user, you can run:
    
      export KUBECONFIG=/etc/kubernetes/admin.conf
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    You can now join any number of the control-plane node running the following command on each as root:
    
      kubeadm join 192.168.2.100:6443 --token mi54yv.6h8c7y36my1hfuwc \
        --discovery-token-ca-cert-hash sha256:2654545895d9d06726ca1b95e278c1c39df4e32c1ef83231c11010ba468ca007 \
        --control-plane --certificate-key 1e625b73c52edf5fc51588960702f8d90f598205e1deeea9e34de830b4f28868
    
    Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
    As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
    "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 192.168.2.100:6443 --token 0u0qqw.6h9tzoyl3cucpqt5 \
        --discovery-token-ca-cert-hash sha256:2654545895d9d06726ca1b95e278c1c39df4e32c1ef83231c11010ba468ca007 

     

    • 加入节点

    安装完成环境,进行kubeadm join即可

    3.3 CNI网络插件安装

    使用CNI网络插件cilium

    • 安装helm
    curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
    sudo apt-get install apt-transport-https --yes
    echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
    sudo apt-get update
    sudo apt-get install helm

     

    • 安装cilium插件
    helm repo add cilium https://helm.cilium.io/
    helm install cilium cilium/cilium --version 1.9.4 \
    --namespace kube-system

     

    3.4 token、discovery-token-ca-cert-hash、certificate-key查看及新建

    • token查看
    sudo kubeadm token list
    TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
    drjvw3.dqf4t4qaujlbd6rb   1h          2021-12-22T19:17:56+08:00   <none>                   Proxy for managing TTL for the kubeadm-certs secret        <none>
    q1me8r.2lzbjloddefgfoon   17h         2021-12-23T10:57:21+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
    • token创建
    kubeadm token create
    • discovery-token-ca-cert-hash查看
    openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
    • certificate-key创建
     sudo kubeadm init phase upload-certs --upload-certs
    • 创建worker节点token
    sudo kubeadm token create --print-join-command

    4. 集群使用

    4.1 集群查看

    kyfq@master-01:~$ kubectl get node -o wide
    NAME        STATUS   ROLES                  AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
    master-01   Ready    control-plane,master   24h   v1.20.0   192.168.2.89   <none>        Ubuntu 20.04.1 LTS   5.11.0-43-generic   docker://19.3.15
    master-02   Ready    control-plane,master   24h   v1.20.0   192.168.2.90   <none>        Ubuntu 20.04.1 LTS   5.4.0-42-generic    docker://19.3.15
    master-03   Ready    control-plane,master   18h   v1.20.0   192.168.2.91   <none>        Ubuntu 20.04.1 LTS   5.4.0-42-generic    docker://19.3.15
    slave-01    Ready    <none>                 99s   v1.20.0   192.168.2.92   <none>        Ubuntu 20.04.1 LTS   5.4.0-42-generic    docker://19.3.15
    kyfq@master-01:~$ kubectl get pod -o wide -n kube-system 
    NAME                                READY   STATUS    RESTARTS   AGE    IP             NODE        NOMINATED NODE   READINESS GATES
    cilium-8xlt8                        1/1     Running   1          24h    192.168.2.89   master-01   <none>           <none>
    cilium-bklpc                        1/1     Running   0          18h    192.168.2.91   master-03   <none>           <none>
    cilium-cxsts                        1/1     Running   0          115s   192.168.2.92   slave-01    <none>           <none>
    cilium-operator-696dc48d8d-pvmp8    1/1     Running   3          24h    192.168.2.90   master-02   <none>           <none>
    cilium-operator-696dc48d8d-s7bf9    1/1     Running   7          24h    192.168.2.97   master-01   <none>           <none>
    cilium-x9hkd                        1/1     Running   0          24h    192.168.2.90   master-02   <none>           <none>
    coredns-7f89b7bc75-7mpcb            1/1     Running   1          24h    10.0.0.8       master-01   <none>           <none>
    coredns-7f89b7bc75-rmwwd            1/1     Running   1          24h    10.0.0.210     master-01   <none>           <none>
    etcd-master-01                      1/1     Running   3          24h    192.168.2.89   master-01   <none>           <none>
    etcd-master-02                      1/1     Running   2          24h    192.168.2.90   master-02   <none>           <none>
    etcd-master-03                      1/1     Running   1          18h    192.168.2.91   master-03   <none>           <none>
    kube-apiserver-master-01            1/1     Running   3          24h    192.168.2.89   master-01   <none>           <none>
    kube-apiserver-master-02            1/1     Running   2          24h    192.168.2.90   master-02   <none>           <none>
    kube-apiserver-master-03            1/1     Running   1          18h    192.168.2.91   master-03   <none>           <none>
    kube-controller-manager-master-01   1/1     Running   5          24h    192.168.2.97   master-01   <none>           <none>
    kube-controller-manager-master-02   1/1     Running   3          24h    192.168.2.90   master-02   <none>           <none>
    kube-controller-manager-master-03   1/1     Running   3          18h    192.168.2.91   master-03   <none>           <none>
    kube-proxy-5qjbr                    1/1     Running   0          115s   192.168.2.92   slave-01    <none>           <none>
    kube-proxy-fmgrf                    1/1     Running   1          24h    192.168.2.97   master-01   <none>           <none>
    kube-proxy-lwpsh                    1/1     Running   0          18h    192.168.2.91   master-03   <none>           <none>
    kube-proxy-z64sf                    1/1     Running   0          24h    192.168.2.90   master-02   <none>           <none>
    kube-scheduler-master-01            1/1     Running   4          24h    192.168.2.97   master-01   <none>           <none>
    kube-scheduler-master-02            1/1     Running   1          24h    192.168.2.90   master-02   <none>           <none>
    kube-scheduler-master-03            1/1     Running   2          18h    192.168.2.91   master-03   <none>           <none>
    kube-vip-master-01                  1/1     Running   1          24h    192.168.2.97   master-01   <none>           <none>
    kube-vip-master-02                  1/1     Running   0          24h    192.168.2.90   master-02   <none>           <none>
    kube-vip-master-03                  1/1     Running   0          18h    192.168.2.91   master-03   <none>           <none>

     

    4.2 YAML编排

    简单演示一个nginx deployment

    • nginx-deployment.yaml
    kind: Deployment                 #接口类型
    metadata:
      name: lct-nginx               #Deployment名称
    spec:
      selector:
        matchLabels:
          app: lct-nginx
      replicas: 1
       #strategy:
        #rollingUpdate:  ##由于replicas为3,则整个升级,pod个数在2-4个之间
          #maxSurge: 1      #滚动升级时会先启动1个pod
          #maxUnavailable: 1 #滚动升级时允许的最大Unavailable的pod个数
      template:         
        metadata:
          labels:
            app: lct-nginx  #模板名称必填
        spec: #定义容器模板,该模板可以包含多个容器
          # nodeSelector:
          #   deploy.type: lct_service
          imagePullSecrets:
          - name: docker-harbor-registry
          containers:                                                                   
            - name: nginx                                                           #镜像名称
              image: nginx #镜像地址
              command: ["/bin/bash","-c","chown www-data:www-data /var/log/nginx/ && exec nginx -g 'daemon off;'"]     #启动命令
              imagePullPolicy: IfNotPresent  #如果不存在则拉取
              ports:
                - containerPort: 80 #对service暴露端口
              volumeMounts:     
              - name: static
                mountPath: /usr/share/nginx/html
    
          volumes:  # 定义磁盘给上面volumeMounts挂载
          - name: static
            hostPath: 
              path: /static/
              type: Directory

     

    • nginx-svc.yaml
    apiVersion: v1
    kind: Service
    metadata:
          name: lct-nginx
    spec:
          type: NodePort
          ports:
          - name: lct-nginx
            port: 80
            protocol: TCP
            targetPort: 80
            nodePort: 30003
          selector:
            app: lct-nginx

     

    kyfq@master-01:~/nginx$ kubectl get pod -o wide
    NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
    lct-nginx-5c4f887d46-drmtd   1/1     Running   0          4m35s   10.0.3.187   slave-01   <none>           <none>
    kyfq@master-01:~/nginx$ curl 192.168.2.89:30003
    hello world!

    5. 总结

    使用kube-vip要比使用HA+keepalive要轻量和方便很多,CNI选择cilium也是可以使用的,后续再对这个网络插件进行分析。

    遇到的坑总结:

    • kubeadmin reset 的时候/etc/kubernetes/manifests/下的yaml全部清空,master初始化时需要注意,没有kube-vip.yaml的情况下会报vip:6443 timeout;
    • 其他加入成为控制节点的主机kube-vip.yaml需要加入集群完成后生成,放入/etc/kubernetes/manifests/下,会自动生成pod,提前放入会报错/etc/kubernetes/manifests/目录不为空;
    • certificate-key,token会过期,加入需要重新生成。

     

    版权声明:
    作者:lee
    链接:https://www.goufusheng.top/archives/740
    来源:苟浮生的桃花扇
    文章版权归作者所有,未经允许请勿转载。

    THE END
    分享
    二维码
    < <上一篇
    下一篇>>
    文章目录
    关闭
    目 录