本文因为头条排版问题，阅读不是很友好，可以私信联系作者获取源文档。

一、架构说明Kubernetes集群组件etcd 一个高可用的K/V键值对存储和服务发现系统flannel 实现跨主机的容器网络的通信kube-apiserver 提供kubernetes集群的API调用kube-controller-manager 确保集群服务kube-scheduler 调度容器，分配到Nodekube-proxy 提供网络代理服务kubeadm:用来初始化集群的指令。kubectl: 用来与集群通信的命令行工具。通过kubectl可以部署和管理应用，查看各种资源，创建、删除和更新各种组件。kubelet 运行在集群中的每个节点上，用来启动 pod 和 container 等。

k8s集群高可用，一般是etcd、kube-apiserver、kube-controller-manager、kube-scheduler服务组件的高可用。

规划

3个master节点，2个worker节点，使用keepalived+haproxy做高可用

二、前期准备工作修改所有主机的hosts文件

cat >/etc/hosts<<EOF130.252.10.235 node1130.252.10.236 node2130.252.10.237 node3130.252.10.238 node4130.252.10.239 node5EOF

同步时间

ntpdate time.windows.comhwclock --systohc

禁用selinux，需要重启

setenforce 0sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinuxsed -i 's/^SELINUX=*/SELINUX=disabled/' /etc/selinux/config

内核参数修改

cat <<EOF | sudo tee /etc/modules-load.d/k8s.confbr_netfilterEOFcat >> /etc/sysctl.d/k8s.conf <<EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_nonlocal_bind = 1net.ipv4.ip_forward = 1vm.swappiness = 0EOFsysctl --system

Linux 节点上的 iptables 需要查看桥接流量，因此要将 net.bridge.bridge-nf-call-iptables 设置为 1，并且加载 br_netfilter 内核模块关闭系统swap

Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。

方法一 通过kubelet的启动参数 –fail-swap-on=fals 更改这个限制方法二 关闭系统的swap

修改/etc/fstab文件，注释掉SWAP的自动挂载，使用free -m确认swap已经关闭。

swapoff -ased -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

关闭防火墙

生产环境不建议关闭防火墙

systemctl disable firewalld.servicesystemctl stop firewalld.service

加载ipvs内核模块

modprobe -- nf_conntrack_ipv4 >/dev/null 2>&1if [ $? -eq 0 ];thennf_conntrack_module=nf_conntrack_ipv4elsenf_conntrack_module=nf_conntrackfi cat > /etc/sysconfig/modules/ipvs.modules <<EOF#!/bin/bashmodprobe -- ip_vsmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shmodprobe -- ${nf_conntrack_module}modprobe -- rbdEOFchmod 755 /etc/sysconfig/modules/ipvs.modulesbash /etc/sysconfig/modules/ipvs.modules

三、安装Docker

yum-utils 提供yum-config-manager工具，devicemapper存储需要device-mapper-persistent-data和lvm2

yum install -y yum-utils device-mapper-persistent-data lvm2

添加yum源仓库官方仓库

yum-config-manager --add-repo 

阿里仓库

yum-config-manager --add-repo 

安装Docker-ce

yum -y install docker-ce docker-ce-cli containerd.io

启动docker，设置开机启动docker

systemctl restart docker && \systemctl enable docker && \systemctl list-unit-files |grep docker

扩展：安装指定版本的docker

yum list docker-ce --showduplicates | sort -r #显示所有可用docker版本yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io官网安装：

配置docker中国镜像加速

修改cgroup driver为systemd，k8s建议使用systemd，配置后重启docker

cat > /etc/docker/daemon.json <<EOF{"registry-mirrors": [";,";,";],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-file": "3","max-size": "100m"},"storage-driver": "overlay2","storage-opts": ["overlay2.override_kernel_check=true"]}EOF# 查看dockerdocker info# 重启systemctl restart docker

安装参考

注意：所有节点都需要安装docker

四、安装haproxy和keepalived安装haproxy和keepalivedhaproxy和keepalived，实现kube-apiserver高可用psmisc提供killall命令

yum install keepalived haproxy psmisc ipvsadm -y

配置haproxy

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bakcat > /etc/haproxy/haproxy.cfg <<EOFglobal    log         127.0.0.1 local2    chroot      /var/lib/haproxy    pidfile     /var/run/haproxy.pid    maxconn     4000    user        haproxy    group       haproxy    daemon    stats socket /var/lib/haproxy/statsdefaults    mode                    tcp    log                     global    option                  tcplog    option                  dontlognull    option                  redispatch    retries                 3    timeout http-request    10s    timeout queue           1m    timeout connect         10s    timeout client          1m    timeout server          1m    timeout http-keep-alive 10s    timeout check           10s    maxconn                 3000listen stats    mode   http    bind :10086    stats   enable    stats   uri     /admin?stats    stats   auth    admin:admin    stats   admin   if TRUEfrontend  k8s_https *:8443    mode   tcp    maxconn     2000    default_backend   https_sribackend https_sri    balance     roundrobin    server  s1  130.252.10.235:6443 check inter 2000 fall 3 rise 3 weight 1    server  s2  130.252.10.236:6443 check inter 2000 fall 3 rise 3 weight 1    server  s3  130.252.10.237:6443 check inter 2000 fall 3 rise 3 weight 1EOF

haproxy 在 10086 端口输出 status 信息；haproxy 监听所有接口的 8443 端口，该端口与环境变量 ${KUBE_APISERVER} 指定的端口必须一致；server 字段列出所有 kube-apiserver 监听的 IP 和端口；配置keepalived

cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bakcat > /etc/keepalived/keepalived.conf <<EOF! Configuration File for keepalivedglobal_defs {   script_user root   enable_script_security   router_id LVS_MASTER}vrrp_script check-haproxy {    script "/usr/bin/killall -0 haproxy"    interval 3    weight -30}vrrp_instance VI_kube-master {    state MASTER    interface ens32    virtual_router_id 60    dont_track_primary    priority 120    advert_int 3    track_script {        check-haproxy    }    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        130.252.10.233    }}EOF

注意：在另外两个节点，设置state为BACKUP，priority分别设置为110和100

启动haproxy和keepalived

在所有节点启动

systemctl restart haproxy && \systemctl enable haproxy && \systemctl status haproxy systemctl restart keepalived && \systemctl enable keepalived && \systemctl status keepalived

查看VIP注意，使用ifconfig命令是看不到的，必须使用ip addr命令

# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000    link/ether 00:0c:29:25:f1:8d brd ff:ff:ff:ff:ff:ff    inet 130.252.10.235/16 brd 130.252.255.255 scope global noprefixroute ens32       valid_lft forever preferred_lft forever    inet 130.252.10.233/32 scope global ens32       valid_lft forever preferred_lft forever    inet6 fe80::ece5:efc4:1cf1:2d7c/64 scope link noprefixroute       valid_lft forever preferred_lft forever3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default    link/ether 02:42:88:7f:d3:1a brd ff:ff:ff:ff:ff:ff    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0       valid_lft forever preferred_lft forever

这时候vip在130.252.10.235上，我们可以通过关闭235上的haproxy，来验证vip是否会漂移到其他节点

五、安装kubeadm、kubelet、kubectlkubeadm: 部署集群用的命令kubelet: 在集群中每台机器上都要运行的组件，负责管理pod、容器的生命周期kubectl: 集群管理工具（可选，只要在控制集群的节点上安装即可）添加kubernetes源

这里使用阿里云的kubernetes的yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl= 

安装kubelet、kubeadm、kubectl

yum install -y kubelet kubeadm kubectlsystemctl enable kubelet && systemctl start kubelet

注意：此时kubelet是无法启动的

注：所有的节点都要安装

六、使用kubeadm初始化cluster准备初始化配置文件

打印kubeadm默认的配置文件

kubeadm config print init-defaults > kubeadm-init.yaml

根据自己的环境修改配置

apiVersion: kubeadm.k8s.io/v1beta1bootstrapTokens:- groups:  - system:bootstrappers:kubeadm:default-node-token  token: abcdef.0123456789abcdef  ttl: 24h0m0s  usages:  - signing  - authenticationkind: InitConfigurationlocalAPIEndpoint:  advertiseAddress: 130.252.10.235 #本机IP  bindPort: 6443 #监听端口nodeRegistration:  criSocket: /var/run/dockershim.sock  name: node1  taints:  - effect: NoSchedule    key: node-role.kubernetes.io/master---apiServer:  timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta1certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrolPlaneEndpoint: "130.252.10.233:8443" #控制平面IP：VIP:端口controllerManager: {}dns:  type: CoreDNSetcd:  local:    dataDir: /var/lib/etcdimageRepository: k8s.gcr.iokind: ClusterConfigurationkubernetesVersion: v1.14.3networking:  dnsDomain: cluster.local  podSubnet: "10.244.0.0/16" #POD网段  serviceSubnet: 10.96.0.0/12 #service网段scheduler: {}---apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationfeatureGates:  SupportIPVSProxyMode: truemode: ipvs---apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationfailSwapOn: falsecgroupDriver: systemd

说明：

advertiseAddress：API Server将要广播的监听地址bindPort：API Server绑定的端口，默认6443serviceSubnet：为service的虚拟IP地址另外指定IP地址段， 缺省值:10.96.0.0/12podSubnet：指明pod网络可以使用的IP地址段。 如果设置了这个参数，control plane将会为每一个节点自动分配CIDRskubeproxy配置段：通过kubeadm config print init-defaults --component-configs KubeProxyConfiguration命令打印来，从Kubernetes 1.11后，SupportIPVSProxyMode默认为truekubelet配置段非必须：可通过kubeadm config print init-defaults --component-configs KubeletConfiguration打印出配置，如果你开启了swap则需要配置failSwapOn: false，并使用kubeadm init --config kubeadm-init.yaml --ignore-preflight-errors=Swap命令初始化

注意：

podSubnet 一定要指定，不然后面flannel启动会失败

具体错误信息如下：Error registering network: failed to acquire lease: node "master1" pod cidr not assigned解决方法：

在/etc/kubernetes/manifests/kube-controller-manager.yaml 文件的command处加入下面两行- --allocate-node-cidrs=true

- --cluster-cidr=10.244.0.0/16或者使用kubeadm reset重新部署处理kubernetes依赖的镜像

master需要的镜像为

kube-apiserverkube-controller-managerkube-schedulerkube-proxycorednspauseetcdflannel

node节点需要的镜像为

kube-proxypauseflannel

查看需要的镜像版本

kubeadm config images list --kubernetes-version=v1.14.3

拉取镜像，此步骤不需要，除非你能访问谷歌

kubeadm config images pull

一般情况，我们通过阿里云镜像服务拉取kubernetes所需要的镜像，然后再重新tag

初始化第一个master节点

注意：如果你禁用了swap分区，则不需要加--ignore-preflight-errors=Swap

# kubeadm init --config kubeadm-init.yaml --ignore-preflight-errors=Swap[init] Using Kubernetes version: v1.14.3[preflight] Running pre-flight checks    [WARNING Swap]: running with swap on is not supported. Please disable swap[preflight] Pulling images required for setting up a Kubernetes cluster[preflight] This might take a minute or two, depending on the speed of your internet connection[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Activating the kubelet service[certs] Using certificateDir folder "/etc/kubernetes/pki"[certs] Generating "front-proxy-ca" certificate and key[certs] Generating "front-proxy-client" certificate and key[certs] Generating "etcd/ca" certificate and key[certs] Generating "etcd/server" certificate and key[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [130.252.10.235 127.0.0.1 ::1][certs] Generating "apiserver-etcd-client" certificate and key[certs] Generating "etcd/healthcheck-client" certificate and key[certs] Generating "etcd/peer" certificate and key[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [130.252.10.235 127.0.0.1 ::1][certs] Generating "ca" certificate and key[certs] Generating "apiserver" certificate and key[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 130.252.10.235 130.252.10.233][certs] Generating "apiserver-kubelet-client" certificate and key[certs] Generating "sa" key and public key[kubeconfig] Using kubeconfig folder "/etc/kubernetes"[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address[kubeconfig] Writing "admin.conf" kubeconfig file[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address[kubeconfig] Writing "kubelet.conf" kubeconfig file[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address[kubeconfig] Writing "controller-manager.conf" kubeconfig file[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address[kubeconfig] Writing "scheduler.conf" kubeconfig file[control-plane] Using manifest folder "/etc/kubernetes/manifests"[control-plane] Creating static Pod manifest for "kube-apiserver"[control-plane] Creating static Pod manifest for "kube-controller-manager"[control-plane] Creating static Pod manifest for "kube-scheduler"[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s[kubelet-check] Initial timeout of 40s passed.[apiclient] All control plane components are healthy after 42.007732 seconds[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster[upload-certs] Skipping phase. Please see --experimental-upload-certs[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule][bootstrap-token] Using token: abcdef.0123456789abcdef[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace[addons] Applied essential addon: CoreDNS[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:  mkdir -p $HOME/.kube  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:   can now join any number of control-plane nodes by copying certificate authoritiesand service account keys on each node and then running the following as root:  kubeadm join 130.252.10.233:8443 --token abcdef.0123456789abcdef \    --discovery-token-ca-cert-hash sha256:4d95989174b67c857bed7313750e021f07ec56beb2e6e764c7ef2de5645187e9 \    --experimental-control-plane       Then you can join any number of worker nodes by running the following on each as root:kubeadm join 130.252.10.233:8443 --token abcdef.0123456789abcdef \    --discovery-token-ca-cert-hash sha256:4d95989174b67c857bed7313750e021f07ec56beb2e6e764c7ef2de5645187e9

说明：

如果初始化时指定了 --upload-certs参数，即使用了kubeadm init --config kubeadm-init.yaml --upload-certs --ignore-preflight-errors=Swap命令来初始化，则不需要手动将证书文件复制到其他节点（第八步操作）--upload-certs：用于将应在所有控制平面实例之间共享的证书上载到群集

初始化过程：

[init]：指定版本进行初始化操作[preflight] ：初始化前的检查和下载所需要的Docker镜像文件[kubelet-start] ：生成kubelet的配置文件”/var/lib/kubelet/config.yaml”，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动失败。[certificates]：生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。[kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。[control-plane]：使用/etc/kubernetes/manifest目录下的YAML文件，安装 Master 组件。[etcd]：使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。[wait-control-plane]：等待control-plan部署的Master组件启动。[apiclient]：检查Master组件服务状态。[uploadconfig]：更新配置[kubelet]：使用configMap配置kubelet。[patchnode]：更新CNI信息到Node上，通过注释的方式记录。[mark-control-plane]：为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。[bootstrap-token]：生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到[addons]：安装附加组件CoreDNS和kube-proxy配置kubectl

mkdir -p $HOME/.kubecp -i /etc/kubernetes/admin.conf $HOME/.kube/configchown $(id -u):$(id -g) $HOME/.kube/config

启用kubectl命令自动补全

安装bash-completion

yum install bash-completion -y

说明：

bash-completion 负责导入 /etc/bash_completion.d 目录中的所有补全脚本。

重新加载shell后，运行命令type _init_completion检查bash-completion是否安装成功

配置kubeclt自动补全

方法一在文件~/.bashrc中导入补全脚本：

echo 'source <(kubectl completion bash)' >>~/.bashrc

方法二将补全脚本添加到目录/etc/bash_completion.d中：

kubectl completion bash >/etc/bash_completion.d/kubectl

注意：

建议使用方法二。方法一会导致shell脚本执行source ~/.bashrc时，出错。

参考：

七、安装Pod网络flannel

kubectl apply -f 

flannel 网络插件不部署，coredns一直处于pending状态。

官网链接：

八、将其他master节点添加到cluster同步证书文件到其他master节点

将node1证书文件复制到其他master节点node2，node3

for host in node2 node3doecho ">>>>>>>>>>>>>>>>>>>>$host<<<<<<<<<<<<<<<<<<<<<"ssh $host "mkdir -p /etc/kubernetes/pki/etcd"scp /etc/kubernetes/pki/ca.* $host:/etc/kubernetes/pki/scp /etc/kubernetes/pki/sa.* $host:/etc/kubernetes/pki/scp /etc/kubernetes/pki/front-proxy-ca.* $host:/etc/kubernetes/pki/scp /etc/kubernetes/pki/etcd/ca.* $host:/etc/kubernetes/pki/etcd/scp /etc/kubernetes/admin.conf $host:/etc/kubernetes/done

将其他master加入集群

查看master的token

kubeadm token list | grep authentication,signing | awk '{print $1}'

查看discovery-token-ca-cert-hash

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

注意：token有效期是有限的，如果旧的token过期，可以使用kubeadm token create --print-join-command重新创建一条token。

分别在master1和master2 执行下面的命令来加入集群，注意要修改token和ca-cert-hash为上面查询出来的结果

kubeadm join 130.252.10.233:8443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:4d95989174b67c857bed7313750e021f07ec56beb2e6e764c7ef2de5645187e9 \--experimental-control-plane --ignore-preflight-errors=Swapmkdir -p $HOME/.kubecp -i /etc/kubernetes/admin.conf $HOME/.kube/configchown $(id -u):$(id -g) $HOME/.kube/config

九、将node节点添加到cluster

kubeadm join 130.252.10.233:8443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:4d95989174b67c857bed7313750e021f07ec56beb2e6e764c7ef2de5645187e9 \--ignore-preflight-errors=Swap

十、检查集群运行健康查看节点状态

# kubectl get nodesNAME STATUS ROLES AGE VERSIONnode1 Ready master 15m v1.14.3node2 Ready master 12m v1.14.3node3 Ready master 10m v1.14.3node4 Ready <none> 25s v1.14.3node5 Ready <none> 2m34s v1.14.3# kubectl get csNAME STATUS MESSAGE ERRORscheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health":"true"}

所有的节点都是NotReady，这是因为每个节点都需要启动若干组件，这些组件都是在pod中运行，需要从Google下载镜像

查看pod的状态

# kubectl get pod --all-namespaces -o wideNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESkube-system coredns-fb8b8dccf-7hdvh 1/1 Running 0 16m 10.244.0.4 node1 <none> <none>kube-system coredns-fb8b8dccf-xcn94 1/1 Running 0 16m 10.244.0.5 node1 <none> <none>kube-system etcd-node1 1/1 Running 0 15m 130.252.10.235 node1 <none> <none>kube-system etcd-node2 1/1 Running 0 13m 130.252.10.236 node2 <none> <none>kube-system etcd-node3 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>kube-system kube-apiserver-node1 1/1 Running 0 15m 130.252.10.235 node1 <none> <none>kube-system kube-apiserver-node2 1/1 Running 0 13m 130.252.10.236 node2 <none> <none>kube-system kube-apiserver-node3 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>kube-system kube-controller-manager-node1 1/1 Running 1 16m 130.252.10.235 node1 <none> <none>kube-system kube-controller-manager-node2 1/1 Running 0 13m 130.252.10.236 node2 <none> <none>kube-system kube-controller-manager-node3 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>kube-system kube-flannel-ds-amd64-dkfhr 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>kube-system kube-flannel-ds-amd64-j2v7w 1/1 Running 0 96s 130.252.10.238 node4 <none> <none>kube-system kube-flannel-ds-amd64-mm82s 1/1 Running 0 14m 130.252.10.235 node1 <none> <none>kube-system kube-flannel-ds-amd64-n9fbw 1/1 Running 0 3m45s 130.252.10.239 node5 <none> <none>kube-system kube-flannel-ds-amd64-psv22 1/1 Running 2 13m 130.252.10.236 node2 <none> <none>kube-system kube-proxy-56wc5 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>kube-system kube-proxy-fqf5x 1/1 Running 0 3m45s 130.252.10.239 node5 <none> <none>kube-system kube-proxy-k8v9v 1/1 Running 0 16m 130.252.10.235 node1 <none> <none>kube-system kube-proxy-mt9g6 1/1 Running 0 13m 130.252.10.236 node2 <none> <none>kube-system kube-proxy-r88kh 1/1 Running 0 96s 130.252.10.238 node4 <none> <none>kube-system kube-scheduler-node1 1/1 Running 1 16m 130.252.10.235 node1 <none> <none>kube-system kube-scheduler-node2 1/1 Running 0 13m 130.252.10.236 node2 <none> <none>kube-system kube-scheduler-node3 1/1 Running 0 11m 130.252.10.237 node3 <none> <none>

Pending、ContainerCreating、ImagePullBackOff 都表明Pod没有就绪，Running才是就绪

所有的服务都变成了集群状态，etcd，kube-apiserver，kube-controller-manager，kube-scheduler服务每个master节点都启动一个Pod

查看集群信息

# kubectl cluster-infoKubernetes master is running at  is running at  further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

查看etcd集群

查看集群是否健康

# docker run --rm -it --net host \-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.3.10 etcdctl \--cert-file /etc/kubernetes/pki/etcd/peer.crt \--key-file /etc/kubernetes/pki/etcd/peer.key \--ca-file /etc/kubernetes/pki/etcd/ca.crt \--endpoints  cluster-healthmember 1625c8dc0eff006e is healthy: got healthy result from  a78e5b84db62da12 is healthy: got healthy result from  be5e1657d6a1f8de is healthy: got healthy result from  is healthy

查看集群的leader

# docker run --rm -it --net host \-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.3.10 etcdctl \--cert-file /etc/kubernetes/pki/etcd/peer.crt \--key-file /etc/kubernetes/pki/etcd/peer.key \--ca-file /etc/kubernetes/pki/etcd/ca.crt \--endpoints  member list1625c8dc0eff006e: name=node1 peerURLs= clientURLs= isLeader=truea78e5b84db62da12: name=node2 peerURLs= clientURLs= isLeader=falsebe5e1657d6a1f8de: name=node3 peerURLs= clientURLs= isLeader=false

注意：

因为是3个节点的ETCD集群，所以只能有1个宕机，如果同时有2个节点宕机，则会出现问题Unable to connect to the server: EOF

ETCD集群需要大多数节点（仲裁）才能就集群状态的更新达成一致，所以ETCD集群节点一般是奇数个，而且只有存活节点数大于下线节点数 才能正常运行，5个节点的ETCD集群，允许同时2个节点故障。

一般建议5个节点，超过5个虽然容错性更高，但是集群写入性就会差。假如etcd实例的总数超过了这个数目。那么多出来的节点（peers）将会以独立（standbys）的方式启动，假如集群中一个活跃的节点挂掉或者被移除掉，那么这些多出来的单独启动的节点将会增加到活跃集群中

检查IPVS

# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.96.0.1:443 rr-> 130.252.10.235:6443 Masq 1 0 0 -> 130.252.10.236:6443 Masq 1 0 0 -> 130.252.10.237:6443 Masq 1 0 1 TCP 10.96.0.10:53 rrTCP 10.96.0.10:9153 rrUDP 10.96.0.10:53 rr

查看当前系统配置

# kubectl get cm -n kube-system kubeadm-config -o yamlapiVersion: v1data:  ClusterConfiguration: |    apiServer:      extraArgs:        authorization-mode: Node,RBAC      timeoutForControlPlane: 4m0s    apiVersion: kubeadm.k8s.io/v1beta1    certificatesDir: /etc/kubernetes/pki    clusterName: kubernetes    controlPlaneEndpoint: 130.252.10.233:8443    controllerManager: {}    dns:      type: CoreDNS    etcd:      local:        dataDir: /var/lib/etcd    imageRepository: k8s.gcr.io    kind: ClusterConfiguration    kubernetesVersion: v1.14.3    networking:      dnsDomain: cluster.local      podSubnet: 10.244.0.0/16      serviceSubnet: 10.96.0.0/12    scheduler: {}  ClusterStatus: |    apiEndpoints:      node1:        advertiseAddress: 130.252.10.235        bindPort: 6443      node2:        advertiseAddress: 130.252.10.236        bindPort: 6443      node3:        advertiseAddress: 130.252.10.237        bindPort: 6443    apiVersion: kubeadm.k8s.io/v1beta1    kind: ClusterStatuskind: ConfigMapmetadata:  creationTimestamp: "2019-06-19T08:56:47Z"  name: kubeadm-config  namespace: kube-system  resourceVersion: "1034"  selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config  uid: 2b62d002-9270-11e9-a471-000c2925f18d

可以看到apiEndpoints有3个

kube-proxy配置

# kubectl get cm -n kube-system kube-proxy -oyamlapiVersion: v1data:  config.conf: |-    apiVersion: kubeproxy.config.k8s.io/v1alpha1    bindAddress: 0.0.0.0    clientConnection:      acceptContentTypes: ""      burst: 10      contentType: application/vnd.kubernetes.protobuf      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf      qps: 5    clusterCIDR: 10.244.0.0/16    configSyncPeriod: 15m0s    conntrack:      max: null      maxPerCore: 32768      min: 131072      tcpCloseWaitTimeout: 1h0m0s      tcpEstablishedTimeout: 24h0m0s    enableProfiling: false    featureGates:      SupportIPVSProxyMode: true    healthzBindAddress: 0.0.0.0:10256    hostnameOverride: ""    iptables:      masqueradeAll: false      masqueradeBit: 14      minSyncPeriod: 0s      syncPeriod: 30s    ipvs:      excludeCIDRs: null      minSyncPeriod: 0s      scheduler: ""      strictARP: false      syncPeriod: 30s    kind: KubeProxyConfiguration    metricsBindAddress: 127.0.0.1:10249    mode: ipvs    nodePortAddresses: null    oomScoreAdj: -999    portRange: ""    resourceContainer: /kube-proxy    udpIdleTimeout: 250ms    winkernel:      enableDSR: false      networkName: ""      sourceVip: ""  kubeconfig.conf: |-    apiVersion: v1    kind: Config    clusters:    - cluster:        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt        server: ;     name: default    contexts:    - context:        cluster: default        namespace: default        user: default      name: default    current-context: default    users:    - name: default      user:        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokenkind: ConfigMapmetadata:  creationTimestamp: "2019-06-19T08:56:51Z"  labels:    app: kube-proxy  name: kube-proxy  namespace: kube-system  resourceVersion: "244"  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-proxy  uid: 2dbcf684-9270-11e9-a471-000c2925f18d

kubelet-config

# kubectl get cm -n kube-system kubelet-config-1.14 -oyamlapiVersion: v1data:  kubelet: |    address: 0.0.0.0    apiVersion: kubelet.config.k8s.io/v1beta1    authentication:      anonymous:        enabled: false      webhook:        cacheTTL: 2m0s        enabled: true      x509:        clientCAFile: /etc/kubernetes/pki/ca.crt    authorization:      mode: Webhook      webhook:        cacheAuthorizedTTL: 5m0s        cacheUnauthorizedTTL: 30s    cgroupDriver: cgroupfs    cgroupsPerQOS: true    clusterDNS:    - 10.96.0.10    clusterDomain: cluster.local    configMapAndSecretChangeDetectionStrategy: Watch    containerLogMaxFiles: 5    containerLogMaxSize: 10Mi    contentType: application/vnd.kubernetes.protobuf    cpuCFSQuota: true    cpuCFSQuotaPeriod: 100ms    cpuManagerPolicy: none    cpuManagerReconcilePeriod: 10s    enableControllerAttachDetach: true    enableDebuggingHandlers: true    enforceNodeAllocatable:    - pods    eventBurst: 10    eventRecordQPS: 5    evictionHard:      imagefs.available: 15%      memory.available: 100Mi      nodefs.available: 10%      nodefs.inodesFree: 5%    evictionPressureTransitionPeriod: 5m0s    failSwapOn: false    fileCheckFrequency: 20s    hairpinMode: promiscuous-bridge    healthzBindAddress: 127.0.0.1    healthzPort: 10248    httpCheckFrequency: 20s    imageGCHighThresholdPercent: 85    imageGCLowThresholdPercent: 80    imageMinimumGCAge: 2m0s    iptablesDropBit: 15    iptablesMasqueradeBit: 14    kind: KubeletConfiguration    kubeAPIBurst: 10    kubeAPIQPS: 5    makeIPTablesUtilChains: true    maxOpenFiles: 1000000    maxPods: 110    nodeLeaseDurationSeconds: 40    nodeStatusReportFrequency: 1m0s    nodeStatusUpdateFrequency: 10s    oomScoreAdj: -999    podPidsLimit: -1    port: 10250    registryBurst: 10    registryPullQPS: 5    resolvConf: /etc/resolv.conf    rotateCertificates: true    runtimeRequestTimeout: 2m0s    serializeImagePulls: true    staticPodPath: /etc/kubernetes/manifests    streamingConnectionIdleTimeout: 4h0m0s    syncFrequency: 1m0s    volumeStatsAggPeriod: 1m0skind: ConfigMapmetadata:  creationTimestamp: "2019-06-19T08:56:47Z"  name: kubelet-config-1.14  namespace: kube-system  resourceVersion: "150"  selfLink: /api/v1/namespaces/kube-system/configmaps/kubelet-config-1.14  uid: 2b656b6f-9270-11e9-a471-000c2925f18d

这里虽然看的的是cgroupfs，但是kubelet启动时会读取/var/lib/kubelet/kubeadm-flags.env文件，此文件中已经指定了cgroupDriver为systemd，通过systemctl status kubelet我们也可以看的

验证集群高可用

为了验证高可用，我们可以关闭其中的master节点，查看是否生效。注意节点关闭后40s后，节点才会被标注为NotReady状态。5分钟后驱逐节点上的Pod

参考etcd