K8s 踩坑日记1-服务器重启后k8s无法使用

使用 Vmware 搭建的学习k8s环境，虚拟机重启后，发现k8s没启动，现在要把他们全部启动起来

使用命令查看 kubectl get nodes

1	报错：The connection to the server 192.168.100.160:6443 was refused - did you specify the right host or port?

检查 Docker 及容器状态

1	systemctl status docker

检查容器是否启动

1	docker ps -a

好家伙，全都没启动，尝试手动启动

挨个启动容器，最后有三个容器无法启动（这三个是网络插件）

到这里，kubectl get node 命令可以使用，且 node1 和 node2 都为 Ready状态，mater为 NotReady

1
2
3

k8s-master   NotReady   control-plane,master   61m   v1.21.10
k8s-node1    Ready      <none>                 60m   v1.21.10
k8s-node2    Ready      <none>                 60m   v1.21.10

CNI 网络插件容器，无法启动，查看容器日志信息

[root@k8s-master ~]# docker logs -f d9df99e20ba8
I1022 12:13:07.803445       1 main.go:204] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W1022 12:13:07.803595       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1022 12:13:07.904757       1 kube.go:126] Waiting 10m0s for node controller to sync
I1022 12:13:07.904902       1 kube.go:420] Starting kube subnet manager
I1022 12:13:08.905497       1 kube.go:133] Node controller sync successful
I1022 12:13:08.905545       1 main.go:224] Created subnet manager: Kubernetes Subnet Manager - k8s-master
I1022 12:13:08.905551       1 main.go:227] Installing signal handlers
I1022 12:13:08.906301       1 main.go:467] Found network config - Backend type: vxlan
I1022 12:13:08.906343       1 match.go:206] Determining IP address of default interface
I1022 12:13:08.906884       1 match.go:259] Using interface with name ens32 and address 192.168.100.160
I1022 12:13:08.906941       1 match.go:281] Defaulting external address to interface address (192.168.100.160)
I1022 12:13:08.907057       1 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I1022 12:13:09.004262       1 main.go:416] Current network or subnet (10.244.0.0/16, 10.244.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I1022 12:13:09.305994       1 main.go:342] Setting up masking rules
I1022 12:13:09.309750       1 main.go:364] Changing default FORWARD chain policy to ACCEPT
I1022 12:13:09.405880       1 main.go:379] Wrote subnet file to /run/flannel/subnet.env
I1022 12:13:09.406186       1 main.go:383] Running backend.
I1022 12:13:09.406795       1 vxlan_network.go:61] watching for new subnet leases
I1022 12:13:09.504856       1 main.go:404] Waiting for all goroutines to exit
I1022 12:13:09.603554       1 iptables.go:219] bootstrap done
I1022 12:13:09.605576       1 iptables.go:219] bootstrap done
I1022 12:40:10.765213       1 main.go:204] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W1022 12:40:10.765942       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1022 12:40:10.766334       1 client_config.go:622] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E1022 12:40:10.767008       1 main.go:221] Failed to create SubnetManager: fail to create kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I1022 12:40:21.858010       1 main.go:204] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W1022 12:40:21.858133       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1022 12:40:21.858157       1 client_config.go:622] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E1022 12:40:21.858518       1 main.go:221] Failed to create SubnetManager: fail to create kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

问题：

很明显 apiserver 没有起来，但是 apiserver 安装的时候是以容器的方式安装的

尝试百度，看了一堆文章，有的文章居然是 kubeadm init，shit 这还有什么好看的

知乎找到个

静态 pod 可以直接被 kubelet 启动，那很有可能是 kubelet 没有正确启动，尝试如下：每台机器上都要操作

swapoff -a
setenforce 0
systemctl daemon-reload
systemctl restart kubelet

然后再用docker ps查看，可以看到 master 节点上的很多 k8s容器已经启动起来了,用kubectl get nodes 发现已经启动了

查看容器状况docker ps -a,what?为什么有这么容器，至此还没法解决，但 k8s 已经可以使用