文章
文章用户商铺文档快讯圈子网址导航供求信息

Kubernetes集群维护

1、查看Kubernetes集群资源使用情况和日志

1、 查看资源使用情况
1）kubectl top查看Node使用CPU和内存情况

$ kubectl top node  #查看所有node
$ kubectl top node k8s01 #查看指定node

2）kubectl top查看Pod使用CPU和内存情况

$ kubectl top pod #查看所有Pod
$ kubectl top pod php-apache-64b6b9d449-t9h4z  #查看指定Pod

注意: top功能需要先安装metrics-server，安装步骤参考6.17章节

2、查看日志
1）K8s相关日志
Linux系统里记录的日志

$ journalctl  -u kubelet

K8s各组件日志
首先查看Pod name

$ kubectl get po -n kube-system  # calico-kube-controllers-xxxx, calico-node-xxx, coredns-xxx, etcd-xxx, kube-apiserver-xxx, kube-controller-manager-xxx, kube-proxy-xxx, kube-scheduler-xxx, metrics-server-xxx

查看指定Pod日志

$ kubectl logs -n kube-system calico-kube-controllers-798cc86c47-44525
$ kubectl logs -n kube-system kube-scheduler-k8s01

另外，可以加上-f选项动态查看指定pod日志，类似tail -f

2）应用日志
跟查看K8s组件日志一样，将Pod名字改为想查看的Pod名字即可

$ kubectl logs php-apache-64b6b9d449-t9h4z

另外，也可以进入到Pod内部去查看应用日志

$ kubectl exec -it pod-name -n namespace-name -- bash  ##进入后，再去查看具体的日志

有时候，我们的应用也会将日志目录给映射到Node上或者共享存储里，那样查看日志就方便多了。

2、维护Kubernetes集群CA证书

2.1 Kubernetes集群中的CA证书

如果使用Kubeadm部署集群，CA证书会自动生成，但如果用二进制方式部署则需要手动生成。
服务器上CA证书在哪里？

tree /etc/kubernetes/pki/
/etc/kubernetes/pki/
├── apiserver.crt
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
├── apiserver.key
├── apiserver-kubelet-client.crt
├── apiserver-kubelet-client.key
├── ca.crt
├── ca.key
├── etcd
│   ├── ca.crt
│   ├── ca.key
│   ├── healthcheck-client.crt
│   ├── healthcheck-client.key
│   ├── peer.crt
│   ├── peer.key
│   ├── server.crt
│   └── server.key
├── front-proxy-ca.crt
├── front-proxy-ca.key
├── front-proxy-client.crt
├── front-proxy-client.key
├── sa.key
└── sa.pub

Kubernetes为了安全，使用的是双向认证（ 除了客户端需要验证服务器的证书，服务器也要通过客户端证书验证客户端的身份。）

1、 CA证书
kubeadm安装的集群中我们都是用3套CA证书来管理和签发其他证书，一套CA给ETCD使用，一套是给kubernates内部组件使用，还有一套是给配置聚合层使用的，当然如果你觉得管理3套CA比较麻烦，您也可以用一套来管理。

1）Etcd证书
Etcd证书位于/etc/kubernetes/pki/etcd目录下，可以用ps查看Etcd的进程以及参数：

# ps aux |grep etcd |grep -v 'kube-apiserver'
root        1796  2.0  3.0 11215492 102036 ?     Ssl  10:18   0:29 etcd --advertise-client-urls=https://192.168.222.101:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://192.168.222.101:2380 --initial-cluster=aminglinux01=https://192.168.222.101:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.222.101:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.222.101:2380 --name=aminglinux01 --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

证书以及说明

├── etcd
│   ├── ca.crt ## 用于Etcd集群节点之间相互认证的CA证书
│   ├── ca.key ## 同上
│   ├── healthcheck-client.crt ## 当Etcd访问其它服务时，它作为客户端使用的CA证书
│   ├── healthcheck-client.key ## 同上
│   ├── peer.crt ## Etcd集群节点之间相互认证的peer证书，这是公钥
│   ├── peer.key ## 同上，这是私钥
│   ├── server.crt  ## Etcd对外提供服务时，比如apiserver连接etcd时，它作为服务端的CA证书，这是公钥
│   └── server.key  ## 同上，这是私钥

2） Kube-apiserver证书
Apiserver对应的证书目录在/etc/kubernetes/pki，可以用ps查看进程

ps aux |grep apiserver
root        1761  3.1 14.6 1254140 482468 ?      Ssl  10:18   4:38 kube-apiserver --advertise-address=192.168.222.101 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.15.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

证书以及说明：

tree /etc/kubernetes/pki/
/etc/kubernetes/pki/
├── apiserver.crt ##Apiserver作为服务端用到的CA证书
├── apiserver.key ##同上
├── apiserver-etcd-client.crt ##Apiserver作为客户端访问Etcd服务时用到的CA证书
├── apiserver-etcd-client.key ##同上
├── apiserver-kubelet-client.crt ##Apiserver访问kublet时，它作为客户端用到的证书
├── apiserver-kubelet-client.key ##同上
├── ca.crt ##用来签发k8s中其它证书CA证书，是一个根证书
├── ca.key ##同上
├── front-proxy-ca.crt ##配置聚合层（Apiserver扩展）的CA证书
├── front-proxy-ca.key ##同上
├── front-proxy-client.crt ##置聚合层（Apiserver扩展）的客户端证书
├── front-proxy-client.key ##同上
├── sa.key ##验证service account token用的私钥
└── sa.pub ##验证service account token用的公钥

3） kube-controller-manager用到的证书
查看进程：

ps aux |grep controller
root        1809  0.8  3.7 826096 122324 ?       Ssl  10:18   1:27 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-cidr=10.18.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.15.0.0/16 --use-service-account-credentials=true
systemd+    3328  0.0  1.7 1125840 56504 ?       Ssl  10:18   0:02 /usr/bin/kube-controllers

说明：
ps看到的进程用到的ca证书如下

/etc/kubernetes/pki/ca.crt 
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/sa.key

这些证书其实是Apiserver相关的证书，而kube-controller-manager用到的证书在/etc/kubernetes/controller-manager.conf这个配置文件里

cat /etc/kubernetes/controller-manager.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.222.101:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-controller-manager
  name: system:kube-controller-manager@kubernetes
current-context: system:kube-controller-manager@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-controller-manager
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGakNDQWY2Z0F3SUJBZ0lJRFRWVStoaEl3Wm93RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMU1qTXdOVFU1TVRGYUZ3MHlOREExTWpJd05UVTVNVFJhTUNreApKekFsQmdOVkJBTVRIbk41YzNSbGJUcHJkV0psTFdOdmJuUnliMnhzWlhJdGJXRnVZV2RsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5pMXN5b1BFZm53amNKdkFJUE9wVWF0VnJkeFNHTDYKcHQ4amY0YjJML05zY1RXcmZCdGt4MzVhbTZoQnFxRW9vUHRhNnBqaktNWWJNUGFWM0Y5SGRKd0NQbTdQbC9FOQowMFluQ2dKbUpkLytuckRLbk02czNRb2Iwa3VSamNHbVY0anNnTDhDZ3NKeldxSFdQOHRadGZNaWtHOTNwbEp0CmViL3VWMDRxMm5VbmxkWTNRWS90WHdGVUNGcjFXYVIxdHdjQS9ZelF5M0FOcVR0OEcwM2tFbFcwYkpkZXE3T3gKNzlGcCsrczlDajBQLzdUTHZ3WTFxNG1MM0hxdEFCQU5jZDBiYXR2cXhZNys3cnZVUHpsV3p0ZFNMaW9ha2U0LwptbDFhL0NqR0RnNDlmdkFOck9rU1E0a1B2c1dOM3Nscit6RDFJUit3a0RuL1ZQdklydDRGWEE4Q0F3RUFBYU5XCk1GUXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIKL3dRQ01BQXdId1lEVlIwakJCZ3dGb0FVM3RmS3BtV3N1QmNmaVlkaWkvOXpsWmdlZFRrd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBSmNFaXNhTEc5THZNNCs3aGRWcEt3U0hrUm1OR2F1NTQ2ODE0ZHZUZUlETWRHTE1lMUFoCnU2VzlhV055MzcvK2Z4eTg2eEEwYmpacE52ZnpSRWUyWnJqR1k2NXVMcjJreHJnNGljMWRSc1Y0aG9Fa2RVTFIKZG5ZdWd4Tk9FZ2xTNWZDUndGUXB1MkZ4KzFjbC9YS2ZrOUozUVdOUlhZWThVYkR1NTNHQW41U3NCcDJRV3I4OApDR3dJazFTUWUvMk9MY1JIRExlVThXdlMwN3NDVVZZWHl1VEd1c25qMnlqNUh6bExMZ29lLzYxbVNDWUpsTDFKCkVxT3NiVEdJUjlnTVdiRy9RZGlCWE5VNWZQWmFQTVBoNzdBUUY1UU9tcXhsUHJlcyt6TEJCNVgrdUJZOEw5dEwKYjh1QXFpMHFqRUNob0VPVjFqNnN2NThHditKMjk0ejJkZU09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBMkxXektnOFIrZkNOd204QWc4NmxScTFXdDNGSVl2cW0zeU4vaHZZdjgyeHhOYXQ4CkcyVEhmbHFicUVHcW9TaWcrMXJxbU9Nb3hoc3c5cFhjWDBkMG5BSSticytYOFQzVFJpY0tBbVlsMy82ZXNNcWMKenF6ZENodlNTNUdOd2FaWGlPeUF2d0tDd25OYW9kWS95MW0xOHlLUWIzZW1VbTE1dis1WFRpcmFkU2VWMWpkQgpqKzFmQVZRSVd2VlpwSFczQndEOWpORExjQTJwTzN3YlRlUVNWYlJzbDE2cnM3SHYwV243NnowS1BRLy90TXUvCkJqV3JpWXZjZXEwQUVBMXgzUnRxMityRmp2N3V1OVEvT1ZiTzExSXVLaHFSN2orYVhWcjhLTVlPRGoxKzhBMnMKNlJKRGlRKyt4WTNleVd2N01QVWhIN0NRT2Y5VSs4aXUzZ1ZjRHdJREFRQUJBb0lCQUJ0MlF5TVVUOUhka3dsTgpxY0lLUU5XSkI5RXRVT0FDSlgxbmNUUzRJL0YwRDVDZWo1YWxHY2JBVXJOcXh1V3ZVVjhvOFV1NGxhVXRNdkk4Cm9YV05oNUJ4bEZuVWdzdTJhdnliMjkvYjgvYkR2SFpvNXFBbU5jaWZKL0lkbXRvc2F6RlZ6eWJsZVk2TXNiS0sKaFFFTytFaThXNU5VbFVZaXkyZndHeTR6cmdWa2FCekRtOC9JdFJWS0VqS0R1QlJmQjZyM1l2Ujl4NTBMSi91NwpXUnlTVzNOTmZCZG93cE55clRZSHdRKzdjZ0RaYzgvRWkzVEZDa1lVOFAxV3RwanNTQmdxa2g4am5PODhxc0tFCk5mZWt0TEZJTTRRMGVDTTdJK0RWazVjR0ZldnFRYmxlcFdhNEMvOGRVQ0dJRWxUSXRWbzhSTEpKMDFVQ3lWWFgKenp5MzBTa0NnWUVBNmNJZENGNGloS3piTzFlbG03MVlub3ZmZ3hJSnZiN3N1UW4xeGdGQ3FrUlA3ZzFMYktoUApGd05mL2U0ekJNYklkQWJKL1liNjl6ZnVXV3krZ0IzT2o1MUk1ekdGZnFVOTU0RGJYcFdrajZRbERDMHp1V1p6Ckg0cE5IYVNTbUdLSHlNVnBPMEU4Q2RNbVpTT3czMUhwaXF4cys5cXpLY2VnR240RS9RTzh1dk1DZ1lFQTdWUlIKc0xpU3dYdzdLc09SZ08yMDcvb3dYb1ozNUdSaHl2WG1PK2Iwc2NmUTFOSTltWm54YkJCZ0JYYkFMRnFWc210eAo4NFFWbHdnUGg5cTlwUHBqR2phZHM5cU5XTk95Ry9CZ2hzSkE5SkxNVVhuMVdvOEl0VVRnVEF5Y1c2OTNCbXBBCjJ5aGN5Tk0vU0RpWEgxQTA2VExUc3dQK3ZNTGgrZ1JuR0RTaEtYVUNnWUVBdkhlRC91WWV3SWU1Ymp2NVN2cTcKZzREVW9aR2Vodnh6VWJFbEZoS1lGRDFsa0ZIcHVONXVmZWg3c1hrbWF6aDJENDRWdHpIb0VBQnBUSEl2bVkzdQpwNzJ4TksycHF2VkJWdkIrYmVFS3Y4RWhHWk1zTzlQTkIrWHl6TzZPUHd4WjI5YkdSRDhSVC82TTIwaU9aMDljCmt6RDM0WXR2bmtUVDRlZ3V6MnYrODE4Q2dZQkN0VUdrNzZ4b2NEZmRSRTFBdFEvM3FCWXlhNCtGaGVXcSs4VTAKTHkyMlR0alhDU1ZCYUhEZ2xCVzVVM0NleHZOdXpCNmllUXpzZjlxRnNDbEFGeFVHQ0RYNW1NUEVKUFRhUDQyUwpNMFc0dVE1dkZGMnZqNmE5OU5jcTZSeUY5T0w2emFEZk9SQWpicVZKOG9ZZnloYWxPYUVobjB6WjE5bFd3aWdNCk9MYllCUUtCZ1FDbGF5dDl4UGlUQkk3amJXdk8zK1VUN05wRDJQRkdXSzlBUWJOc2NCcUVKV21BaGVMaFdKZHgKUEFJaTgrbmRiQjQwellFWXRqZlp1QVk5TkQraWVCakNKOTZ2TTZjM0FXNVAwckhJREJORHFWd2ZNRjJjYUlhTQpZNlNWa08wV3NvSSt4akRYNU9nbzkxR04rWGR5VnNvSkFJRE5ZUzhQSGF1ZVc2L282K1NDcWc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Kubernetes 这里的设计是这样的：kube-controller-mananger、kube-scheduler、kube-proxy、kubelet等组件，采用一个kubeconfig 文件中配置的信息来访问 kube-apiserver。该文件中包含了 kube-apiserver 的地址，验证 kube-apiserver 服务器证书的 CA 证书，自己的客户端证书和私钥等访问信息。

4）Kube-scheduler
跟Kube-controller-namager一样，Kube-scheduler用的也是kubeconfig

cat /etc/kubernetes/scheduler.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.222.101:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-scheduler
  name: system:kube-scheduler@kubernetes
current-context: system:kube-scheduler@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-scheduler
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREVENDQWZXZ0F3SUJBZ0lJQkFIcmNyRTJGMzR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMU1qTXdOVFU1TVRGYUZ3MHlOREExTWpJd05UVTVNVFJhTUNBeApIakFjQmdOVkJBTVRGWE41YzNSbGJUcHJkV0psTFhOamFHVmtkV3hsY2pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU13TXVRd05TTks3NURBTWk2WTlLaHpYQ1dHOXVNSEpjNmFNb1krOUs2WjQKbmRQNGJHNlU1WU5WbE9xVkd5UWoyejFEU05yLzYvdnJTRS84ZEU3SitGT2VDZEhCTWp6ZjhjdExOU0U1WGo2Kwo4eFYwMTNHNlo4aTh4d3JwZjZUZ2VpTmJ0UjJkaytlUXJCNDh0MmxMWUE2blZSSVl5RDRZR3EyNTdaRlM4Z3dTCmZlN3YzUU9Ud1RYeDlTeHRwc2x2YzV5NFllUXBuWVN4L1lLeUhwUWFFMjBLZCtxcWFQTjBnMFVOSTF0ZmdwbDAKc002T2F5RERwNVZSeml5TkhaL2RpK0JZYldOSnlaWWsxZ3h0dVdMeGd6eGxVRGRaVk9KaFRxSk5yc081L2RRRwpRNkxDaXJWUWk4aWpwbHBlTkRucFVicFRydno5TWc0MEpqSTJWQU92K0YwQ0F3RUFBYU5XTUZRd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0h3WUQKVlIwakJCZ3dGb0FVM3RmS3BtV3N1QmNmaVlkaWkvOXpsWmdlZFRrd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQgpBRnVkRjJ2VmNtQjdJVFNYa29GUTlWUVFwQXVxLzZLZDFvcG9XVE9yQW85ZG9Bc2JMWlNKKzY3UjlUeFNEbFRnClFLZEdEYTJZNDZtZGtpOXgzUFdXZ0h3S0xyMGlWS0s0aEc4Zy8zbEhhMXJDTnJHMFo0UG1GR3c1Q3RXazZtWkgKZFlJaEs2ODZJRjcvdHZMbFJsdk12NDdXKzVNZGtFZXdKamNlWEtqR2ZDRmRaa2ErTmFySTV5MXhwbkxOYmtwbwpsY1dTKzUyb3BVZGlpcEJGOUt6bzdwL0ZuWEcweVQvQnFwbXdHUUdDUlZCdll5L1NwanpTb0hKK25vRytIK2dKCjIvWXR0eVdlQnEvQ2xCOHhhUHRJRDVLT0psa2VCOGRFOENoSjk1TkxKdmN5aWpUcmhuLzNBSGYxUzZsWmtkTHoKbUcySDZacE5zRHdCUjlPcjMvMmNINFE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBekF5NURBMUkwcnZrTUF5THBqMHFITmNKWWIyNHdjbHpwb3loajcwcnBuaWQwL2hzCmJwVGxnMVdVNnBVYkpDUGJQVU5JMnYvcisrdElUL3gwVHNuNFU1NEowY0V5UE4veHkwczFJVGxlUHI3ekZYVFgKY2JwbnlMekhDdWwvcE9CNkkxdTFIWjJUNTVDc0hqeTNhVXRnRHFkVkVoaklQaGdhcmJudGtWTHlEQko5N3UvZApBNVBCTmZIMUxHMm15Vzl6bkxoaDVDbWRoTEg5Z3JJZWxCb1RiUXAzNnFwbzgzU0RSUTBqVzErQ21YU3d6bzVyCklNT25sVkhPTEkwZG45Mkw0Rmh0WTBuSmxpVFdERzI1WXZHRFBHVlFOMWxVNG1GT29rMnV3N245MUFaRG9zS0sKdFZDTHlLT21XbDQwT2VsUnVsT3UvUDB5RGpRbU1qWlVBNi80WFFJREFRQUJBb0lCQUdibDc1YzFSOGtGZVZBRgpzNUswZTMwcHI1QjdsU1VjU3RidFNqZnJKSTBCZkUyOWxDaTdxM1hzZlhsQ2tmcEtucStwTTU5RXVhTkRoaWJYCjMyc3dES3RXSWd1RnlxaktJZXZyMURJWjlQM2RXMFl4c3NlbVFkb0g0TFFQQmhDR0p2R1h1S0RqcXdkMmZHV1AKSnNyUmVQSkt6ZXFmVnJPQmdrcWpYVHNRd2dEMWxySm9mUzFyUk0vUXlGYzNZYWUya1l3SEhtbUV3K25pWFJ3YwpPM0FkNHZDZGVYN2lwbmUrTFE1dGpCWWVwVHJlU0VlNS80alNyV1ZaN0F3VDAvZW02NnFJbGRMcHZPQXRpZm9xCjFQSXc1VHBPNGxlSHF1K2VKRlJLcU9sM3h0Sm1hbUIrODJTT1c2Z0pkNmlubEZ5UmRBNXhJeG1RN2pPWFVhTksKSnVkRi9oMENnWUVBKy9DWmxHSzVQRGJ4a0pGek1sYjBoNFk2OTBad096NThIRHMwVWxPRVkzN2VCSVF4MjV0Uwp3TlJiTHB0dU9WNmhLeWEzVTdvSTN6YzBzZHVtdk1hSXVtQWVOb08vZUR3V0tlRm1JSFBRbzJrZXVmVFAyUyt0Cm5ydndGcFVxdnk0UW4rRURBTTljUVZyMEN2TFdMSmNYSTVxK3ZVdUJlNjVnaFFlL2xnT1dzMDhDZ1lFQXoxYU0KUXZWNzFtUWxLWktUZE5GNTdiK211UTQvLytMblZianFIVjNRM2VaQkFneVhrb254SDV6eG0vdDdIZkxRbGM1ZAo5TThIWThEdmg1VnFOMVdDYnJvb2lQaFF6Z2lDM01qamJoc2FwcVppNk1pTFJ0UXlBTFFiOGVWYWs3RHBINjNnCmdiN1dxUTByUHBjTFA0bk5lVFQxb0Z5ZjhVMXJBR1l1RVNpd1hwTUNnWUVBaGxOTHJ4L0wzMXowQXI1cUN3YnQKcld4VitBMG9QWTRkSWZnMjkwWHNLaGcwUzNnb0ZReUZYS1pjVDBjeng0YTZxNFVpN1pNY3M5WjlVKzdtd1hPZwo1cTNrcmZBa24vaDVZSzc2Z29iOVVJTDFqUFFIOWFaRURZUHFpRC9UNE1hd3VtMS82bWlrcVh2UzdodDNNYU1GCkJVaTJOYnNTT0YxS3ZPTGF0U01Jc0dVQ2dZQTNxV3A2UjJENWkyVVdwZzBFSFlCNFBTMUxIUEk5aGZMSTVwMmoKU0o1Y0ZBWjd3Rm1vczNPU0l4WVI1QStIV0xwNm5TNUlISlJGU1hTRWw2NHVNTHlWY1ZDVWhPVmNpcGhuSVY4OQpIZlducTU5K0V1aWhHVEpiVWY3Mmo3WWpWQ2tob2hKVmdxQXFQaWpQNHNqWVEreHZqN0lwWFBScldYZFNZaHdYCjZ5NStGUUtCZ1FDUjR3VVdKS29pdkdIeExVUkQ0U0NEWnorYlFOaVFMUkJ6RjR1WUVSOWxjTlp1cXdxY3dPaXkKQXpCY0tMR3RlaGxpcjJOR21XWUpTZ0dBenRQWmZ5RnFPcUlHMU9QT3F3c1dqTjNISkdoREFGWktWLzBUb3dsMQo2MVBUOHV2ODhZTGZOeU1mSTh4UzY5QWNhQlpYWk1TTkFkdlZSKysrcFBmZDhrSEQxYVhDdUE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

可以看到，配置文件里的证书内容和Kube-controller-namager一样。

5）Kubelet
Kubelet用的也是kubeconfig

cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.222.101:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:aminglinux01
  name: system:node:aminglinux01@kubernetes
current-context: system:node:aminglinux01@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:aminglinux01
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

这个certificate-authority-data对应的数据和上面几个组件一样。而最下面的user配置段里有client-certificate和client-key，为kubelet作为客户端时用的CA证书。

2、续签证书
CA证书是有时效性的，如果过期了会影响到业务。如何查看证书何时到期呢？

$ openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt

notBefore=Sep 27 12:25:11 2023 GMT  # 什么时候产生
notAfter=Sep 26 12:30:12 2024 GMT   # 什么时候失效

可见证书有效期为1年。

如果你的Kubernetes集群是由kubeadm搭建，那么还有一种方法，使用kubeadm查看整个集群所有证书有效期：

$ kubeadm  certs check-expiration

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 26, 2024 12:30 UTC   329d            ca                      no      
apiserver                  Sep 26, 2024 12:30 UTC   329d            ca                      no      
apiserver-etcd-client      Sep 26, 2024 12:30 UTC   329d            etcd-ca                 no      
apiserver-kubelet-client   Sep 26, 2024 12:30 UTC   329d            ca                      no      
controller-manager.conf    Sep 26, 2024 12:30 UTC   329d            ca                      no      
etcd-healthcheck-client    Sep 26, 2024 12:30 UTC   329d            etcd-ca                 no      
etcd-peer                  Sep 26, 2024 12:30 UTC   329d            etcd-ca                 no      
etcd-server                Sep 26, 2024 12:30 UTC   329d            etcd-ca                 no      
front-proxy-client         Sep 26, 2024 12:30 UTC   329d            front-proxy-ca          no      
scheduler.conf             Sep 26, 2024 12:30 UTC   329d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Sep 24, 2033 12:30 UTC   9y              no      
etcd-ca                 Sep 24, 2033 12:30 UTC   9y              no      
front-proxy-ca          Sep 24, 2033 12:30 UTC   9y              no

如果到期，使用kubeadm可以续签证书，方法是：

$ kubeadm certs renew all

看输出，最后面有一句提醒，You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
需要重启这些服务：kube-apiserver, kube-controller-manager, kube-scheduler and etcd

3、Kubernetes集群版本升级

1）为什么要升级

  • ① 为了使用新功能
  • ② 当前版本存在bug
  • ③ 当前版本存在安全漏洞

2）注意事项：

  • ① 不支持跨版本升级（这个跨版本指的是主要版本和次要版本，比如1.24.2，其中1为主要版本，24为次要版本，2为补丁版本）
  • 示例:
1.20.2 -->  1.21.4  支持
1.20.2 -->  1.22.3  不支持
1.25.0 --> 1.25.4  支持
  • ② 升级前做备份
  • ③ 升级前拿测试环境做演练

3）升级流程

  • ① Node层面
  • 先升级Master k8s01（如果有多个Master，需要一台一台升级） –> 再升级Worker节点k8s02和k8s03
  • ② 软件层面
  • 升级kubeadm –> 节点执行drain操作 –> 升级各组件（apiserver, coredns, kube-proxy, controller-manager, scheduler等）–> 取消drain –> 升级kubelet和kubectl

官方升级文档： https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

4）升级步骤
示例： 1.26.9 –> 1.27.6

① 升级Master
查看最新版本

$ yum list --showduplicates kubeadm

升级kubeadm

$ yum install -y kubeadm-1.27.6-0  ##需要指定版本号

驱逐Master上的Pod

$ kubectl drain master01 --ignore-daemonsets

查看集群是否能够升级

$ kubeadm upgrade plan

执行升级

$ kubeadm upgrade apply v1.27.6

升级kubelet和kubectl

$ yum install -y kubelet-1.27.6-0  kubectl-1.27.6-0

重启kubelet

$ systemctl daemon-reload
$ systemctl restart kubelet

恢复调度，上线

$ kubectl uncordon master01

② 升级Work第一个节点
升级kubeadm（node01上执行）

$ yum -y install kubeadm-1.27.6-0  ##需要指定版本号

驱逐node01上的Pod（master01上执行）

$ kubectl drain node01 --ignore-daemonsets --delete-emptydir-data

升级kubelet配置（node01上执行）

$ kubeadm upgrade node

升级kubelet和kubectl（node01上执行）

$ yum install -y kubelet-1.27.6-0  kubectl-1.27.6-0

重启kubelet（node01上执行）

$ systemctl daemon-reload
$ systemctl restart kubelet

恢复调度，上线（master01上执行）

$ kubectl uncordon node01

③ 升级Work第二个节点
升级kubeadm（node02上执行）

$ yum install -y kubeadm-1.27.6-0  ##需要指定版本号

驱逐node02上的Pod（master01上执行）

$ kubectl drain node02 --ignore-daemonsets --delete-emptydir-data

升级kubelet配置（node02上执行）

$ kubeadm upgrade node

升级kubelet和kubectl（node02上执行）

$ yum install -y kubelet-1.27.6-0  kubectl-1.27.6-0

重启kubelet（node02上执行）

$ systemctl daemon-reload
$ systemctl restart kubelet

恢复调度，上线（master01上执行）

$ kubectl uncordon node02

如果有其它Node，继续模仿上面的③操作即可。

查看集群状态

$ kubectl get node
NAME       STATUS     ROLES           AGE   VERSION
master01   Ready      control-plane   35d   v1.27.6
master02   Ready      control-plane   35d   v1.27.6
master03   Ready      control-plane   35d   v1.27.6
node01     NotReady   worker          35d   v1.27.6
node02     Ready      worker          35d   v1.27.6
node03     Ready      worker          35d   v1.27.6

4、Kubernetes节点上线和下线

4.1 新节点上线

1）准备工作
关闭防火墙、SELINUX

$ systemctl stop firewalld && systemctl disable firewalld
$ sed -i 's/enforcing/disabled/g' /etc/selinux/config
$ setenforce 0

配置主机名

node节点,名称为node04

$ hostnamectl set-hostname node04

配置Host文件

$ cat > /etc/hosts <<EOF
10.0.1.200 master01
10.0.1.201 master02
10.0.1.202 master03
10.0.1.203 node01
10.0.1.204 node02
10.0.1.205 node03
10.0.1.206 node04
EOF

时间同步配置

$ yum install -y chrony
$ systemctl start chronyd && systemctl enable chronyd

配置内核转发及网桥过滤

$ cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
EOF

添加网桥过滤及内核转发配置文件
$ cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

加载br_netfilter模块
$ modprobe overlay
$ modprobe br_netfilter


查看是否加载        
$ lsmod | grep    br_netfilter    
br_netfilter    22256   0
bridge    151336  1 br_netfilter


加载网桥过滤及内核转发配置文件
$ sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0

安装ipset及ipvsadm

$ dnf install -y ipset ipvsadm

配置ipvsadm模块加载方式，添加需要加载的模块

$ cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

授权、运行、检查是否加载

$ chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

关闭SWAP分区

$ swapoff -a    # 临时 --有用
$ sed -ri 's/.*swap.*/#&/' /etc/fstab

配置containerd
官方文档：https://github.com/containerd/containerd/blob/main/docs/getting-started.md
下载并解压
地址：https://github.com/containerd/containerd/releases

$ wget https://github.com/containerd/containerd/releases/download/v1.7.6/containerd-1.7.6-linux-amd64.tar.gz
$ tar xzvf /usr/local containerd-1.7.6-linux-amd64.tar.gz -C /usr/local

配置systemd

$ wget -O /usr/lib/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
$ systemctl daemon-reload
$ systemctl enable --now containerd

配置runc
地址： https://github.com/opencontainers/runc/releases

$ wget https://github.com/opencontainers/runc/archive/refs/tags/v1.1.8.tar.gz
$ install -m 755 runc.amd64 /usr/local/sbin/runc

配置CNI plugins
地址：https://github.com/containernetworking/plugins/releases

$ wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
$ mkdir -p /opt/cni/bin
$ tar xzvf cni-plugins-linux-amd64-v1.1.1.tgz -C  /opt/cni/bin

配置cgroup

$ mkdir /etc/containerd
$ /usr/local/bin/containerd config default > /etc/containerd/config.toml
$ sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
$ sed -i 's/sandbox_image = "registry.k8s.io/pause:3.8"/sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"/g' /etc/containerd/config.toml
$ systemctl restart containerd

3）配置kubernetes仓库

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF

说明：kubernetes用的是RHEL7的源，和8是通用的

4）安装kubeadm和kubelet

$ yum install -y kubelet-1.27.6 kubeadm-1.27.6 kubectl-1.27.6

启动kubelet服务

$ systemctl start kubelet.service
$ systemctl enable kubelet.service

为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。

$ vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

或 sed -i 's/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"/g' /etc/sysconfig/kubelet

5）设置crictl连接 containerd

$ crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

6）到master节点上，获取join token

$ kubeadm token create --print-join-command

7）到新节点，加入集群

$ kubeadm join k8s.zhoumx.cc:6443 --token 17c6pl.9lwnl4q39wa0lx68 --discovery-token-ca-cert-hash sha256:1299ad3afbf3e37c0421bb6abbf2a45191accdbeca410b358546db69d2e1c293

8）master上查看node信息

$ kubectl get node

NAME       STATUS     ROLES           AGE   VERSION
master01   Ready      control-plane   35d   v1.27.6
master02   Ready      control-plane   35d   v1.27.6
master03   Ready      control-plane   35d   v1.27.6
node01     Ready      worker          35d   v1.27.6
node02     Ready      worker          35d   v1.27.6
node03     Ready      worker          35d   v1.27.6
node04     NotReady   <none>          5s    v1.27.6

等待calico pod完成后再次查看node信息

$ kubectl get node
NAME       STATUS   ROLES           AGE     VERSION
master01   Ready    control-plane   35d     v1.27.6
master02   Ready    control-plane   35d     v1.27.6
master03   Ready    control-plane   35d     v1.27.6
node01     Ready    worker          35d     v1.27.6
node02     Ready    worker          35d     v1.27.6
node03     Ready    worker          35d     v1.27.6
node04     Ready    <none>          2m41s   v1.27.6

4.2 节点下线

1）下线之前，先创建一个测试Deployment

命令行创建deployment，指定Pod副本为7
$ kubectl create deployment testdp2 --image=nginx:1.23.2 --replicas=7

查看Pod

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
testdp2-7b965b84bf-5gmr7            1/1     Running   0          2m39s   10.224.59.214    master02   <none>           <none>
testdp2-7b965b84bf-7b6h4            1/1     Running   0          2m39s   10.224.241.97    master01   <none>           <none>
testdp2-7b965b84bf-7wz42            1/1     Running   0          2m39s   10.224.235.28    master03   <none>           <none>
testdp2-7b965b84bf-bbdbp            1/1     Running   0          2m39s   10.224.186.211   node03     <none>           <none>
testdp2-7b965b84bf-qqwwd            1/1     Running   0          2m39s   10.224.196.143   node01     <none>           <none>
testdp2-7b965b84bf-qwkg2            1/1     Running   0          2m39s   10.224.140.65    node02     <none>           <none>
testdp2-7b965b84bf-wmqzk            1/1     Running   0          2m39s   10.224.248.195   node04     <none>           <none>

2）驱逐下线节点上的Pod，并设置不可调度（master01上执行）

$ kubectl drain node04  --ignore-daemonsets

3）恢复可调度（master01上执行）

$ kubectl uncordon  node04

4）移除节点

$ kubectl delete node  node04

查看node信息

$ kubectl get node
NAME       STATUS   ROLES           AGE   VERSION
master01   Ready    control-plane   35d   v1.27.6
master02   Ready    control-plane   35d   v1.27.6
master03   Ready    control-plane   35d   v1.27.6
node01     Ready    worker          35d   v1.27.6
node02     Ready    worker          35d   v1.27.6
node03     Ready    worker          35d   v1.27.6
声明：本站所有文章，如无特殊说明或标注，均为本站原创发布。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。

给TA打赏
共{{data.count}}人
人已打赏
Kubernetes
Kubernetes云原生

Kubernetes 存储

2025-4-9 8:38:38

SkyWalking云原生

SkyWalking全链路监控

2025-4-9 8:41:37

0 条回复 A文章作者 M管理员
    暂无讨论，说说你的看法吧
❯

解锁会员权限

开通会员

解锁海量优质VIP资源

立刻开通

个人中心
购物车
优惠劵
今日签到
有新私信 私信列表
搜索

  • 扫码打开当前页

返回顶部

文章目录

← Index