用户反映访问系统无法访问,页面刷新不出来。
访问kubesphere页面和业务系统,无响应。
以下为测试环境模拟复现,有些报错不完全一致,大体流程可参考。
docker ps
看到k8s相关容器一直在重启
查看节点情况kubectl get nodes
该截图为测试环境复现截图,现场执行时未提示过期,而是6443端口连接失败
查看pod情况kubectl get pods -A
该截图为测试环境复现截图,现场执行时未提示过期,而是6443端口连接失败
由于现场kubectl命令无法使用,只好使用docker查看日志,优先查看kube-apiserver日志
docker ps | grep apiserver
第二个为对应pod
docker logs -f 0dfc2f141a24 #日志较多,使用下条命令
docker logs -n 100 0dfc2f141a24
现场排查时只看到证书过期,未看到最后一行时间2018年在2024年之前的提示,不确定当时是不是看漏了
既然看到证书过期,由于现场去年部署,忘记具体时间,以为确实过期了,没有自动续期导致,只好先查看过期时间。
发现到2025年过期,有效期却剩余6年,于是查看系统时间date
,发现为2018年
PS:证书有效期有时间范围,当前时间晚于结束时间或者早于开始时间都不行。
首先甩锅
联系客户那边修改云服务器时间
修改时间后,很快集群恢复正常。
再次查看证书有效期
通过kk工具查看集群证书有效期和重新生成
[root@node1 ks]# ./kk certs
cluster certs
Usage:
kk certs [command]
Available Commands:
check-expiration Check certificates expiration for a Kubernetes cluster
renew renew a cluster certs
Flags:
--debug Print detailed information
-h, --help help for certs
--ignore-err Ignore the error message, remove the host which reported error and force to continue
--namespace string KubeKey namespace to use (default "kubekey-system")
-y, --yes Skip confirm check
Use "kk certs [command] --help" for more information about a command.
通过kubeadm
[root@node1 4]# kubeadm certs
Commands related to handling kubernetes certificates
Usage:
kubeadm certs [command]
Aliases:
certs, certificates
Available Commands:
certificate-key Generate certificate keys
check-expiration Check certificates expiration for a Kubernetes cluster
generate-csr Generate keys and certificate signing requests
renew Renew certificates for a Kubernetes cluster
Flags:
-h, --help help for certs
Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level)
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity
Use "kubeadm certs [command] --help" for more information about a command.
使用kk 查询和生成证书时,依赖ssh端口,如果不是22端口,需要指定 -f 集群创建配置文件
./kk certs check-expiration
kubeadm certs check-expiration
使用kk 查询和生成证书时,依赖ssh端口,如果不是22端口,需要指定 -f 集群创建配置文件
./kk certs renew
kubeadm certs renew all
[root@node1 ks]# kubeadm certs renew -h
This command is not meant to be run on its own. See list of available subcommands.
Usage:
kubeadm certs renew [flags]
kubeadm certs renew [command]
Available Commands:
admin.conf Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
all Renew all available certificates
apiserver Renew the certificate for serving the Kubernetes API
apiserver-etcd-client Renew the certificate the apiserver uses to access etcd
apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
controller-manager.conf Renew the certificate embedded in the kubeconfig file for the controller manager to use
etcd-healthcheck-client Renew the certificate for liveness probes to healthcheck etcd
etcd-peer Renew the certificate for etcd nodes to communicate with each other
etcd-server Renew the certificate for serving etcd
front-proxy-client Renew the certificate for the front proxy client
scheduler.conf Renew the certificate embedded in the kubeconfig file for the scheduler manager to use
Flags:
-h, --help help for renew
Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level)
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity
Use "kubeadm certs renew [command] --help"for more information about a command.