【信创-k8s】海光/兆芯+银河麒麟V10离线部署k8s1.31.8+kubesphere4.1.3

编码如写诗

发布于 2025-05-21 20:40:34

29300

代码可运行

运行总次数：0

代码可运行

❝KubeSphere V4已经开源半年多，而且v4.1.3也已经出来了，修复了众多bug。介于V4优秀的LuBan架构设计，核心组件非常少，资源占用也显著降低，同时带来众多功能和便利性。我们决定与时俱进，使用1.30+版本的Kubernetes 和v4版本的KubeSphere。

环境涉及软件版本信息

服务器芯片: 海光3350/兆芯开先KX-5000
操作系统：麒麟 V10 SP3
Containerd: 1.7.13
Kubernetes：v1.31.8
KubeSphere：v4.1.3
KubeKey: v3.1.9
Docker: 24.0.9
DockerCompose: v2.26.1
Harbor: v2.10.1

服务器基本信息

[root@0003 k8s-init]# uname -a
Linux node1 4.19.90-52.22.v2207.ky10.x86_64 #1 SMP Tue Mar 14 12:19:10 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

[root@0003 k8s-init]# cat /etc/os-release 
NAME="Kylin Linux Advanced Server"
VERSION="V10 (Lance)"
ID="kylin"
VERSION_ID="V10"
PRETTY_NAME="Kylin Linux Advanced Server V10 (Lance)"
ANSI_COLOR="0;31"

1 说明

本文由 [编码如写诗-天行1st] 原创编写，有任何问题可添加作者微信 [sd_zdhr] 获取协助。转载请获取作者同意。

❝关于我：

主要从事后端开发，兼具前端、运维及全栈工程师，热爱Golang、Docker、kubernetes、KubeSphere。
信创服务器k8s&KubeSphere布道者、KubeSphere离线部署布道者
公众号：编码如写诗，作者：天行1st，微信：sd_zdhr

2 前提条件

参考如下示例准备至少三台主机

主机名	IP	架构	OS	用途
node	192.168.3.249	x86_64	Ubuntu24.04	联网主机，用于制作离线包
master	192.168.85.138	x86_64	麒麟V10	离线环境主节点
node1	192.168.85.139	x86_64	麒麟V10	离线环境的工作节点和镜像仓库节点

3 构建离线包

在node可联网节点上操作

3.1 下载kk

curl -sSL https://get-kk.kubesphere.io | sh -

3.2 编辑 manifest 文件

vi manifest-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind:Manifest
metadata:
name:sample
spec:
arches:
-amd64
operatingSystems:[]
kubernetesDistributions:
-type:kubernetes
    version:v1.31.8
components:
    helm:
      version:v3.14.3
    cni:
      version:v1.2.0
    etcd:
      version:v3.5.13
    containerRuntimes:
    -type:docker
      version:24.0.9
    -type:containerd
      version:1.7.13
    calicoctl:
      version:v3.27.4
    crictl:
      version:v1.29.0
    docker-registry:
      version:"2"
    harbor:
      version:v2.10.1
    docker-compose:
      version:v2.26.1
images:
-registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9
-registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.31.8
-registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.31.8
-registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.31.8
-registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.31.8
-registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3
-registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
-registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4
-registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
-registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
-registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4
# ks
-swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-extensions-museum:v1.1.2
-swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-controller-manager:v4.1.3
-swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-apiserver:v4.1.3
-swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-console:v4.1.3
-swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.16
registry:
    auths:{}

3.3 导出离线制品

export KKZONE=cn
./kk artifact export -m manifest-sample.yaml -o artifact-k8s-1318-ks413.tar.gz

3.4 下载 KubeSphere Core Helm Chart

安装helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

下载 KubeSphere Core Helm Chart

VERSION=1.1.3     # Chart 版本
helm fetch https://charts.kubesphere.io/main/ks-core-${VERSION}.tgz

4 离线部署准备

4.1 移除麒麟系统自带的podman

podman是麒麟系统自带的容器引擎，容易与docker冲突。不卸载的话后续coredns/nodelocaldns会受影响无法启动以及各种docker权限问题。这里直接卸载，以便后续使用docker，所有节点执行

yum remove podman

4.2 将安装包拷贝至离线环境

将下载的 KubeKey 、制品 artifact 、脚本和导出的镜像通过 U 盘、堡垒机SSH等介质拷贝至master主节点。

后续无特殊说明，只在master节点操作。

4.3 安装k8s依赖包

所有节点执行,上传k8s-init-KylinV10.tar.gz解压后执行install.sh

4.4 修改配置文件

主要修改相关节点和harbor信息

vi config-sample.yaml

kind: Cluster
metadata:
name:sample
spec:
hosts:
-{name:master,address:192.168.85.138,internalAddress:192.168.85.138,user:root,password:"123456"}
-{name:node1,address:192.168.85.139,internalAddress:192.168.85.139,user:root,password:"123456"}
roleGroups:
    etcd:
    -master
    control-plane:
    -master
    worker:
    -node1
    registry:
    -node1
controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain:lb.kubesphere.local
    address:""
    port:6443
kubernetes:
    version:v1.31.8
    clusterName:cluster.local
    autoRenewCerts:true
    containerManager:containerd
etcd:
    type:kubekey
network:
    plugin:calico
    kubePodsCIDR:10.233.64.0/18
    kubeServiceCIDR:10.233.0.0/18
    multusCNI:
      enabled:false
registry:
    type:harbor
    registryMirrors:[]
    insecureRegistries:[]
    privateRegistry:"dockerhub.kubekey.local"
    namespaceOverride:"kubesphereio"
    auths:# if docker add by `docker login`, if containerd append to `/etc/containerd/config.toml`
      "dockerhub.kubekey.local":
        username:"admin"
        password:Harbor@123#此处可自定义，kk3.1.8新特性
        skipTLSVerify:true# Allow contacting registries over HTTPS with failed TLS verification.
        plainHTTP:false# Allow contacting registries over HTTP.
        certsPath:"/etc/docker/certs.d/dockerhub.kubekey.local"
addons:[]

4.5 创建镜像仓库

./kk init registry -f config-sample.yaml -a artifact-k8s-1318-ks413.tar.gz

4.6 创建harbor项目

说明：

Harbor 管理员账号：admin，密码：Harbor@123。密码同步使用配置文件中的对应password

harbor 安装文件在 /opt/harbor 目录下，可在该目录下对 harbor 进行运维。

vi create_project_harbor.sh

#!/usr/bin/env bash

url="https://dockerhub.kubekey.local"# 或修改为实际镜像仓库地址
user="admin"
passwd="Harbor@123"

harbor_projects=(
        ks
        kubesphere
        kubesphereio
)

for project in"${harbor_projects[@]}"; do
    echo"creating $project"
    curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json""${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k  # 注意在 curl 命令末尾加上 -k
done

创建 Harbor 项目

chmod +x create_project_harbor.sh

./create_project_harbor.sh

验证

5 安装Kubernetes

执行以下命令创建 Kubernetes 集群：

./kk create cluster -f config-sample.yaml -a artifact-k8s-1318-ks413.tar.gz --with-local-storage

等待大概两分钟左右看到成功消息

验证

6 安装 KubeSphere

helm upgrade --install -n kubesphere-system --create-namespace ks-core ks-core-1.1.5.tgz \
     --set global.imageRegistry=dockerhub.kubekey.local/ks \
     --set extension.imageRegistry=dockerhub.kubekey.local/ks \
     --set ksExtensionRepository.image.tag=v1.1.5 \
     --debug \
     --wait

等待大概30秒左右看到成功消息