本文实现的是基于有主机和网络(无PXE、无dhcp、无dns)的情况下,安装OCP4.6.3。
对安装OCP4.X都有很大的参考价值。
用途 | CPU | 内存 | 系统盘 | 数据盘 | 备注 |
---|---|---|---|---|---|
HA | 2 | 4 | 60 | 0 | 负载均衡。我们这里使用haproxy。生产环境可考虑2台高可用安装 |
dns | 2 | 4 | 60 | 0 | 域名服务,我们这里使用。生产环境可考虑2台高可用安装 |
registry | 4 | 8 | 60 | 300 | 镜像仓库,我们这里使用harbor。生产环境可考虑2台高可用安装。 |
OCP4.6-init | 2 | 4 | 60 | 0 | 安装机,在此机器启动http server |
OCP4.6-bootstrap | 2 | 4 | 60 | 0 | |
OCP4.6-master | 32 | 64 | 60 | 300 | |
OCP4.6-master | 32 | 64 | 60 | 300 | |
OCP4.6-master | 32 | 64 | 60 | 300 | |
OCP4.6-worker | 32 | 64 | 60 | 300 | |
OCP4.6-worker<n> | 32 | 64 | 60 | 300 |
说明: 1、以上CPU/内存/磁盘配置为参考值,请根据实际情况分配(master内存不小于8G,worker不小于16G) 2、HA(haproxy)非必须,如果环境中有LB组件(例如:F5)可使用,直接使用即可 3、dns非必须,如果环境中域名服务,直接使用即可 4、registry非必须,如果环境中有镜像仓库,直接使用即可 5、harbor需要ssl证书,建议提前准备 6、以上主机都可以初始化安装centos7(记得关闭防火墙) systemctl disable firewalld systemctl stop firewalld
域名 | 解析IP |
---|---|
harbor.xxx.com | harbor的VIP |
api.xxx.com | HA VIP |
api-int.xxx.com | HA VIP |
*.apps.xxx.com | HA VIP |
init.xxx.com | init虚机IP |
bootstrap.xxx.com | bootstrap虚机IP |
master1.xxx.com | master1虚机IP |
master2.xxx.com | master2虚机IP |
master3.xxx.com | master3虚机IP |
worker1.xxx.com | worker1虚机IP |
workerN.xxx.com | workerN虚机IP |
SRV记录_etcd-server-ssl._tcp.xxx.com,etcd-0.xxx.com,2380,0,100 | master1虚机IP |
SRV记录_etcd-server-ssl._tcp.xxx.com,etcd-1.xxx.com,2380,0,100 | master2虚机IP |
SRV记录_etcd-server-ssl._tcp.xxx.com,etcd-2.xxx.com,2380,0,100 | master3虚机IP |
如果内网环境没有dns服务,需要安装dns服务,如果有,不用安装dns服务,但是需要申请在已有的dns服务配置本次安装需要的域名
docker pull jpillora/dnsmasq
docker run \
--name dnsmasq \
-d \
-p 53:53/udp \
-p 8080:8080 \
-v /opt/dnsmasq.conf:/etc/dnsmasq.conf \
--log-opt "max-size=100m" \
-e "HTTP_USER=admin" \
-e "HTTP_PASS=123456" \
--restart always \
jpillora/dnsmasq
dnsmasq.conf设置参考:
cache-size=10000
dns-forward-max=10000000
address=/harbor.xxx.com/<harbor的ip>
address=/api.xxx.com/<HA的ip>
address=/api-int.xxx.com/<HA的ip>
address=/apps.xxx.com/<HA的ip>
address=/bootstrap.xxx.com/<bootstrap的ip>
address=/master1.xxx.com/<master1的ip>
address=/master2.xxx.com/<master2的ip>
address=/master3.xxx.com/<master3的ip>
address=/worker1.xxx.com/<wworker1的ip>
address=/worker2.xxx.com/<wworker2的ip>
address=/worker3.xxx.com/<wworker3的ip>
address=/etcd-0.xxx.com/<master1的ip>
address=/etcd-1.xxx.com/<master2的ip>
address=/etcd-2.xxx.com/<master3的ip>
srv-host=_etcd-server-ssl._tcp.xxx.com,etcd-0.xxx.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.xxx.com,etcd-1.xxx.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.xxx.com,etcd-2.xxx.com,2380,0,100
如果内网没有F5服务(或haproxy、或nginx、或其他SLB),则安装haproxy
yum -y install haporxy
haproxy.cfg配置样例
#----------------------------------------------------------------
# Global settings
#--------------------------------------------------------------
global
log 127.0.0.1 local0
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen admin_stats
stats enable
bind *:8080
mode http
option httplog
log global
maxconn 10
stats refresh 30s
stats uri /
stats realm haproxy
stats auth admin:admin
stats hide-version
stats admin if TRUE
frontend openshift-api-server6443
bind :6443
default_backend openshift-api-server6443
mode tcp
#mode http
option tcplog
backend openshift-api-server6443
balance source
mode tcp
server bootstrap <bootstrap的ip>:6443 check
server master1 <master1的ip>:6443 check
server master2 <master2的ip>:6443 check
server master3 <master3的ip>:6443 check
frontend machine-config-server22623
bind :22623
default_backend machine-config-server22623
mode tcp
#mode http
option tcplog
backend machine-config-server22623
# balance source
mode tcp
server bootstrap <bootstrap的ip>:22623 check
server master1 <master1的ip>:22623 check
server master2 <master2的ip>:22623 check
server master3 <master3的ip>:22623 check
#---------------------------------------------------------------------
# configure 80 and 443 to point to the worker nodes.
# add master nodes if mastersSchedulable is true.
frontend ingress-http80
bind :80
default_backend ingress-http80
mode tcp
#mode http
option tcplog
backend ingress-http80
balance source
mode tcp
server master1 <master1的ip>:80 check
server master2 <master2的ip>:80 check
server master3 <master3的ip>:80 check
server worker1 <wworker1的ip>:80 check
server workerN <wworkerN的ip>:80 check
frontend ingress-https443
bind :443
default_backend ingress-https443
mode tcp
#mode http
option tcplog
backend ingress-https443
balance source
mode tcp
server master1 <master1的ip>:443 check
server master2 <master2的ip>:443 check
server master3 <master3的ip>:443 check
server worker1 <wworker1的ip>:443 check
server workerN <wworkerN的ip>:443 check
说明:haproxy主机打开haproxy日志设置方法参考 https://www.cnblogs.com/liufarui/p/11078172.html
说明:如果待安装环境内网没有容器镜像仓库,需要安装镜像仓库
这里以安装harbor为例
高可用环境安装具体参考官网https://github.com/goharbor/harbor
下面示意非高可用环境安装
step1、从官网下载offline安装包 例如:https://github.com/goharbor/harbor/releases/download/v2.1.3/harbor-offline-installer-v2.1.3.tgz step2、安装docker wget -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y epel-release docker-ce-18.09.9-3.el7 docker-compose systemctl start docker systemctl enable docker step3、解压安装包,编辑harbor.yml 重点注意如下图红色框的参数
step4、安装 执行:bash install.sh --with-clair --with-trivy --with-chartmuseum
说明:如果待安装环境内网没有有已导入的ocp4镜像,需要导入ocp4镜像
我们这里直接在harbor主机操作导入ocp4镜像为例
step1、检查是否可以从互联网pull镜像 例如:执行docker pull jenkins。如果失败,您需要: a、有一台访问公网的正向代理 b、或者移步到可以访问公网的主机上执行导入ocp4镜像 下面以“有一台访问公网的正向代理”条件做配置说明 mkdir -p /etc/systemd/system/docker.service.d vi /etc/systemd/system/docker.service.d/http-proxy.conf,写入如下内容: Service Environment="HTTP_PROXY=http://正向代理IP:正向代理端口" "HTTPS_PROXY=http://正向代理IP:正向代理端口" "NO_PROXY=localhost,127.0.0.1,刚安装的harbor的域名" 然后重启docker systemctl daemon-reload systemctl restart docker 再次执行step1、检查是否可以从互联网pull镜像,直至成功pull为止 step2、web登录harbor,建立ocp4项目
step3、web登录https://cloud.redhat.com/openshift/install 下载 Pull Secret
step4、docker login harbor 将step3下载的文件内容写入 ~/.docker/config.json,然后执行docker login harbor.xxx.com
(harbor.xxx.com为刚安装的harbor的地址) step5、导入镜像 我们本次导入OCP4.6.3,如果导入其他版本,只需改变BUILDNUMBER变量。 导入过程依次执行如下: export BUILDNUMBER=4.6.3 export LOCAL_REG='harbor.xxx.com' export OCP_RELEASE=${BUILDNUMBER} export LOCAL_REPO='ocp4/openshift4' export UPSTREAM_REPO='openshift-release-dev' export LOCAL_SECRET_JSON="~/.docker/config.json" export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} export RELEASE_NAME="ocp-release" wget -O release.txt https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/release.txt wget -O openshift-client-linux-${BUILDNUMBER}.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-client-linux-${BUILDNUMBER}.tar.gz wget -O openshift-install-linux-${BUILDNUMBER}.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-install-linux-${BUILDNUMBER}.tar.gz tar -xzf openshift-client-linux-${BUILDNUMBER}.tar.gz -C /usr/local/sbin/ tar -xzf openshift-install-linux-${BUILDNUMBER}.tar.gz -C /usr/local/sbin/ ./oc adm release mirror -a ${LOCAL_SECRET_JSON} \ --from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-x86_64 \ --to-release-image=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} \ --to=${LOCAL_REG}/${LOCAL_REPO} 等待片刻,直至导入成功。成功后,harbor显示ocp4有121个镜像
wget -o /etc/yum.repos.d/docker-ce.repo
https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-18.09.9-3.el7
systemctl start docker
ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/new_rsa
拷贝2.2的~/.docker/config.json
apiVersion: v1
baseDomain: xxx.com #需要修改为实际域名
compute:
- hyperthreading: Enabled
name: worker
replicas: 3
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: edge-ocp1#需要根据实际修改为[ocpX]
networking:
clusterNetworks:
- cidr: 10.254.0.0/16
hostPrefix: 24
networkType: OpenShiftSDN
serviceNetwork:
- 10.255.0.0/16
platform:
none: {}
pullSecret: '{"auths":{"harbor.xxx.com":{"auth":"xxx"}}}' #harbor密钥
sshKey: 'ssh-rsa xxx' #ssh密钥
imageContentSources: #修改 harbor.xxx.com 域名即可;
- mirrors:
- harbor.xxx.com/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
- harbor.xxx.com/ocp4/openshift4
source: quay.io/ocp4/openshift4
说明: 1、install-config.yaml文件不要写中文 2、install-config.yaml文件定稿后,请备份
export VERSION=4.6.3
export OCP_RELEASE=$VERSION-x86_64
export LOCAL_REGISTRY=harbor.xxx.com
export LOCAL_REPOSITORY=ocp4/openshift4
export PRODUCT_REPO=openshift-release-dev
export RELEASE_NAME=ocp-release
./oc adm -a pull-push_secret.json release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
#验证
./openshift-install version
说明: oc命令来自于wget -O openshift-client-linux-4.6.3.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${BUILDNUMBER}/openshift-client-linux-4.6.3.tar.gz 执行./oc adm需要连接公网,如果使用正向代理访问公网,需要参考如下设置
export https_proxy=http://正向代理IP:正向代理端口
export http_proxy=http://正向代理IP:正向代理端口
export no_proxy=localhost,127.0.0.1,harbor.xxx.com
验证:执行 ./openshift-install version
返回类似如下说明正确
./openshift-install 4.6.3
built from commit <commit_id>
release image harbor.xxx.com/ocp4/openshift4@sha256:<sha256id>
./openshift-install create manifests --dir $clusterconfig
说明:如果规划master/worker有独立挂数据盘,需要执行此步骤 在$clusterconfig/openshift 目录下新建 98-var-partition-worker.yaml 在$clusterconfig/openshift 目录下新建 98-var-partition-master.yaml 内容如下:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker #如果是 master 这里改为 master
name: 98-var-partition
spec:
config:
ignition:
version: 3.1.0
storage:
disks:
- device: /dev/vdb
partitions:
- size: 163840
start: 0
label: var
filesystems:
- path: /var
device: /dev/disk/by-partlabel/var
format: xfs
systemd:
units:
- name: var.mount
enabled: true
contents: |
[Unit]
Before=local-fs.target
[Mount]
Where=/var
What=/dev/disk/by-partlabel/var
[Install]
WantedBy=local-fs.target
说明:执行如下步骤成功后会删除前面编辑的yml文件,请做好备份
./openshift-install create ignition-configs --dir $clusterconfig
验证 #现在配置文件就好了。在$clusterconfig 目录下启动一个 HTTP 服务: nohup python -m SimpleHTTPServer 80 & #验证服务器是否正常 curl localhost/master.ign #返回结果就 OK
从https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/下载,请下载与ocp4配套版本的img(例如:我的是4.6.3)
下载如下文件 rhcos-4.6.3-x86_64-live-initramfs.x86_64.img rhcos-4.6.3-x86_64-live-kernel-x86_64 mg rhcos-4.6.3-x86_64-live-rootfs.x86_64.img rhcos-4.6.3-x86_64-live.x86_64.iso rhcos-4.6.3-x86_64-metal.x86_64.raw.gz
init文件格式
ip=PARAM1::PARAM2:PARAM3:PARAM4:ens3:none nameserver=PARAM5 coreos.inst.install_dev=vda coreos.inst.image_url=PARAM6 coreos.inst.ignition_url=PARAM7 coreos.inst=yes rd.neednet=1 coreos.inst.platform_id=qemu coreos.inst.insecure
各参数解释
PARAM1:待安装机器的IP
PARAM2:待安装机器IP的网关
PARAM3:待安装机器IP的掩码
PARAM4:待安装机器的hostname(申请的对应的域名)
PARAM5:dns 服务器IP
PARAM6:coreos 镜像地址,http://<init机器IP>/rhcos-4.6.3-x86_64-metal.x86_64.raw.gz
PARAM7:不同类型主机 ign 文件地址;例如: http://<init机器IP>/worker.ign
说明:每安装一台机器前,都要编辑好此init文件后,再安装
最后,init目录结构如下,并验证可以通过http://<init机器IP>访问
.
|-- auth
| |-- kubeadmin-password
| `-- kubeconfig
|-- bootstrap.ign
|-- init
|-- install-config.yaml-bak
|-- master.ign
|-- metadata.json
|-- oc
|-- openshift-install
|-- rhcos-4.6.3-x86_64-live-initramfs.x86_64.img
|-- rhcos-4.6.3-x86_64-live-kernel-x86_64
|-- rhcos-4.6.3-x86_64-live-rootfs.x86_64.img
|-- rhcos-4.6.3-x86_64-metal.x86_64.raw.g
|-- worker.json
所有待安装的机器都预先安装centos7,并且内置了脚本
#!/bin/bash ip a add 172.16.105.80/24 dev ens18 cd /boot;curl -O http://<init主机IP>/rhcos-4.6.3-x86_64-live-initramfs.x86_64.img cd /boot;curl -O http://<init主机IP>/rhcos-4.6.3-x86_64-live-kernel-x86_64 init=
curl -s http://<init主机IP>/init
cat >> /etc/grub.d/40_custom <<EOF menuentry 'RHEL CoreOS (Live)' --class fedora --class gnu-linux --class gnu --class os { linux /rhcos-4.6.3-x86_64-live-kernel-x86_64 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal coreos.live.rootfs_url=http://<init主机IP>/rhcos-4.6.3-x86_64-live-rootfs.x86_64.img $init initrd /rhcos-4.6.3-x86_64-live-initramfs.x86_64.img } EOF grub2-set-default 'RHEL CoreOS (Live)' grub2-mkconfig -o /boot/grub2/grub.cfg reboot
注:
1、当http://<init主机IP>/init编辑好后,才能执行此脚本
2、172.16.105.80/24请替换为实际的主机IP
3、ens18请替换为实际的主机网卡名称
编辑init示例参数如下:
ip=bootstrap的ip::gateway:netmask:bootstrap.xxx.com:ens18:none nameserver=dns的ip coreos.inst.install_dev=vda coreos.inst.image_url=http://init的ip/rhcos-4.6.1-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://init的ip
/bootstrap.ign coreos.inst=yes rd.neednet=1 coreos.inst.platform_id=qemu coreos.inst.insecure
ssh -i ~/.ssh/new_rsa core@BootStrap_IP
验证方式:在 bootstrap 主机上 curl -k https://localhost:22623/config/master 如果有返回就标明 OK。
编辑init示例参数如下:
ip=master的ip::gateway:netmask:bootstrap.xxx.com:ens18:none nameserver=dns的ip coreos.inst.install_dev=vda coreos.inst.image_url=http://init的ip/rhcos-4.6.1-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://init的ip
/bootstrap.ign coreos.inst=yes rd.neednet=1 coreos.inst.platform_id=qemu coreos.inst.insecure
验证方式:在 boostrap 主机上使用之前在工作机:$HOME/clusterconfig/auth 目录下的 kubeconfig 配置文件,使用 oc get nodes 如果能看到正确的 node 进来即为成功。
等待三个 master 成功后,记得将 haproxy 主机上的 bootstrap 主机的配置去掉。(6443 端口和 22623 端口,bootstrap 只用来启动 master)
编辑init示例参数如下:
ip=worker的ip::gateway:netmask:bootstrap.xxx.com:ens18:none nameserver=dns的ip coreos.inst.install_dev=vda coreos.inst.image_url=http://init的ip/rhcos-4.6.1-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://init的ip
/bootstrap.ign coreos.inst=yes rd.neednet=1 coreos.inst.platform_id=qemu coreos.inst.insecure
验证方式:在 master 主机上使用之前在工作机:$HOME/clusterconfig/auth 目录下的 kubeconfig 配置文件,执行 oc get csr -o name | xargs oc adm certificate approve
然后使用 oc get nodes 如果能看到正确的 node 进来即为成功,过程需要等待几分钟;
安装过程中,主机会自己重启。
在init机 install-config.yaml 文件目录下执行:./openshift-install wait-for bootstrap-complete --log-level debug
cache-size=10000
dns-forward-max=10000000
还是老老实实的设置搞https吧
需要给网卡设置ip(具体见第3章)
ip a add 172.16.105.80/24 dev ens18
有的IaaS平台创建虚机的时候是可以选择虚拟网关是intel 1000还是 RTL 8319的,所以建议统一你的选择。
全选intel 1000
或
全选RTL 8319
忽略它,让它继续,可以自动完成安装
https://bugzilla.redhat.com/show_bug.cgi?id=1895024
SSH into node
touch /var/run/ovs-config-executed
Delete ovs* pod on the node, verify that it restarts and says something like "openvswitch is running in systmed"
systemctl start ovs-configuration (this starts all dependent ovs services as well)
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。