CentOS7.3.1611部署k8s1.5.2集群

CentOS7.3.1611部署k8s1.5.2集群

刚得知12小时前更新了最新的k8s1.5.3和1.4.9，安装方法应该类似

参考资料

Kubernetes权威指南（第二版）

http://jevic.blog.51cto.com/2183736/1881455

https://my.oschina.net/u/1791060/blog/830023

http://blog.csdn.net/lic95/article/details/55015284

https://coreos.com/etcd/docs/latest/clustering.html

下列文档简单的系统的测试了k8s 1.5.x系列：包括部署集群、创建POD、域名解析、仪表盘、监控、反向代理、存储、日志，另外双向认证自己建证书不太实用就没有列出。本系列文档环境部署使用二进制程序绿色安装，适用于1.5.2、1.5.3、1.5.4及后续版本，只是记得随时更新github上样例url即可。

k8s集群安装部署

http://jerrymin.blog.51cto.com/3002256/1898243

k8s集群RC、SVC、POD部署

http://jerrymin.blog.51cto.com/3002256/1900260

k8s集群组件kubernetes-dashboard和kube-dns部署

http://jerrymin.blog.51cto.com/3002256/1900508

k8s集群监控组件heapster部署

http://jerrymin.blog.51cto.com/3002256/1904460

k8s集群反向代理负载均衡组件部署

http://jerrymin.blog.51cto.com/3002256/1904463

k8s集群挂载volume之nfs

http://jerrymin.blog.51cto.com/3002256/1906778

k8s集群挂载volume之glusterfs

http://jerrymin.blog.51cto.com/3002256/1907274

k8s集群日志收集ELK架构

http://jerrymin.blog.51cto.com/3002256/1907282

架构

k8s-master  安装etcd，kubernetes-server/client

k8s-node1   安装docker,kubernetes-node/client，flannel

k8s-node2   安装docker,kubernetes-node/client，flannel

一，YUM安装的版本如下

CentOS7.3.1611 Yum安装

kubernetes-1.4.0-0.1.git87d9d8d.el7

会安装kubernets-master,node,client及其相关依赖项

kubernetes-master-1.4.0-0.1.git87d9d8d.el7

会产生三个二进制程序kube-apiserver  kube-controller-manager  kube-scheduler 

kubernetes-node-1.4.0-0.1.git87d9d8d.el7

会安装很多依赖包包括docker-1.12.5-14.el7.centos，会安装kubelet  kube-proxy 

kubernetes-client-1.4.0-0.1.git87d9d8d.el7

会产生一个二进制程序kubectl 

kubernetes-unit-test-1.4.0-0.1.git87d9d8d.el7

会安装很多依赖包包括etcd-3.0.15-1.el7，golang，gcc，glibc，rsync等                 

flannel-0.5.5-2.el7 

会产生一个二进制程序flannel

二，本文选择二进制包版本安装最新版测试

github地址：

etct: https://github.com/coreos/etcd/releases

flannel: https://github.com/coreos/flannel/releases

kubernetes: https://github.com/kubernetes/kubernetes/releases

docker: https://docs.docker.com/engine/installation/linux/centos/

k8s 1.5.2

https://dl.k8s.io/v1.5.2/kubernetes-server-linux-amd64.tar.gz

会产生11个二进制程序hyperkube  kubectl   kubelet  kube-scheduler  kubeadm  kube-controller-manager  kube-discovery  kube-proxy  kube-apiserver kube-dns  kubefed         

https://dl.k8s.io/v1.5.2/kubernetes-client-linux-amd64.tar.gz

会产生两个二进制程序kube-proxy  kubefed

etcd 3.1.10

https://github.com/coreos/etcd/releases/download/v3.1.0/etcd-v3.1.0-linux-amd64.tar.gz

docker 1.13.1

https://get.docker.com/builds/Linux/x86_64/docker-1.13.1.tgz

flannel

https://github.com/coreos/flannel/releases/download/v0.7.0/flannel-v0.7.0-linux-amd64.tar.gz

三，部署环境

1，准备工作

1），系统最小化安装,然后yum update,升级到最新版本CentOS7.3.1611

2），设置hostname及hosts

[root@k8s-master ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.3.20  k8s-master

172.17.3.7   k8s-node1

172.17.3.8   k8s-node2

3），校对时间

[root@k8s-master ~]# ntpdate ntp1.aliyun.com &&hwclock -w

4），关闭selinux及防火墙

[root@k8s-master ~]# sed -i s'/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

[root@k8s-master ~]# systemctl disable firewalld; systemctl stop firewalld

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

5），重启服务器

2，Master节点部署

1），部署etcd服务（目前单点）

[root@k8s-master ~]# tar zxvf etcd-v3.1.0-linux-amd64.tar.gz -C /usr/local/

[root@k8s-master ~]# mv /usr/local/etcd-v3.1.0-linux-amd64/ /usr/local/etcd

[root@k8s-master ~]# ln -s /usr/local/etcd/etcd /usr/local/bin/etcd

[root@k8s-master ~]# ln -s /usr/local/etcd/etcdctl /usr/local/bin/etcdctl

设置systemd服务文件/usr/lib/systemd/system/etcd.service

[Unit]

Description=Eted Server 

After=network.target

[Service]

WorkingDirectory=/data/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

ExecStart=/usr/local/bin/etcd

Type=notify

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

其中WorkingDirector表示etcd数据保存的目录，需要在启动etcd服务之前进行创建

etcd单点默认配置

[root@k8s-master ~]# cat /etc/etcd/etcd.conf 

ETCD_NAME=k8s1

ETCD_DATA_DIR="/data/etcd"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

etcd服务启动

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable etcd.service

[root@k8s-master ~]# systemctl start etcd.service

etcd服务检查

[root@k8s-master ~]# etcdctl cluster-health

member 869f0c691c5458a3 is healthy: got healthy result from http://0.0.0.0:2379

cluster is healthy

[root@k8s-master ~]# etcdctl member list

869f0c691c5458a3: name=k8s1 peerURLs=http://172.17.3.20:2380 clientURLs=http://0.0.0.0:2379 isLeader=true

2）部署kube-apiserver服务

安装kube-apiserver

[root@k8s-master ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz  -C /usr/local/

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-apiserver /usr/local/bin/kube-apiserver

其他服务顺便做下软链接

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/hyperkube /usr/local/bin/hyperkube

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubeadm /usr/local/bin/kubeadm

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-controller-manager /usr/local/bin/kube-controller-manager

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubectl  /usr/local/bin/kubectl

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-discovery /usr/local/bin/kube-discovery

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-dns  /usr/local/bin/kube-dns

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubefed /usr/local/bin/kubefed

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubelet /usr/local/bin/kubelet      

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-proxy /usr/local/bin/kube-proxy

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-scheduler /usr/local/bin/kube-scheduler

配置kubernetes system config

[root@k8s-master ~]# cat /etc/kubernetes/config 

KUBE_LOGTOSTDERR="--logtostderr=false"

KUBE_LOG_DIR="--log-dir=/data/logs/kubernetes"

KUBE_LOG_LEVEL="--v=2"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://172.17.3.20:8080"

设置systemd服务文件/usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

After=etcd.service

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/apiserver

ExecStart=/usr/local/bin/kube-apiserver \

   $KUBE_LOGTOSTDERR \

   $KUBE_LOG_LEVEL \

   $KUBE_ETCD_SERVERS \

   $KUBE_API_ADDRESS \

   $KUBE_API_PORT \

   $KUBELET_PORT \

   $KUBE_ALLOW_PRIV \

   $KUBE_SERVICE_ADDRESSES \

   $KUBE_ADMISSION_CONTROL \

   $KUBE_API_ARGS

Restart=on-failure

Type=notify

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置kuber-apiserver启动参数

[root@k8s-master ~]# cat /etc/kubernetes/apiserver

KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

KUBE_API_ARGS=" "

启动kube-api-servers服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-apiserver.service

Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.

[root@k8s-master ~]# systemctl start kube-apiserver.service

验证服务

http://172.17.3.20:8080/

3）部署kube-controller-manager服务

设置systemd服务文件/usr/lib/systemd/system/kube-controller-manager.service 

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=kube-apiserver.service

Requires=kube-apiserver.service

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/controller-manager

ExecStart=/usr/local/bin/kube-controller-manager \

   $KUBE_LOGTOSTDERR \

   $KUBE_LOG_LEVEL \

   $KUBE_LOG_DIR \

   $KUBE_MASTER \

   $KUBE_CONTROLLER_MANAGER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置kube-controller-manager启动参数

[root@k8s-master ~]# cat /etc/kubernetes/controller-manager 

KUBE_CONTROLLER_MANAGER_ARGS=""

启动kube-controller-manager服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-controller-manager

Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

[root@k8s-master ~]# systemctl start kube-controller-manager

4）部署kube-scheduler服务

设置systemd服务文件/usr/lib/systemd/system/kube-scheduler.service 

[Unit]

Description=Kubernetes Scheduler Plugin

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=kube-apiserver.service

Requires=kube-apiserver.service

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/scheduler

ExecStart=/usr/local/bin/kube-scheduler \

            $KUBE_LOGTOSTDERR \

            $KUBE_LOG_LEVEL \

            $KUBE_LOG_DIR \

            $KUBE_MASTER \

            $KUBE_SCHEDULER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置kube-schedulerr启动参数

[root@k8s-master ~]# cat /etc/kubernetes/schedulerr

KUBE_SCHEDULER_ARGS=""

启动kube-scheduler服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-scheduler

Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.

[root@k8s-master ~]# systemctl start kube-scheduler

2，Node节点部署

1）安装docker(或者yum instll docker)

[root@k8s-node1 ~]# tar zxvf docker-1.13.1.tgz -C /usr/local

这里默认docker安装并启动，方便后面测试

[root@k8s-node1 ~]# systemctl start docker.service

2）安装kubernetes客户端

安装kubelet，kube-proxy

[root@k8s-master ~]# tar zxvf kubernetes-client-linux-amd64.tar.gz  -C /usr/local/

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubectl /usr/local/bin/kubectl

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubefed /usr/local/bin/kubefed

kube-proxy包默认client没有可以从server拷贝过来

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kube-proxy /usr/local/bin/kube-proxy

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubelet /usr/local/bin/kubelet  

3）部署kubelet服务

配置kubernetes system config

[root@k8s-node1 ~]# cat /etc/kubernetes/config 

KUBE_LOGTOSTDERR="--logtostderr=false"

KUBE_LOG_DIR="--log-dir=/data/logs/kubernetes"

KUBE_LOG_LEVEL="--v=2"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://172.17.3.20:8080"

设置systemd服务文件/usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=docker.service

Requires=docker.service

[Service]

WorkingDirectory=/data/kubelet

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/kubelet

ExecStart=/usr/local/bin/kubelet \

   $KUBE_LOGTOSTDERR \

   $KUBE_LOG_LEVEL \

   $KUBE_LOG_DIR \

   $KUBELET_API_SERVER \

   $KUBELET_ADDRESS \

   $KUBELET_PORT \

   $KUBELET_HOSTNAME \

   $KUBE_ALLOW_PRIV \

   $KUBELET_POD_INFRA_CONTAINER \

   $KUBELET_ARGS

Restart=on-failure

[Install]

WantedBy=multi-user.target

配置kubelet启动参数

[root@k8s-node1 ~]# cat /etc/kubernetes/kubelet

KUBELET_ADDRESS="--address=0.0.0.0"

KUBELET_PORT="--port=10250"

KUBELET_HOSTNAME="--hostname-override=k8s-node1"

KUBELET_API_SERVER="--api-servers=http://172.17.3.20:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS=""

启动kubelet服务

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable kubelet.service

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

[root@k8s-node1 ~]# systemctl start kubelet.service

4），部署kube-proxy服务

设置systemd服务文件/usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Kube-Proxy Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/proxy

ExecStart=/usr/local/bin/kube-proxy \

   $KUBE_LOGTOSTDERR \

   $KUBE_LOG_LEVEL \

   $KUBE_LOG_DIR \

   $KUBE_MASTER \

   $KUBE_PROXY_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置kubelet启动参数

[root@k8s-node1 ~]# cat /etc/kubernetes/proxy

KUBE_PROXY_ARGS=""

启动kubelet服务

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable kube-proxy.service

Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

[root@k8s-node1 ~]# systemctl start kube-proxy.service

验证节点是否启动

[root@k8s-node1 ~]# kubectl get nodes

NAME        STATUS    AGE

k8s-node1   Ready     9m

3，配置网络

1），配置etcd

[root@k8s-master ~]# etcdctl set /k8s/network/config '{ "Network": "10.1.0.0/16" }'

{ "Network": "10.1.0.0/16" }

[root@k8s-master ~]# etcdctl get /k8s/network/config   

{ "Network": "10.1.0.0/16" }

2)，安装flannel

[root@k8s-node1 ~]# tar zxvf flannel-v0.7.0-linux-amd64.tar.gz -C /usr/local/flannel

[root@k8s-node1 ~]# ln -s /usr/local/flannel/flannel /usr/local/bin/flanneld 

[root@k8s-node1 ~]# ln -s /usr/local/flannel/mk-docker-opts.sh   /usr/local/bin/mk-docker-opts.sh

3），配置flannel（配置比较麻烦，start脚本和启动脚本参考yum安装时生成的配置）

设置systemd服务文件/usr/lib/systemd/system/flanneld.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network.target

After=network-online.target

Wants=network-online.target

After=etcd.service

Before=docker.service

[Service]

Type=notify

EnvironmentFile=/etc/sysconfig/flanneld

EnvironmentFile=-/etc/sysconfig/docker-network

ExecStart=/usr/local/bin/flanneld-start $FLANNEL_OPTIONS

ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

Restart=on-failure

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

其中flanneld-start为

[root@k8s-node1 ~]# cat /usr/local/bin/flanneld-start

#!/bin/sh

exec /usr/local/bin/flanneld \

-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS:-${FLANNEL_ETCD}} \

-etcd-prefix=${FLANNEL_ETCD_PREFIX:-${FLANNEL_ETCD_KEY}} \

"$@"

编辑flannel，设置etcd相关信息

[root@k8s-node1 ~]# cat /etc/sysconfig/flanneld 

FLANNEL_ETCD_ENDPOINTS="http://172.17.3.20:2379"

FLANNEL_ETCD_PREFIX="/k8s/network"

4），启动flannel

注意启动flannel前要关闭docker这样flannel才会覆盖docker0网桥

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable flanneld.service

[root@k8s-node1 ~]# systemctl stop docker.service

[root@k8s-node1 ~]# systemctl start flanneld.service

flanneld服务启动后就会根据etcd里面配置划分子网了，划分子网是给docker使用的，docker想使用还得折腾一翻，其实就是想办法把几个重要变量传过去，使docker启动时能够使用

注意启动docker前要使某些变量生效，需要source  /run/flannel/docker  source /run/flannel/subnet.env

[root@k8s-node1 ~]# cat /run/flannel/docker 

DOCKER_OPT_BIP="--bip=10.1.89.1/24"

DOCKER_OPT_IPMASQ="--ip-masq=true"

DOCKER_OPT_MTU="--mtu=1472"

DOCKER_NETWORK_OPTIONS=" --bip=10.1.89.1/24 --ip-masq=true --mtu=1472"

[root@k8s-node1 bin]# cat /run/flannel/docker 

DOCKER_OPT_BIP="--bip=10.1.89.1/24"

DOCKER_OPT_IPMASQ="--ip-masq=true"

DOCKER_OPT_MTU="--mtu=1472"

DOCKER_NETWORK_OPTIONS=" --bip=10.1.89.1/24 --ip-masq=true --mtu=1472"

[root@k8s-node1 ~]# cat /run/flannel/subnet.env 

FLANNEL_NETWORK=10.1.0.0/16

FLANNEL_SUBNET=10.1.89.1/24

FLANNEL_MTU=1472

FLANNEL_IPMASQ=false

确保docker启动时带有--bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} 这样docker0才会成为flannel0的子网这个启动参数是通过ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker 实现的

-kSet the combined options key to this value (default DOCKER_OPTS=)

-dPath to Docker env file to write to. Defaults to /run/docker_opts.env

最后才启动docker

[root@k8s-node1 ~]# systemctl start docker.service

5），最后确认效果

完成后确认网络接口docker0的IP地址属于flannel0的子网

网络启动后node1和node2节点会添加很多路由条目，并且会自动开启防火墙虽然之前我们关闭了，里面有很多策略目的是node直接的docker0网络可以互通，这样各个node间通过物理网卡--flannel0--docker0和容器通信

[root@k8s-node1 ~]# ip addr

6: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500

    link/none 

    inet 10.1.89.0/16 scope global flannel0

       valid_lft forever preferred_lft forever

7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 

    link/ether 02:42:f1:e4:7c:a3 brd ff:ff:ff:ff:ff:ff

    inet 10.1.89.1/24 scope global docker0

       valid_lft forever preferred_lft forever

[root@k8s-node2 ~]# ip addr

6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 

    link/ether 02:42:33:a8:38:21 brd ff:ff:ff:ff:ff:ff

    inet 10.1.8.1/24 scope global docker0

       valid_lft forever preferred_lft forever

7: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500

    link/none 

    inet 10.1.8.0/16 scope global flannel0

       valid_lft forever preferred_lft forever

node1上 ping node2的docker0能通就行

[root@k8s-node1 ~]# ping 10.1.8.1

PING 10.1.8.1 (10.1.8.1) 56(84) bytes of data.

64 bytes from 10.1.8.1: icmp_seq=1 ttl=62 time=0.498 ms

64 bytes from 10.1.8.1: icmp_seq=2 ttl=62 time=0.463 ms