ETCD集群部署
ETCD集群部署
附件
/opt/soft/etcd/etcd-v3.4.4-linux-amd64.tar.gz 下载地址：https://github.com/etcd-io/etcd/releases
服务器
192.168.1.54、192.168.1.65、192.168.1.105
安装
1、解压包(每台机器)
ETCD_VER=v3.4.4
cd /opt/soft/etcd
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz --strip-components=1
rm -f etcd-${ETCD_VER}-linux-amd64.tar.gz
./etcd --version
./etcdctl version
2、创建etcd配置文件(每台机器) /opt/soft/etcd/etcd.conf 下面配置文件中的IP地址分别修改为本机IP地址，ETCD_NAME分别为etcd01、etcd02、etcd03
#[Member]
#ETCD_CORS=""
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.65:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.65:2379,https://127.0.0.1:2379"
#ETCD_WAL_DIR=""
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.65:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.65:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.54:2380,etcd02=https://192.168.1.65:2380,etcd03=https://192.168.1.105:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
3、创建etcd.service配置文件(每台机器) /usr/lib/systemd/system/etcd.service 这个是systemd中etcd的启动配置文件，配置完之后就可以用systemd启停etc服务了
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/opt/soft/etcd/
EnvironmentFile=/opt/soft/etcd/etcd.conf
ExecStart=/opt/soft/etcd/etcd \
--initial-cluster-state=new \
--cert-file=/opt/soft/etcd/ssl/server.pem \
--key-file=/opt/soft/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/soft/etcd/ssl/server.pem \
--peer-key-file=/opt/soft/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/soft/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/soft/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
4、创建TLS证书 找一个有网络的机器生成证书，否则你就手工下载相关文件
mkdir -p /opt/soft/etcd/ssl && cd $_
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssl-certinfo /usr/local/bin/cfssljson
tls.sh 文件内容如下全部内容(先修改其中的IP地址,其中hosts尽可能多加)
# etcd
# cat ca-config.json
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
# cat ca-csr.json
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
# cat server-csr.json
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.1.65",
"192.168.1.54",
"192.168.1.105"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
执行如下命令
sh tls.sh
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
ls *.pem
然后将生成的4个pem文件证书复制到各个机器的/opt/soft/etcd/ssl目录中
5、启动服务
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
6、查看集群状态
集群状态主要是etcdctl endpoint status和etcdctl endpoint health两条命令
cd /opt/soft/etcd/ && ./etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379" \
--cacert=ssl/ca.pem \
--key=ssl/server-key.pem \
--cert=ssl/server.pem \
endpoint health
输出内容is healthy(健康)
https://192.168.1.105:2379 is healthy: successfully committed proposal: took = 29.874089ms
https://192.168.1.65:2379 is healthy: successfully committed proposal: took = 29.799246ms
https://192.168.1.54:2379 is healthy: successfully committed proposal: took = 39.710904ms
测试(正好配置一个flanneld的网络信息) 因为flanneld(目前最新版v0.11.0)只支持V2版本命令，所以请使用下面的第二段V2命令操作
# V2命令(set设值,ls查看)
ETCDCTL_API=2 etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379" \
--ca-file=ssl/ca.pem \
--key-file=ssl/server-key.pem \
--cert-file=ssl/server.pem \
set /flannel/network/config '{"Network":"10.244.0.0/16", "SubnetMin": "10.244.1.0", "SubnetMax": "10.244.254.0", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
ETCDCTL_API=2 etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379" \
--ca-file=ssl/ca.pem \
--key-file=ssl/server-key.pem \
--cert-file=ssl/server.pem \
ls /flannel/network
# 输出如下内容表示正常
/flannel/network/config
/flannel/network/subnets
etcd 3.4注意事项
ETCD3.4版本ETCDCTL_API=3 etcdctl 和 etcd --enable-v2=false 成为了默认配置，如要使用v2版本，执行etcdctl时候需要设置ETCDCTL_API环境变量，例如：ETCDCTL_API=2 etcdctl ETCD3.4版本会自动读取环境变量的参数，所以EnvironmentFile文件中有的参数，不需要再次在ExecStart启动参数中添加，二选一，如同时配置，会触发以下类似报错“etcd: conflicting environment variable “ETCD_NAME” is shadowed by corresponding command-line flag (either unset environment variable or disable flag)” flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API 注意：flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API，为了兼容flannel，将默认开启v2版本，故需要配置文件/opt/soft/etcd/etcd.conf中设置 ETCD_ENABLE_V2=“true” 在配合K8S使用的时候，写入的Network网段必须是 /16 段地址，必须与K8S的kube-controller-manager、kube-proxy的cluster-cidr参数值一致（原则上应该反过来说）
另外，flanneld 配置文件中要将集群的地址和证书都配置上，如下示例：
FLANNEL_ETCD_ENDPOINTS="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379"
FLANNEL_ETCD_PREFIX="/flannel/network"
FLANNEL_OPTIONS="-etcd-cafile=/opt/soft/etcd/ssl/ca.pem -etcd-keyfile=/opt/soft/etcd/ssl/server-key.pem -etcd-certfile=/opt/soft/etcd/ssl/server.pem"
还有flanneld.service的片段：
ExecStart=/opt/soft/flannel/flanneld -ip-masq -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} $FLANNEL_OPTIONS -etcd-prefix=${FLANNEL_ETCD_PREFIX}
腾讯云开发者
