前言: 之前在国际版环境使用Spinnaker集群进行k8s容器的部署管理,由于Spinnaker由Netflix开源,在集群安装过程中需要访问外国网站来安装一些包。本篇文章将简单记录下在国内如何快速搭建和配置可用的Spinnaker集群环境。并且在生产环境使用
minio
作为持久化层,使用自定义域名,同时对接jenkine来统一对业务代码进行持续构建和管理。
前提条件
1. 网络环境配置
# 访问国外网站(准备一个访问外国网站代理)
$ export http_proxy=http://172.29.202.140:8118
$ export https_proxy=http://172.29.202.140:8118
$ curl www.google.com -I
HTTP/1.1 200 OK
Date: Tue, 27 Aug 2019 08:21:35 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
......
2. minio持久化
# minio 持久化层(真生生产环境需要高可用)
$ docker run -itd -p 9000:9000 -v /opt/data/minio:/data/ --name minio1 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" --restart=on-failure:3 minio/minio server /data
0719d218a9bcd83c730b88234bb22f75b37fd993fb3c149cf70638ea6c811009
$ docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0719d218a9bc minio/minio "/usr/bin/docker-ent…" 4 seconds ago Up 3 seconds 0.0.0.0:9000->9000/tcp minio1
# 查看minio 相关信息
$ docker logs minio1
Endpoint: http://10.0.0.2:9000 http://127.0.0.1:9000
AccessKey: AKIAIOSFODNN7EXAMPLE
SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Browser Access:
http://10.0.0.2:9000 http://127.0.0.1:9000
Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
$ mc config host add myminio http://10.0.0.2:9000 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
3. Jenkins(可选)
Spinnaker主要用来做CD,也就是持续部署,同时也包含了一些应用管理以及部署流程上的相关特性。因此,在标准的CI/CD流水线中其实是可以融合在一起的,所以Jenkins可有可无,如果有的话,可以直接从Spinnaker中进行任务触发然后自动化部署。
注意:Jenkins使用基本认证是可以直接接入Spinnaker集群的,如果接了LDAP认证的可能会有些问题(无法获取到job信息)
0.整体步骤
hal deploy apply
进行部署** 1.安装hal工具**
注意:Spinnaker部署环境必须是Ubuntu或Debian,当然官方也提供了Docker容器部署方式。推荐先使用Ubuntu机器进行安装。
# 进入ubuntu安装hal环境
$ curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/debian/InstallHalyard.sh
$ apt-get update && apt-get install curl vim -y
$ export http_proxy=http://172.29.202.140:8118
$ export https_proxy=http://172.29.202.140:8118
$ curl www.facebook.com -I
HTTP/1.1 302 Found
Location: https://www.facebook.com/
Content-Type: text/html; charset="utf-8"
X-FB-Debug: Bw0gsFvK4VhuMpkmeG3Bc+myLg0Mgz+EkLsMmuq0NVLIvEjiAqckf/N2cjIFRYs4lOWzx3J0P8nvJmtSaZGjCg==
Date: Tue, 27 Aug 2019 08:34:31 GMT
Connection: keep-alive
Content-Length: 0
Proxy-Connection: keep-alive
# 创建spinnaker用户以及依赖环境
$ chmod a+x InstallHalyard.sh && useradd spinnaker
$ apt-get install software-properties-common -y
# 安装hal(1.22.0-20190718020510 推荐指定版本,默认会安装最新版本)
$ bash InstallHalyard.sh --version 1.22.0-20190718020510
Please supply a non-root user to run Halyard as: spinnaker
Halyard version will be stable
Halyard will be downloaded from gs://spinnaker-artifacts/halyard
Halyard config will come from bucket gs://halconfig
Halconfig will be stored at /home/spinnaker/.hal/config
Uninstall script is located at /home/spinnaker/.hal/uninstall.sh
.....
.....
The halyard daemon isn't running yet... starting it manually..
1.22.0-20190718020510
Would you like to configure halyard to use bash auto-completion? [default=Y]:
Where is your bash RC? [default=/home/spinnaker/.bashrc]:
grep: /home/spinnaker/.bashrc: No such file or directory
Bash auto-completion configured.
To use the auto-completion either restart your shell, or run
. /home/spinnaker/.bashrc
# 确认hal版本以及相关情况
$ hal -v
1.22.0-20190718020510
$ . /home/spinnaker/.bashrc
2. 配置hal的代理
注意:为Halyard设置代理,因为每次更新需要去墙外直接获取数据.修改DEFAULT_JVM_OPTS参数(export http_proxy会直接影响整个终端环境)
# 修改/opt/halyard/bin/halyard文件(最后一定要给localhost和example.com设置不代理)
# example.com即为最终想要访问Spinnaker集群的域名
DEFAULT_JVM_OPTS='"-Djava.security.egd=file:/dev/./urandom" "-Dspring.config.additional-location=/opt/spinnaker/config/" "-Dhttp.proxyHost=172.29.202.140" "-Dhttp.proxyPort=8118" "-Dhttps.proxyHost=172.29.202.140" "-Dhttps.proxyPort=8118" "-Dhttp.nonProxyHosts=\"localhost|*.example.com\""'
# 重启hal(shutdown之后任意命令启动)
$ hal shutdown
# 查看当前hal版本支持的Spinnaker版本(能正常获取到Spinnaker版本就是没有问题的)
$ hal version list
+ Get current deployment
Success
+ Get Spinnaker version
Success
+ Get released versions
Success
+ You are on version "", and the following are available:
- 1.13.12 (BirdBox):
Changelog: https://gist.github.com/spinnaker-release/9ee98b0cbed65e334cd498bc31676295
Published: Tue Jul 30 02:18:59 CST 2019
(Requires Halyard >= 1.17)
- 1.14.14 (LoveDeathAndRobots):
Changelog: https://gist.github.com/spinnaker-release/ad1e0eb6b6547b296c9103eb21d9beec
Published: Wed Aug 14 21:37:51 CST 2019
(Requires Halyard >= 1.17)
- 1.15.2 (ExtremelyWickedShockinglyEvilAndVile):
Changelog: https://gist.github.com/spinnaker-release/e72cc8015d544738d07d57a183cb5404
Published: Tue Aug 13 04:48:52 CST 2019
(Requires Halyard >= 1.17)
- 1.15.3 (ExtremelyWickedShockinglyEvilAndVile):
Changelog: https://gist.github.com/spinnaker-release/bed366b82e09498dbb536c098ac11f14
Published: Tue Aug 27 05:09:21 CST 2019
(Requires Halyard >= 1.17)
注意:每个hal的版本都会对应一个可用的Spinnaker的版本列表,其实挺恶心的(镜像版本不一致)
3. 配置kubectl环境
$ apt install snapd -y
$ snap install kubectl --classic
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-21T13:09:06Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
# 配置kubeconfig(将master节点的kubeconfig拷贝到spinnaker部署机器环境中[~/.kube/config或者指定目录])
$ kubectl --kubeconfig kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-1 Ready master 90d v1.14.1
k8s-node-1 Ready <none> 90d v1.14.2
k8s-node-2 Ready <none> 90d v1.14.2
4. 配置spinnaker环境
# 开始k8s支持
$ hal config provider kubernetes enable
# 开启jenkins支持
$ hal config ci jenkins enable
# 获取k8s的上下文
$ kubectl --kubeconfig kubeconfig config current-context
kubernetes-admin@kubernetes
# 配置k8s环境(添加一个账户[即一个k8s集群])
# 指定账户名为spinnaker1 指定版本为v2(v1版本更多是复选框的填写,v2版本更多倾向于Pipeline方式) 指定k8s的kubeconfig配置文件
$ hal config provider kubernetes account add spinnaker1 --provider-version v2 --context kubernetes-admin@kubernetes --kubeconfig-file /root/kubeconfig
# 配置jenkins环境
$ hal config ci jenkins master add my-jenkins-master --address http://10.0.0.1:8080 --username root --password passwd
# 指定账户设置集群部署模式(distributed|local|local-git)
# 因为我们是部署在k8s集群中的,这里选择分布式
$ hal config deploy edit --type distributed --account-name spinnaker1
# 配置持久化存储
# 官方文档中默认的示例是redis存储,使用redis会把pipeline执行信息存放在里面,可能会导致pipeline相关信息不稳定,而且官方也不建议生产环境使用redis
$ hal config --set-current-deployment default
$ hal config storage edit --type s3
$ mkdir /home/spinnaker/.hal/default/profiles
$ echo "spinnaker.s3.versioning: false" > /home/spinnaker/.hal/default/profiles/front50-local.yml
# 配置minio的参数,私钥,以及bucket(bucket需要提前创建)
$ hal config storage s3 edit --endpoint http://172.16.33.101:9000 --access-key-id AKIAIOSFODNN7EXAMPLE --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --bucket spinnaker
# 查看镜像版本(当前hal版本支持的spinnaker版本列表)
$ hal version list
+ Get current deployment
Success
+ Get Spinnaker version
Success
+ Get released versions
Success
+ You are on version "", and the following are available:
- 1.13.12 (BirdBox):
Changelog: https://gist.github.com/spinnaker-release/9ee98b0cbed65e334cd498bc31676295
Published: Tue Jul 30 02:18:59 CST 2019
(Requires Halyard >= 1.17)
- 1.14.14 (LoveDeathAndRobots):
Changelog: https://gist.github.com/spinnaker-release/ad1e0eb6b6547b296c9103eb21d9beec
Published: Wed Aug 14 21:37:51 CST 2019
(Requires Halyard >= 1.17)
- 1.15.2 (ExtremelyWickedShockinglyEvilAndVile):
Changelog: https://gist.github.com/spinnaker-release/e72cc8015d544738d07d57a183cb5404
Published: Tue Aug 13 04:48:52 CST 2019
(Requires Halyard >= 1.17)
- 1.15.3 (ExtremelyWickedShockinglyEvilAndVile):
Changelog: https://gist.github.com/spinnaker-release/bed366b82e09498dbb536c098ac11f14
Published: Tue Aug 27 05:09:21 CST 2019
(Requires Halyard >= 1.17)
# 选择一个版本进行部署
$ hal config version edit --version 1.14.11
# 查看spinnaker版本对应的镜像(后面的version其实就是镜像版本)
$ hal version bom 1.14.11
+ Get BOM for 1.14.11
Success
version: 1.14.11
timestamp: '2019-07-19 16:28:04'
services:
echo:
version: 2.5.2-20190708184501
commit: afcbb51661a36260ed85649f3e5f137371c8b4e0
clouddriver:
version: 4.7.0-20190713034015
commit: 387dde5d27098c83837b9bad3353fff189eee719
deck:
version: 2.9.8-20190718101338
commit: 30dadbd80921caa2ec757a46b56fa916f77cc0b2
fiat:
version: 1.5.2-20190704034018
commit: ff44172d40a14400a8989a33fc35b54a416db3fc
front50:
version: 0.17.0-20190510203645
commit: 0540599582fdefaf5249009c3efdb10d26854956
gate:
version: 1.8.4-20190719034017
commit: 97f6477ffac66205d01949a05e8c6efabc65b7ef
igor:
version: 1.3.0-20190515102735
commit: b3f354f0bd49d0656217998108f522519b30444c
kayenta:
version: 0.9.1-20190628120214
commit: dd8a91d0cf2bd10f2d58564f6e6730030e26f814
orca:
version: 2.7.6-20190719122741
commit: dfafeef02ce273ed4e9f565af7dd754e4b1057f2
rosco:
version: 0.12.0-20190517180000
commit: 59f79299a3fcfb39229cbd0a2e52bf435c340f4a
defaultArtifact: {}
monitoring-third-party:
version: 0.13.0-20190430163248
commit: bf01bf225168919ad13d63b82f93c39061e0b544
monitoring-daemon:
version: 0.13.0-20190430163248
commit: bf01bf225168919ad13d63b82f93c39061e0b544
dependencies:
redis:
version: 2:2.8.4-2
consul:
version: 0.7.5
vault:
version: 0.7.0
artifactSources:
debianRepository: https://dl.bintray.com/spinnaker-releases/debians
dockerRegistry: gcr.io/spinnaker-marketplace
googleImageProject: marketplace-spinnaker-release
gitPrefix: https://github.com/spinnaker
# 整理后的相关镜像地址如下:
$ cat images-1.14.11.txt
gcr.io/spinnaker-marketplace/echo:2.5.2-20190708184501
gcr.io/spinnaker-marketplace/clouddriver:4.7.0-20190713034015
gcr.io/spinnaker-marketplace/deck:2.9.8-20190718101338
gcr.io/spinnaker-marketplace/fiat:1.5.2-20190704034018
gcr.io/spinnaker-marketplace/front50:0.17.0-20190510203645
gcr.io/spinnaker-marketplace/gate:1.8.4-20190719034017
gcr.io/spinnaker-marketplace/igor:1.3.0-20190515102735
gcr.io/spinnaker-marketplace/kayenta:0.9.1-20190628120214
gcr.io/spinnaker-marketplace/orca:2.7.6-20190719122741
gcr.io/spinnaker-marketplace/rosco:0.12.0-20190517180000
gcr.io/spinnaker-marketplace/monitoring-daemon:0.13.0-20190430163248
gcr.io/spinnaker-marketplace/monitoring-third-party:0.13.0-20190430163248
# 可以将相关镜像缓存在企业内部的本地镜像仓库
$ cat retag.py
import os
images = open("images-1.14.11.txt", 'r')
for line in images.readlines():
name = line.strip()
tag = name.replace("gcr.io/spinnaker-marketplace/", "registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-")
os.system('docker pull %s' % name)
os.system('docker tag %s %s' % (name, tag))
os.system('docker push %s' % tag)
# 配置spinnaker的自定义参数(自定义域名和参数)
# 让Spinnaker集群能够识别到自定义的镜像和自定义的域名
$ mkdir /home/spinnaker/.hal/default/service-settings
$/home/spinnaker/.hal/default/service-settings# for i in `ls`;do echo $i;cat $i;done
clouddriver.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-clouddriver:4.7.0-20190713034015
deck.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-deck:2.9.8-20190718101338
overrideBaseUrl: http://deck-spinnaker.example.com/
echo.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-echo:2.5.2-20190708184501
fiat.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-fiat:1.5.2-20190704034018
front50.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-front50:0.17.0-20190510203645
gate.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-gate:1.8.4-20190719034017
overrideBaseUrl: http://gate-spinnaker.example.com/
igor.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-igor:1.3.0-20190515102735
kayenta.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-kayenta:0.9.1-20190628120214
orca.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-orca:2.7.6-20190719122741
redis.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-redis-cluster:v2
rosco.yml
artifactId: registry.cn-hangzhou.aliyuncs.com/spinnaker/spinnaker-rosco:0.12.0-20190517180000
# 修改目录权限
chown -R spinnaker.spinnaker -R /home/spinnaker/
# 查看配置好的部署文件(注意:需要修改时区:Asia/Shanghai)
$ ls /home/spinnaker/.hal/config
# 开始部署(需要确保网络可访问外国网站)
$ hal deploy apply
....
....
+ Deploy spin-redis
Success
+ Deploy spin-clouddriver
Success
+ Deploy spin-front50
Success
+ Deploy spin-orca
Success
+ Deploy spin-deck
Success
+ Deploy spin-echo
Success
+ Deploy spin-gate
Success
+ Deploy spin-igor
Success
+ Deploy spin-rosco
Success
+ Run `hal deploy connect` to connect to Spinnaker.
# 查看部署详情
$ kubectl --kubeconfig kubeconfig get pods -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-5fb4c87f9-wlrgw 1/1 Running 0 5m40s
spin-deck-6cfc48db45-d2xvb 1/1 Running 0 5m41s
spin-echo-c556dfb86-t5whn 1/1 Running 0 5m41s
spin-front50-5b5ff6b78c-x2j79 1/1 Running 5 5m39s
spin-gate-57487bc64f-l9r27 1/1 Running 0 5m41s
spin-igor-55d9cb4594-c42vd 1/1 Running 0 5m40s
spin-orca-67c57f555c-rd2js 1/1 Running 0 5m41s
spin-redis-85959ff9d-9dsw9 1/1 Running 2 55d
spin-rosco-5ff55dd7d8-8v67d 1/1 Running 0 5m39s
# 为spinnaker创建一个ingress
$ cat spinnaker-ingress.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
generation: 1
name: deck-spinnaker.example.com
namespace: spinnaker
spec:
rules:
- host: deck-spinnaker.example.com
http:
paths:
- backend:
serviceName: spin-deck
servicePort: 9000
path: /
status:
loadBalancer: {}
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
generation: 1
name: gate-spinnaker.example.com
namespace: spinnaker
spec:
rules:
- host: gate-spinnaker.example.com
http:
paths:
- backend:
serviceName: spin-gate
servicePort: 8084
path: /
status:
loadBalancer: {}
$ kubectl --kubeconfig kubeconfig create -f spinnaker-ingress.yaml
ingress.extensions/deck-spinnaker.example.com created
ingress.extensions/gate-spinnaker.example.com created
# 测试访问
$ curl -H 'host:deck-spinnaker.example.com' k8s-node-1 -I
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 17727
Content-Type: text/html
Date: Wed, 28 Aug 2019 03:36:02 GMT
Etag: "453f-58df54dd71b00"
Last-Modified: Thu, 18 Jul 2019 14:19:56 GMT
Server: Apache/2.4.38 (Debian)
Vary: Accept-Encoding
5. Spinnaker使用
注意:deck组件是spinnaker的ui,gate组件为spinnaker的api网关,因此这两个组件的域名必须都能解析
在用户本地绑定k8s-node-ip deck-spinnaker.example.com gate-spinnaker.example.com
hosts配置后即可通过http://deck-spinnaker.example.com 进行访问.
6. Spinnaker升级
# 更新spinnaker配置的config文件
hal config version edit --version 1.15.2
# 查看1.15.2相关的镜像
$ hal version bom 1.15.2
+ Get BOM for 1.15.2
Success
version: 1.15.2
timestamp: '2019-08-12 19:38:56'
services:
echo:
version: 2.6.0-20190709142816
commit: 4aae0bc62eb0fb530ea51044473c0683f332bffd
clouddriver:
version: 6.1.0-20190812153835
commit: 96ba2f78c3f819a1701e0cc0c5c9571368bd8002
deck:
version: 2.10.2-20190812115844
commit: 09e4382eda9c757f25033ef63a6ee290881c0e50
fiat:
version: 1.6.1-20190802051323
commit: fced26ecb8aceefb1c5adce7bca7c6d0c776b893
front50:
version: 0.18.0-20190709142816
commit: e6c5f942d0480a57b603f9b714be01782124cd63
gate:
version: 1.10.0-20190809134220
commit: 0743b2c2290ccf03896b5aef16db869d25632c9f
igor:
version: 1.4.0-20190709142816
commit: 324596955e9f4699b1e1356a8feba6608eeb70bc
kayenta:
version: 0.10.1-20190802051323
commit: 6a3c60f9467938b2c2d91a23489fb14d5c180e14
orca:
version: 2.8.2-20190812153835
commit: f70d10fb4b259bb92785e6ed095a6c0b9d77ae09
rosco:
version: 0.13.0-20190709142816
commit: f01311c659c5408fa28a25595e3d2b1c9d891c99
defaultArtifact: {}
monitoring-third-party:
version: 0.14.0-20190702202823
commit: a37ddcef19500350bf48fb2a9ae94f24c26e8e81
monitoring-daemon:
version: 0.14.0-20190702202823
commit: a37ddcef19500350bf48fb2a9ae94f24c26e8e81
dependencies:
redis:
version: 2:2.8.4-2
consul:
version: 0.7.5
vault:
version: 0.7.0
artifactSources:
debianRepository: https://dl.bintray.com/spinnaker-releases/debians
dockerRegistry: gcr.io/spinnaker-marketplace
googleImageProject: marketplace-spinnaker-release
gitPrefix: https://github.com/spinnaker