Harbor-hlem镜像库重新部署后PV数据恢复

三杯水Plus

发布于 2020-07-16 15:44:19

1.3K0

发布于 2020-07-16 15:44:19

文章被收录于专栏：运维

起因

开发反馈habor镜像库登陆不了，初步查看是证书过期了。

解决方案 之前Harbor-helm部署镜像库文档可以回顾链接https://minminmsn.com/middleware/698/ 1.首先新建新证书的secret

[root@elasticsearch01 harbor-helm]#  kubectl  create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt

2.然后修改harbor-helm的value.yaml，把secretName替换下

[root@elasticsearch01 harbor-helm]# head -n 20 values.yaml
expose:
  # Set the way how to expose the service. Set the type as "ingress", 
  # "clusterIP" or "nodePort" and fill the information in the corresponding 
  # section
  type: ingress
  tls:
    # Enable the tls or not. Note: if the type is "ingress" and the tls 
    # is disabled, the port must be included in the command when pull/push
    # images. Refer to https://github.com/goharbor/harbor/issues/5291 
    # for the detail.
    enabled: true
    # Fill the name of secret if you want to use your own TLS certificate
    # and private key. The secret must contain keys named tls.crt and 
    # tls.key that contain the certificate and private key to use for TLS
    # The certificate and private key will be generated automatically if 
    # it is not set
    secretName: "ingress-secret2021"
    # By default, the Notary service will use the same cert and key as
    # described above. Fill the name of secret if you want to use a 
    # separated one. Only needed when the type is "ingress".

3.最后使用helm upgrade更新版本

[root@elasticsearch01 harbor-helm]#  helm upgrade  minminmsn . -f values.yaml

到这个时候应该能解决需求，可是事与愿违，不知道哪儿除了问题，这时登陆Harbor证书问题是解决了，但是项目及库访问不了提示内部错误，看Pod的运行状态也都是Running。最后打算使用helm先delete掉再install，但是这样创建的harbor看起来一切正常，实际上是个初始化环境，是自动生成的新PV并没有原来的数据。此时发现原来的PV还在，下面就开始找PV恢复的方案。

调整PV状态

1.查询此时PV与PVC状态

[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                            STORAGECLASS   REASON   AGE                    9h
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Released   default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Released   default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 harbor-helm]# kubectl get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-minminmsn-harbor-redis-0               Bound    pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
database-data-minminmsn-harbor-database-0   Bound    pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-chartmuseum                Bound    pvc-6c903857-c5f0-11ea-9386-52540089b2b6   50Gi       RWO            rbd            9h
minminmsn-harbor-jobservice                 Bound    pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9h
minminmsn-harbor-registry                   Bound    pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6   500Gi      RWO            rbd            9h

2.修改PV状态先把PV的状态由Released改变成备注：默认创建的PV的回收策略是Delete就是用完就删除，之前特意把RECLAIM POLICY改为了Retain，在线修改PV回收策略可以参考文档https://minminmsn.com/cloud/1091/。否则这里Helm Delete后就会自动删除PV，就没有后来这篇PV数据恢复操作了。在线编辑PV,需要把其中claimRef这段删除，这样状态就可以变成Available了。

  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain

具体如下修改

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: ceph.com/rbd
    rbdProvisionerIdentity: ceph.com/rbd
  creationTimestamp: "2019-05-24T06:33:55Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  resourceVersion: "91736100"
  selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
  uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: minminmsn-harbor-chartmuseum
    namespace: default
    resourceVersion: "91736092"
    uid: b31ec8ca-c649-11ea-9386-52540089b2b6
  persistentVolumeReclaimPolicy: Retain
  rbd:
    image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905
    keyring: /etc/ceph/keyring
    monitors:
    - 10.0.4.8:6789
    pool: rbd-k8s
    secretRef:
      name: ceph-secret
      namespace: default
    user: admin
  storageClassName: rbd
  volumeMode: Filesystem
status:
  phase: Released

3.其他四个PV同样操作

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6

4.查看效果现在看PV的STATUS已经变成了Available，然后CLAIM也变空了，这样就可以在后面绑定使用了

[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Available                               rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d

创建PVC

1.先设置好PVC及PV对应关系

[root@elasticsearch01 yaml]# cat minminmsn.pvc 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-registry
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 2000Gi
  volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-jobservice
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minminmsn-harbor-chartmuseum
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 50Gi
  volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-data-minminmsn-harbor-database-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-minminmsn-harbor-redis-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "rbd"
  resources:
    requests:
      storage: 20Gi
  volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"

2.创建PVC

[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc 
persistentvolumeclaim/minminmsn-harbor-registry created
persistentvolumeclaim/minminmsn-harbor-jobservice created
persistentvolumeclaim/minminmsn-harbor-chartmuseum created
persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created
persistentvolumeclaim/data-minminmsn-harbor-redis-0 created

3.检查PV与PVC

[root@elasticsearch01 yaml]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS   REASON   AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Bound    default/minminmsn-harbor-chartmuseum                rbd                     417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/minminmsn-harbor-jobservice                 rbd                     417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Bound    default/minminmsn-harbor-registry                   rbd                     417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/database-data-minminmsn-harbor-database-0   rbd                     417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/data-minminmsn-harbor-redis-0               rbd                     417d
[root@elasticsearch01 yaml]# kubectl get pvc
NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-rbd-pv-claim                        Bound     ceph-rbd-pv                                20Gi       RWO                           540d
data-minminmsn-harbor-redis-0               Pending   pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
database-data-minminmsn-harbor-database-0   Pending   pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-chartmuseum                Pending   pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-jobservice                 Pending   pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12s
minminmsn-harbor-registry                   Bound     pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            rbd            12s
[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry
Name:          minminmsn-harbor-registry
Namespace:     default
StorageClass:  rbd
Status:        Bound
Volume:        pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...
               pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2000Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    <none>

使用Hlem重新部署Harbor镜像库

1.部署前先删除版本

[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn
helm delete --purge minminmsn
release "minminmsn" deleted

2.修改Harbor-helm的values.yaml中PVC相关值注意existingClaim: ""由空值改成上面生成的PVC名字，注意对应关系，其他不变，具体变更如下

persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete 
  # operation. Leaving it empty will delete PVCs after the chart deleted
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound
      existingClaim: "minminmsn-harbor-registry"
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used(the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 2000Gi
    chartmuseum:
      existingClaim: "minminmsn-harbor-chartmuseum"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 50Gi
    jobservice:
      existingClaim: "minminmsn-harbor-jobservice"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external database is used, the following settings for database will 
    # be ignored
    database:
      existingClaim: "database-data-minminmsn-harbor-database-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    # If external Redis is used, the following settings for Redis will 
    # be ignored
    redis:
      existingClaim: "data-minminmsn-harbor-redis-0"
      storageClass: "rbd"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi

3.重新部署

[root@elasticsearch01 harbor-helm]# helm  install . --name minminmsn
NAME:   minminmsn
LAST DEPLOYED: Wed Jul 15 11:18:13 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)            AGE
minminmsn-harbor-adminserver    ClusterIP  10.254.58.23    <none>       80/TCP             1s
minminmsn-harbor-chartmuseum    ClusterIP  10.254.154.44   <none>       80/TCP             1s
minminmsn-harbor-clair          ClusterIP  10.254.25.107   <none>       6060/TCP           1s
minminmsn-harbor-core           ClusterIP  10.254.56.153   <none>       80/TCP             1s
minminmsn-harbor-database       ClusterIP  10.254.65.18    <none>       5432/TCP           1s
minminmsn-harbor-jobservice     ClusterIP  10.254.81.97    <none>       80/TCP             1s
minminmsn-harbor-notary-server  ClusterIP  10.254.99.90    <none>       4443/TCP           1s
minminmsn-harbor-notary-signer  ClusterIP  10.254.175.105  <none>       7899/TCP           1s
minminmsn-harbor-portal         ClusterIP  10.254.242.113  <none>       80/TCP             1s
minminmsn-harbor-redis          ClusterIP  10.254.127.40   <none>       6379/TCP           1s
minminmsn-harbor-registry       ClusterIP  10.254.158.222  <none>       5000/TCP,8080/TCP  1s

==> v1/Deployment
NAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
minminmsn-harbor-adminserver    1        1        1           0          1s
minminmsn-harbor-chartmuseum    1        1        1           0          1s
minminmsn-harbor-clair          1        0        0           0          1s
minminmsn-harbor-core           1        0        0           0          1s
minminmsn-harbor-jobservice     1        0        0           0          1s
minminmsn-harbor-notary-server  1        0        0           0          1s
minminmsn-harbor-notary-signer  1        0        0           0          1s
minminmsn-harbor-portal         1        0        0           0          1s
minminmsn-harbor-registry       1        0        0           0          1s

==> v1/StatefulSet
NAME                    DESIRED  CURRENT  AGE
minminmsn-harbor-database  1        1        1s
minminmsn-harbor-redis     1        1        1s

==> v1beta1/Ingress
NAME                   HOSTS                                                    ADDRESS  PORTS  AGE
minminmsn-harbor-ingress  core-harbor.minminmsn.com,notary-harbor.minminmsn.com  80, 443  1s

==> v1/Pod(related)
NAME                                          READY  STATUS             RESTARTS  AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd     0/1    ContainerCreating  0         1s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb    0/1    Pending            0         1s
minminmsn-harbor-clair-54465ff7dd-d7bxx          0/1    Pending            0         1s
minminmsn-harbor-core-587cc5d9b5-2xxl9           0/1    Pending            0         1s
minminmsn-harbor-jobservice-764bb697d-wsxqx      0/1    Pending            0         1s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c  0/1    Pending            0         1s
minminmsn-harbor-notary-signer-8466d68f5b-klv76  0/1    Pending            0         1s
minminmsn-harbor-database-0                      0/1    Pending            0         1s
minminmsn-harbor-redis-0                         0/1    Pending            0         1s

==> v1/Secret
NAME                       TYPE    DATA  AGE
minminmsn-harbor-adminserver  Opaque  4     1s
minminmsn-harbor-chartmuseum  Opaque  1     1s
minminmsn-harbor-core         Opaque  4     1s
minminmsn-harbor-database     Opaque  1     1s
minminmsn-harbor-jobservice   Opaque  1     1s
minminmsn-harbor-registry     Opaque  1     1s

==> v1/ConfigMap
NAME                         DATA  AGE
minminmsn-harbor-adminserver    39    1s
minminmsn-harbor-chartmuseum    24    1s
minminmsn-harbor-clair          1     1s
minminmsn-harbor-core           1     1s
minminmsn-harbor-jobservice     1     1s
minminmsn-harbor-notary-server  5     1s
minminmsn-harbor-registry       2     1s

NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. 
For more details, please visit https://github.com/goharbor/harbor.

3.查看新生成Pods的信息
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS              RESTARTS   AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      0/1     ContainerCreating   0          9s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     0/1     ContainerCreating   0          9s
minminmsn-harbor-clair-54465ff7dd-d7bxx           0/1     Running             0          9s
minminmsn-harbor-core-587cc5d9b5-2xxl9            0/1     Running             0          9s
minminmsn-harbor-database-0                       0/1     Init:0/1            0          9s
minminmsn-harbor-jobservice-764bb697d-wsxqx       0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   0/1     ContainerCreating   0          9s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   0/1     ContainerCreating   0          9s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          0/1     ContainerCreating   0          8s
minminmsn-harbor-redis-0                          0/1     ContainerCreating   0          9s
minminmsn-harbor-registry-755746c5bb-q8m55        0/2     ContainerCreating   0          8s

再等2分钟查看就上恢复了

[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
jenkins-0                                      1/1     Running   0          62d
rbd-provisioner-67b4857bcd-rjwlg               1/1     Running   0          61d
minminmsn-harbor-adminserver-b5d58db8c-wmrbd      1/1     Running   1          2m33s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     1/1     Running   0          2m33s
minminmsn-harbor-clair-54465ff7dd-d7bxx           1/1     Running   1          2m33s
minminmsn-harbor-core-587cc5d9b5-2xxl9            1/1     Running   1          2m33s
minminmsn-harbor-database-0                       1/1     Running   0          2m33s
minminmsn-harbor-jobservice-764bb697d-wsxqx       1/1     Running   0          2m33s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c   1/1     Running   0          2m33s
minminmsn-harbor-notary-signer-8466d68f5b-klv76   1/1     Running   0          2m33s
minminmsn-harbor-portal-64cf8b9b69-xm8nl          1/1     Running   0          2m32s
minminmsn-harbor-redis-0                          1/1     Running   0          2m33s
minminmsn-harbor-registry-755746c5bb-q8m55        2/2     Running   0          2m32s

4.Harbor控制验证证书更新了项目也恢复了 https://core-harbor.minminmsn.com/harbor/projects