MongoDB集群同步

原创

雪人

发布于 2024-07-12 20:55:29

1140

发布于 2024-07-12 20:55:29

实现 MongoDB Cluster-to-Cluster 即集群同步的工具是：mongosync

详情可参考如下官方文档：

https://www.mongodb.com/zh-cn/docs/cluster-to-cluster-sync/current/quickstart/

以上这个地址的文档一看就是机器翻译的，可能有不恰当的地方，但基本可参考使用。

以下是本次在某项目地配置集群同步的简要步骤，可参考使用。

一、首先在目的集群安装同步工具mongosync

本次使用的是版本匹配的1.5.0版本：

https://fastdl.mongodb.org/tools/mongosync/mongosync-rhel80-x86_64-1.5.0.tgz

以下是我习惯使用的安装脚本：

KDR=/u01
cd ${KDR}
TGZ=mongosync-rhel80-x86_64-1.5.0
#cp_unzip_chown_ln:
cp /u01/nfs/MongoDB/${TGZ}.tgz .
tar zxvf ${TGZ}.tgz
chown -R mongod:mongod ${TGZ}
ln -s ${KDR}/${TGZ}/bin/* /usr/local/bin/

二、集群规划信息配置

在/etc/hosts文件追加host定义(mongodb25~33)，方便用于MongoDB集群同步脚本调用以下代码已做脱敏处理

cat >> /etc/hosts << EOF
# 生产集群
2409:xx:1a mongodb25
2409:xx:1b mongodb26
2409:xx:1c mongodb27
# 容灾集群
2409:xx:1d mongodb28
2409:xx:1e mongodb29
2409:xx:1f mongodb30
# 历史数据集群
2409:xx:20 mongodb31
2409:xx:21 mongodb32
2409:xx:22 mongodb33
EOF

三、mongosync 执行

LOGFILE=mongosync.liking.`date +%Y%m%d-%H:%M`.log
mongosync \
    --logPath /var/log/mongosync \
    --cluster0 mongodb://admin:passwd@mongodb28:20000,mongodb29:20000,mongodb30:20000/?authSource=admin \
    --cluster1 mongodb://admin:passwd@mongodb31:20000,mongodb32:20000,mongodb33:20000/?authSource=admin \
    --verbosity INFO > ${LOGFILE} 2>&1 &

启用同步服务后，可以在日志目录/var/log/mongosync查看日志信息。

四、POST 同步命令 start

■ 同步整个集群【本次使用】

curl "http://localhost:27182/api/v1/start" -XPOST --data '{
  "source": "cluster0",
  "destination": "cluster1"
}'

■ 同步指定的数据库

curl "http://localhost:27182/api/v1/start" -XPOST --data '{
  "source": "cluster0",
  "destination": "cluster1"
  "includeNameSpaces": [
    {
      "database": "flowtest"
    }
  ]
}'

■ 同步指定的集合

curl "http://localhost:27182/api/v1/start" -XPOST --data '{
  "source": "cluster0",
  "destination": "cluster1",
  "includeNameSpaces": [
    {
      "database": "flowtest",
      "collections": [ "activityConfiguration", "history_task", "processConfiguration", "processDataObj",  "snapshotResource" ]
    }
  ]
}'

■ 同步时的运维操作

curl "http://localhost:27182/api/v1/progress" -XGET
curl "http://localhost:27182/api/v1/pause" -XPOST --data '{}'
curl "http://localhost:27182/api/v1/resume" -XPOST --data '{}'

五、全量同步日志1

本次全量同步的数据规模如下：

[mongos] testdb> show databases
admin     252.00 KiB
config     35.56 MiB
flowtest  624.74 GiB

同步开始阶段的重要信息如下：

{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","uri":"/api/v1/start","method":"POST","body":"{\n \"source\": \"cluster0\",\n  \"destination\": \"cluster1\"\n}","clientIP":"::1","traceID":"fec8eb4b-8123-4a33-a1a5-d6cf57b431ea","time":"2024-04-18T16:47:49.511390067+08:00","message":"received request"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","source":"cluster0","destination":"cluster1","reversible":false,"enableUserWriteBlocking":false,"time":"2024-04-18T16:47:49.511783755+08:00","message":"Start handlercalled"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","sourceURI":"mongodb28:20000,mongodb29:20000,mongodb30:20000","dstURI":"mongodb31:20000,mongodb32:20000,mongodb33:20000","time":"2024-04-18T16:47:49.511812312+08:00","message":"Creating clients and connecting to clusters."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","clusterType":"src","operation description":"Pinging deployment at mongodb://admin:<REDACTED>@mongodb28:20000,mongodb29:20000,mongodb30:20000/?authSource=admin to establish initial connection.","attemptNumber":0,"durationSoFarSecs":0,"durationLimitSecs":10,"time":"2024-04-18T16:47:49.512143259+08:00","message":"Trying operation."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","clusterType":"dst","operation description":"Pinging deployment at mongodb://admin:<REDACTED>@mongodb31:20000,mongodb32:20000,mongodb33:20000/?authSource=admin to establish initial connection.","attemptNumber":0,"durationSoFarSecs":0,"durationLimitSecs":10,"time":"2024-04-18T16:47:49.530184069+08:00","message":"Trying operation."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","time":"2024-04-18T16:47:49.545274189+08:00","message":"Successfully initialized source and destination clients"}

全量同步完毕阶段的重要日志信息如下：

{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Collection Copy","time":"2024-04-18T20:29:55.31864523+08:00","message":"All Collection Copy components have stopped."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Collection Copy","time":"2024-04-18T20:29:55.318660079+08:00","message":"Completed writing 6487 / 6487 partitions to destination cluster"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","totalBytes":2717579843528,"copiedBytes":2717920929608,"time":"2024-04-18T20:29:55.318684889+08:00","message":"Finished collection copy phase."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","time":"2024-04-18T20:29:55.330386515+08:00","message":"Waiting for collection copy phase to complete."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","time":"2024-04-18T20:29:55.351147335+08:00","message":"Running change event application phase."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Change Event Application","time":"2024-04-18T20:29:55.365007664+08:00","message":"Starting index checker service which runs a cycle every 60 seconds."}

以上日志可见：

620G数据，同步耗时3.5小时左右，性能比之前测试稍慢。

再次印证了mongodb本身的压缩算法较好，实际fs层面的数据大小如下：

"totalBytes":2717579843528,"copiedBytes":2717920929608,"time":"2024-04-18T20:29:55.318684889+08:00","message":"Finished collection copy phase."

六、全量同步日志2

当晚实际生产集群全量同步的数据规模如下：

flowtest            684.20 GiB
{"estimatedTotalBytes":2972595055214,"estimatedCopiedBytes":2973012240862}

实际耗时2h55m，可见同样硬件、网络环境夜间性能稍好。

七、思考

通过以上实践，显而易见，MongoDB集群同步技术使得我们后续迁移数据库变得轻而易举，实际割接时间可控制在几分钟以内。

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

mongodb

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

mongodb

登录后参与评论

0 条评论

热度