目录
SOFARegistry 是蚂蚁金服开源的一个生产级、高时效、高可用的服务注册中心。本系列将带领大家一起分析其MetaServer的实现机制,本文为第一篇,介绍MetaServer总体架构。
本系列总体参考了官方的博客,具体请参见"0xFF 参考"。大家可以把参考作为总纲,我这系列文章作为注释补遗翻阅。
在微服务架构下,一个互联网应用的服务端背后往往存在大量服务间的相互调用。例如服务 A 在链路上依赖于服务 B,那么在业务发生时,服务 A 需要知道服务 B 的地址,才能完成服务调用。而分布式架构下,每个服务往往都是集群部署的,集群中的机器也是经常变化的,所以服务 B 的地址不是固定不变的。如果要保证业务的可靠性,服务调用者则需要感知被调用服务的地址变化。
既然成千上万的服务调用者都要感知这样的变化,那这种感知能力便下沉成为微服务中一种固定的架构模式:服务注册中心。
服务注册中心里,有服务提供者和服务消费者两种重要的角色,服务调用方是消费者,服务被调方是提供者。对于同一台机器,往往兼具两者角色,既被其它服务调用,也调用其它服务。服务提供者将自身提供的服务信息发布到服务注册中心,服务消费者通过订阅的方式感知所依赖服务的信息是否发生变化。
服务注册中心在服务调用的场景中,扮演一个“中介”的角色,服务发布者 (Publisher) 将服务发布到服务注册中心,服务调用方 (Subscriber) 通过访问服务注册中心就能够获取到服务信息,进而实现调用。
Subscriber 在第一次调用服务时,会通过 Registry 找到相应的服务的 IP 地址列表,通过负载均衡算法从 IP 列表中取一个服务提供者的服务器调用服务。同时 Subscriber 会将 Publisher 的服务列表数据缓存到本地,供后续使用。当 Subscriber 后续再调用 Publisher 时,优先使用缓存的地址列表,不需要再去请求Registry。
+----------+ +---------+
|Subscriber| <--------+ +--------+ |Publisher|
+----------+ | | +---------+
| |
+----------+ | v
|Subscriber| <--------+ +----------++ +---------+
+----------+ <---+ Registry | <-------+Publisher|
| +----------++ +---------+
+----------+ | ^
|Subscriber| <--------+ |
+----------+ |
| +---------+
+----------+Publisher|
+---------+
服务注册中心Registry的最主要能力是服务注册和服务发现两个过程。
SOFARegistry 作为服务注册中心,分为4个层,分别为:
Client 层是应用服务器集群。Client 层是应用层,每个应用系统通过依赖注册中心相关的客户端 jar 包,通过编程方式来使用服务注册中心的服务发布和服务订阅能力。
Session层是服务器集群。顾名思义,Session 层是会话层,通过长连接和 Client 层的应用服务器保持通讯,负责接收 Client 的服务发布和服务订阅请求。
在服务注册中心的服务端因为每个存储节点对应的客户端的链接数据量有限,必须进行特殊的一层划分用于专门收敛无限扩充的客户端连接,然后在透传相应的请求到存储层。
该层只在内存中保存各个服务的发布订阅关系,对于具体的服务信息,只在 Client 层和 Data 层之间透传转发。Session 层是一个无数据状态的代理层,可以随着 Client 层应用规模的增长而扩容。
因为 SOFARegistry 的服务发现需要较高的时效性,对外表现为主动推送变更到客户端,所以推送的主体实现也集中在 Session 层,内部的推拉结合主要是通过 Data 存储层的数据版本变更推送到所有 Session 节点,各个 Session 节点根据存储的订阅关系和首次订阅获取的数据版本信息进行比对,最终确定推送给那些服务消费方客户端。
数据服务器集群。Data 层通过分片存储的方式保存着所用应用的服务注册数据。数据按照 dataInfoId(每一份服务数据的唯一标识)进行一致性 Hash 分片,多副本备份,保证数据的高可用。Data 层可以随着数据规模的增长,在不影响业务的前提下实现平滑的扩缩容。
元数据服务器集群。这个集群管辖的范围是 Session 服务器集群和 Data 服务器集群的服务器信息,其角色就相当于 SOFARegistry 架构内部的服务注册中心,只不过 SOFARegistry 作为服务注册中心是服务于广大应用服务层,而 Meta 集群是服务于 SOFARegistry 内部的 Session 集群和 Data 集群,Meta 层能够感知到 Session 节点和 Data 节点的变化,并通知集群的其它节点。
SOFARegistry 内部为什么要进行数据分层,是因为系统容量的限制。
在 SOFARegistry 的应用场景中,体量庞大的数据主要有两类:Session 数据、服务信息数据。两类数据的相同之处在于其数据量都会不断扩展,而不同的是其扩展的原因并不相同:
所以 SOFARegistry 通过分层设计,将两种数据隔离,从而使二者的扩容互不影响。
这也是 SOFARegistry 设计三层模型的原因,通过 SessionServer 来负责与 Client 的连接,将每个 Client 的连接数收敛到 1,这样当 Client 数量增长时,只需要扩容 SessionServer 集群就可以了。 所以从设计初衷上我们就能够看出来 SessionServer 必须要满足的两个主要能力:从 DataServer 获取服务信息数据;以及保存与 Client 的会话。
MetaServer 在 SOFARegistry 中,承担着集群元数据管理的角色,用来维护集群成员列表,可以认为是 SOFARegistry 注册中心的注册中心。
MetaServer 作为 SOFARegistry 的元数据中心,其核心功能可以概括为:集群成员列表管理。比如:
当 SessionServer 和 DataServer 需要知道集群列表,并且需要扩缩容时,MetaServer 将会提供相应的数据。
其内部架构如下图所示:
MetaServer 基于 Bolt, 通过 TCP 私有协议的形式对外提供服务,包括 DataServer, SessionServer 等,处理节点的注册,续约和列表查询等请求。
同时也基于 Http 协议提供控制接口,比如可以控制 session 节点是否开启变更通知, 健康检查接口等。
成员列表数据存储在 Repository 中,Repository 被一致性协议层进行包装,作为 SOFAJRaft 的状态机实现,所有对 Repository 的操作都会同步到其他节点, 通过Rgistry来操作存储层。
MetaServer 使用 Raft 协议保证高可用和数据一致性, 同时也会保持与注册的节点的心跳,对于心跳超时没有续约的节点进行驱逐,来保证数据的有效性。
在可用性方面,只要未超过半数节点挂掉,集群都可以正常对外提供服务,半数以上挂掉,Raft 协议无法选主和日志复制,因此无法保证注册的成员数据的一致性和有效性。整个集群不可用 不会影响 Data 和 Session 节点的正常功能,只是无法感知节点列表变化。
空谈无用,just show the code。于是让我们带着问题来思考,即从宏观和微观角度来思考MetaServer应该实现什么功能,具体是怎么实现的。
思考:
下面我们就一一分析。
我们在 sofa-registry-5.4.2/server/server/meta/src/main/java/com/alipay/sofa/registry/server/meta
看看目录和文件结构。
按照目录我们可以大致了解功能
具体代码结构如下:
.
├── MetaApplication.java
├── bootstrap
│ ├── AbstractNodeConfigBean.java
│ ├── EnableMetaServer.java
│ ├── MetaServerBootstrap.java
│ ├── MetaServerConfig.java
│ ├── MetaServerConfigBean.java
│ ├── MetaServerConfiguration.java
│ ├── MetaServerInitializerConfiguration.java
│ ├── NodeConfig.java
│ ├── NodeConfigBeanProperty.java
│ └── ServiceFactory.java
├── executor
│ └── ExecutorManager.java
├── listener
│ ├── DataNodeChangePushTaskListener.java
│ ├── PersistenceDataChangeNotifyTaskListener.java
│ ├── ReceiveStatusConfirmNotifyTaskListener.java
│ └── SessionNodeChangePushTaskListener.java
├── node
│ ├── DataNodeService.java
│ ├── MetaNodeService.java
│ ├── NodeOperator.java
│ ├── NodeService.java
│ ├── SessionNodeService.java
│ └── impl
│ ├── DataNodeServiceImpl.java
│ ├── MetaNodeServiceImpl.java
│ └── SessionNodeServiceImpl.java
├── registry
│ ├── MetaServerRegistry.java
│ └── Registry.java
├── remoting
│ ├── DataNodeExchanger.java
│ ├── MetaClientExchanger.java
│ ├── MetaServerExchanger.java
│ ├── RaftExchanger.java
│ ├── SessionNodeExchanger.java
│ ├── connection
│ │ ├── DataConnectionHandler.java
│ │ ├── MetaConnectionHandler.java
│ │ ├── NodeConnectManager.java
│ │ └── SessionConnectionHandler.java
│ └── handler
│ ├── AbstractServerHandler.java
│ ├── DataNodeHandler.java
│ ├── FetchProvideDataRequestHandler.java
│ ├── GetNodesRequestHandler.java
│ ├── RenewNodesRequestHandler.java
│ └── SessionNodeHandler.java
├── repository
│ ├── NodeConfirmStatusService.java
│ ├── NodeRepository.java
│ ├── RepositoryService.java
│ ├── VersionRepositoryService.java
│ ├── annotation
│ │ └── RaftAnnotationBeanPostProcessor.java
│ └── service
│ ├── DataConfirmStatusService.java
│ ├── DataRepositoryService.java
│ ├── MetaRepositoryService.java
│ ├── SessionConfirmStatusService.java
│ ├── SessionRepositoryService.java
│ └── SessionVersionRepositoryService.java
├── resource
│ ├── BlacklistDataResource.java
│ ├── DecisionModeResource.java
│ ├── HealthResource.java
│ ├── MetaDigestResource.java
│ ├── MetaStoreResource.java
│ ├── PersistentDataResource.java
│ ├── RenewSwitchResource.java
│ └── StopPushDataResource.java
├── store
│ ├── DataStoreService.java
│ ├── MetaStoreService.java
│ ├── RenewDecorate.java
│ ├── SessionStoreService.java
│ └── StoreService.java
└── task
├── AbstractMetaServerTask.java
├── Constant.java
├── DataNodeChangePushTask.java
├── MetaServerTask.java
├── PersistenceDataChangeNotifyTask.java
├── ReceiveStatusConfirmNotifyTask.java
├── SessionNodeChangePushTask.java
└── processor
├── DataNodeSingleTaskProcessor.java
├── MetaNodeSingleTaskProcessor.java
└── SessionNodeSingleTaskProcessor.java
16 directories, 75 files
启动可以参考 https://www.sofastack.tech/projects/sofa-registry/server-quick-start/
SOFARegistry 支持两种部署模式,分别是集成部署模式及独立部署模式。
启动命令:sh bin/startup.sh
双击 bin 目录下的 startup.bat 运行文件。
通过下列log我们可以看到启动信息。
MetaApplication
[2020-09-12 20:23:05,463][INFO][main][MetaServerBootstrap] - Open meta server port 9612 success!
[2020-09-12 20:23:08,198][INFO][main][MetaServerBootstrap] - Open http server port 9615 success!
[2020-09-12 20:23:10,298][INFO][main][MetaServerBootstrap] - Raft server port 9614 start success!group RegistryGroup
[2020-09-12 20:23:10,322][INFO][main][MetaServerInitializerConfiguration] - Started MetaServer
DataApplication
[2020-09-12 20:23:25,004][INFO][main][DataServerBootstrap] - Open http server port 9622 success!
[2020-09-12 20:23:26,084][INFO][main][DataServerBootstrap] - start server success
[2020-09-12 20:23:26,094][INFO][main][DataApplication] - Started DataApplication in 10.217 seconds (JVM running for 11.316)
SessionApplication
[2020-09-12 20:23:50,243][INFO][main][SessionServerBootstrap] - Open http server port 9603 success!
[2020-09-12 20:23:50,464][INFO][main][SessionServerInitializer] - Started SessionServer
[2020-09-12 20:23:50,526][INFO][main][SessionApplication] - Started SessionApplication in 12.516 seconds (JVM running for 13.783)
各个 Server 的默认端口分别为:
meta.server.sessionServerPort=9610
meta.server.dataServerPort=9611
meta.server.metaServerPort=9612
meta.server.raftServerPort=9614
meta.server.httpServerPort=9615
可访问三个角色提供的健康监测 API,或查看日志 logs/registry-startup.log:
# 查看meta角色的健康检测接口:
$ curl http://localhost:9615/health/check
{"success":true,"message":"... raftStatus:Leader"}
# 查看data角色的健康检测接口:
$ curl http://localhost:9622/health/check
{"success":true,"message":"... status:WORKING"}
# 查看session角色的健康检测接口:
$ curl http://localhost:9603/health/check
{"success":true,"message":"..."}
在这里我们可以看出来各种集群是如何搭建,以及如何在集群内部通讯,分布式协调。
按照常理来说,每个集群都应该依赖zookeeper之类的软件来进行自己内部的协调,比如统一命名服务、状态同步服务、集群管理、分布式应用配置项。但实际上我们没有发现类似的使用。
具体看配置文件发现,每台机器都要设置所有的metaServer的host。这说明Data Server, Session Server则强依赖Meta Server。
实际上,MetaServer 使用 Raft 协议保证高可用和数据一致性, 同时也会保持与注册的节点的心跳,对于心跳超时没有续约的节点进行驱逐,来保证数据的有效性。Meta 层能够感知到 Session 节点和 Data 节点的变化,并通知集群的其它节点。
这就涉及到各个角色的 failover 机制:
每台机器在部署时需要修改 conf/application.properties 配置:
# 将3台meta机器的ip或hostname配置到下方(填入的hostname会被内部解析为ip地址)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
每台机器在部署时需要修改 conf/application.properties 配置:
# 将3台 meta 机器的 ip 或 hostname 配置到下方(填入的 hostname 会被内部解析为 ip 地址)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
data.server.numberOfReplicas=1000
每台机器在部署时需要修改 conf/application.properties 配置:
# 将3台 meta 机器的 ip 或 hostname 配置到下方(填入的 hostname 会被内部解析为 ip 地址)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
nodes.localRegion=DefaultZone
MetaServer 在启动时,会启动三个 Bolt Server,并且注册 Processor Handler,处理对应的请求:
然后启动 HttpServer, 用于处理 Admin 请求,提供推送开关,集群数据查询等 Http 接口。
最后启动 Raft 服务, 每个节点同时作为 RaftClient 和 RaftServer, 用于集群间的变更和数据同步。
MetaServer 是一个SpringBootApplication,主要起作用的就是EnableMetaServer。
@EnableMetaServer
@SpringBootApplication
public class MetaApplication {
public static void main(String[] args) {
SpringApplication.run(MetaApplication.class, args);
}
}
具体参见下图
+-------------------+
| @EnableMetaServer |
| |
| MetaApplication |
+-------------------+
EnableMetaServer注解引入了MetaServerConfiguration
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(MetaServerConfiguration.class)
public @interface EnableMetaServer {
}
MetaServerConfiguration是各种配置,并且引入 MetaServerInitializerConfiguration负责启动。
@Configuration
@Import(MetaServerInitializerConfiguration.class)
@EnableConfigurationProperties
public class MetaServerConfiguration {
}
所以我们可以知道:
于是程序总结结构演化为下图:
(Init)
+------------------------------------+
| MetaServerInitializerConfiguration |
+-------------+----------------------+
^
+-------------------+ |
| @EnableMetaServer | |
| | |
| MetaApplication | |
+-------------+-----+ | (Configuration)
| +---------+---------------+
+--------------> | MetaServerConfiguration |
+-------------------------+
下面我们开始引入配置。
MetaServer 模块的各个 bean 在 JavaConfig 中统一配置,JavaConfig 类为 MetaServerConfiguration。
MetaServerConfiguration 具体有以下几类配置:
具体缩略版代码如下 :
@Configuration
@Import(MetaServerInitializerConfiguration.class)
@EnableConfigurationProperties
public class MetaServerConfiguration {
@Bean
@ConditionalOnMissingBean
public MetaServerBootstrap metaServerBootstrap() {
}
@Configuration
protected static class MetaServerConfigBeanConfiguration {
}
@Configuration
public static class MetaServerServiceConfiguration {
}
@Configuration
public static class MetaServerRepositoryConfiguration {
}
@Configuration
public static class MetaServerRemotingConfiguration {
}
@Configuration
public static class ResourceConfiguration {
}
@Configuration
public static class MetaServerTaskConfiguration {
}
@Configuration
public static class ExecutorConfiguation {
}
@Configuration
public static class MetaDBConfiguration {
}
}
我们的图进化如下:
(Init)
+------------------------------------+
| MetaServerInitializerConfiguration |
+-------------+----------------------+ +---------------------+
^ +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer | | | +---------------------------------+
| | | +--> |MetaServerConfigBeanConfiguration|
| MetaApplication | | | +---------------------------------+
+--------------+----+ | | +------------------------------+
| | +--> |MetaServerServiceConfiguration|
| | | +------------------------------+
| | | +---------------------------------+
| | +--> |MetaServerRepositoryConfiguration|
| | | +---------------------------------+
| | (Configuration) | +-------------------------------+
| +---------+---------------+ +--> |MetaServerRemotingConfiguration|
+--------------> | MetaServerConfiguration | +-----+ +-------------------------------+
+-------------------------+ | +----------------------+
+--> |ResourceConfiguration |
| +----------------------+
| +---------------------------+
+--> |MetaServerTaskConfiguration|
| +---------------------------+
| +---------------------+
+--> |ExecutorConfiguation |
| +---------------------+
| +--------------------+
+--> |MetaDBConfiguration |
+--------------------+
下图为了手机阅读
这里要特殊提一下handler的配置,因为这是后续分析的主体之一,是三个 Bolt Server的handler。
@Configuration
public static class MetaServerRemotingConfiguration {
@Bean
public Exchange boltExchange() {
return new BoltExchange();
}
@Bean
public Exchange jerseyExchange() {
return new JerseyExchange();
}
@Bean(name = "sessionServerHandlers")
public Collection<AbstractServerHandler> sessionServerHandlers() {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(sessionConnectionHandler());
list.add(sessionNodeHandler());
list.add(renewNodesRequestHandler());
list.add(getNodesRequestHandler());
list.add(fetchProvideDataRequestHandler());
return list;
}
@Bean(name = "dataServerHandlers")
public Collection<AbstractServerHandler> dataServerHandlers() {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(dataConnectionHandler());
list.add(getNodesRequestHandler());
list.add(dataNodeHandler());
list.add(renewNodesRequestHandler());
list.add(fetchProvideDataRequestHandler());
return list;
}
@Bean(name = "metaServerHandlers")
public Collection<AbstractServerHandler> metaServerHandlers() {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(metaConnectionHandler());
list.add(getNodesRequestHandler());
return list;
}
}
于是我们的总体架构进化具体见下图
(Init)
+------------------------------------+
| MetaServerInitializerConfiguration |
+--------------+---------------------+
^
|
|
|
|
| +---------------------+
| +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer | | | +---------------------------------+
| | | +--> |MetaServerConfigBeanConfiguration|
| MetaApplication | | | +---------------------------------+
+--------------+----+ | | +------------------------------+ +-----------------------+
| | +--> |MetaServerServiceConfiguration| +---> | sessionServerHandlers |
| | | +------------------------------+ | +-----------------------+
| | | +---------------------------------+ | +--------------------+
| | +--> |MetaServerRepositoryConfiguration+-------> | dataServerHandlers |
| | | +---------------------------------+ | +--------------------+
| | (Configuration) | +-------------------------------+ | +--------------------+
| +---------+---------------+ +--> |MetaServerRemotingConfiguration| +---> | metaServerHandlers |
+-----------> | MetaServerConfiguration | +-----+ +-------------------------------+ +--------------------+
+-------------------------+ | +----------------------+
+--> |ResourceConfiguration |
| +----------------------+
| +---------------------------+
+--> |MetaServerTaskConfiguration|
| +---------------------------+
| +---------------------+
+--> |ExecutorConfiguation |
| +---------------------+
| +--------------------+
+--> |MetaDBConfiguration |
+--------------------+
手机上阅读如下:
关于handler配置,进一步细化如下图
(Init)
+--> sessionConnectionHandler
|
|
+--> sessionNodeHandler
+-----------------------+ |
+---> | sessionServerHandlers +--+
+---------------------+ | +-----------------------+ +--> renewNodesRequestHandler
+--> | MetaServerBootstrap | | |
| +---------------------+ | |
| +---------------------------------+ | +--> getNodesRequestHandler
+--> |MetaServerConfigBeanConfiguration| | |
| +---------------------------------+ | |
| +------------------------------+ | +--> fetchProvideDataRequestHandler
+--> |MetaServerServiceConfiguration| |
| +------------------------------+ |
| +---------------------------------+ | +--> dataConnectionHandler
+--> |MetaServerRepositoryConfiguration+---+ |
| +---------------------------------+ | |
| +-------------------------------+ | +--------------------+ +--> getNodesRequestHandler
+-------------------------+ +--> |MetaServerRemotingConfiguration| +---> | dataServerHandlers +-----+
| MetaServerConfiguration | +-----+ +-------------------------------+ | +--------------------+ |
+-------------------------+ | +----------------------+ | +--> dataNodeHandler
+--> |ResourceConfiguration | | |
| +----------------------+ | |
| +---------------------------+ | +--> renewNodesRequestHandler
+--> |MetaServerTaskConfiguration| | |
| +---------------------------+ | |
| +---------------------+ | +--> fetchProvideDataRequestHandler
+--> |ExecutorConfiguation | |
| +---------------------+ |
| +--------------------+ |
+--> |MetaDBConfiguration | | +---> metaConnectionHandler
+--------------------+ | +--------------------+ |
+----> | metaServerHandlers +-----+
+--------------------+ +---> getNodesRequestHandler
手机上如图
这个就对应了参考中的图例:
MetaServer 在启动时,会启动三个 Bolt Server,并且注册 Processor Handler,处理对应的请求:
系统是通过对MetaServerBootstrap的控制来完成了启动。
MetaServer 模块的各个 bean 在 JavaConfig 中统一配置,JavaConfig 类为 MetaServerConfiguration。
启动入口类为 MetaServerInitializerConfiguration,该类不由 JavaConfig 管理配置,而是继承了 SmartLifecycle 接口,在启动时由 Spring 框架调用其 start 方法。
public class MetaServerInitializerConfiguration implements SmartLifecycle {
@Autowired
private MetaServerBootstrap metaServerBootstrap;
@Override
public void start() {
metaServerBootstrap.start();
MetaServerInitializerConfiguration.this.running.set(true);
}
@Override
public void stop() {
this.running.set(false);
metaServerBootstrap.destroy();
}
}
具体见下图,因为 metaServerBootstrap 是通过配置生成,所以init过程指向配置部分。
(Init)
+------------------------------------+ start,stop
| MetaServerInitializerConfiguration +----------------+
+--------------+---------------------+ |
^ |
| |
| |
| |
| v
| +---------------------+
| +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer | | | +---------------------------------+
| | | +--> |MetaServerConfigBeanConfiguration|
| MetaApplication | | | +---------------------------------+
+--------------+----+ | | +------------------------------+ +-----------------------+
| | +--> |MetaServerServiceConfiguration| +---> | sessionServerHandlers |
| | | +------------------------------+ | +-----------------------+
| | | +---------------------------------+ | +--------------------+
| | +--> |MetaServerRepositoryConfiguration+-------> | dataServerHandlers |
| | | +---------------------------------+ | +--------------------+
| | (Configuration) | +-------------------------------+ | +--------------------+
| +---------+---------------+ +--> |MetaServerRemotingConfiguration| +---> | metaServerHandlers |
+-----------> | MetaServerConfiguration | +-----+ +-------------------------------+ +--------------------+
+-------------------------+ | +----------------------+
+--> |ResourceConfiguration |
| +----------------------+
| +---------------------------+
+--> |MetaServerTaskConfiguration|
| +---------------------------+
| +---------------------+
+--> |ExecutorConfiguation |
| +---------------------+
| +--------------------+
+--> |MetaDBConfiguration |
+--------------------+
手机上如下
MetaServerBootstrap是核心启动类,该类主要包含了三类组件:外部节点通信组件、Raft 服务通信组件、定时器组件。
MetaServerBootstrap的定义如下:
public class MetaServerBootstrap {
@Autowired
private MetaServerConfig metaServerConfig;
@Autowired
private Exchange boltExchange;
@Autowired
private Exchange jerseyExchange;
@Autowired
private ExecutorManager executorManager;
@Resource(name = "sessionServerHandlers")
private Collection<AbstractServerHandler> sessionServerHandlers;
@Resource(name = "dataServerHandlers")
private Collection<AbstractServerHandler> dataServerHandlers;
@Resource(name = "metaServerHandlers")
private Collection<AbstractServerHandler> metaServerHandlers;
@Autowired
private ResourceConfig jerseyResourceConfig;
@Autowired
private ApplicationContext applicationContext;
@Autowired
private RaftExchanger raftExchanger;
private Server sessionServer;
private Server dataServer;
private Server metaServer;
private Server httpServer;
}
可以参见下图
+-----------------+
+----> | metaServerConfig|
| +-----------------+
| +--------------+
+----> | boltExchange |
| +--------------+
| +--------------+
+----> |jerseyExchange|
| +--------------+
| +---------------+
+----> |executorManager|
| +---------------+
| +---------------------+
+----> |sessionServerHandlers|
| +---------------------+
| +-----------------+
+----> |dataServerHandler|
(Init) | +-----------------+
+------------------------------------+ start,stop +---------------------+ | +------------------+
| MetaServerInitializerConfiguration +-------------> | MetaServerBootstrap | +--------> |metaServerHandlers|
+------------------------------------+ +---------------------+ | +------------------+
| +-------------+
+----> |raftExchanger|
| +-------------+
| +-------------+
+----> |sessionServer|
| +-------------+
| +-----------+
+----> |dataServer |
| +-----------+
| +-----------+
+----> |metaServer |
| +-----------+
| +----------+
+----> |httpServer|
| +----------+
| +---------------------+
+----> |jerseyResourceConfig |
| +---------------------+
| +-------------------+
+----> |applicationContext |
+-------------------+
手机参见下图
因为前面代码中有
@Autowired
private Exchange boltExchange;
@Autowired
private Exchange jerseyExchange;
这里要特殊说明下Exchange。
Exchange 作为 Client / Server 连接的抽象,负责节点之间的连接。在建立连接中,可以设置一系列应对不同任务的 handler (称之为 ChannelHandler),这些 ChannelHandler 有的作为 Listener 用来处理连接事件,有的作为 Processor 用来处理各种指定的事件,比如服务信息数据变化、Subscriber 注册等事件。
图 - 每一层各司其职,协同实现节点通信
各种节点在启动的时候,利用 Exchange 设置了一系列 ChannelHandler,比如:
private void openDataRegisterServer() {
try {
if (dataStart.compareAndSet(false, true)) {
dataServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getDataServerPort()), dataServerHandlers
.toArray(new ChannelHandler[dataServerHandlers.size()]));
}
}
}
前面已经提到,启动入口类为 MetaServerInitializerConfiguration,该类不由 JavaConfig 管理配置,而是继承了 SmartLifecycle 接口,在启动时由 Spring 框架调用其 start 方法。
该方法中调用了 MetaServerBootstrap # start 方法,用于启动一系列的初始化服务。
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.SmartLifecycle;
public class MetaServerInitializerConfiguration implements SmartLifecycle {
@Autowired
private MetaServerBootstrap metaServerBootstrap;‘
@Override
public void start() {
metaServerBootstrap.start();
MetaServerInitializerConfiguration.this.running.set(true);
}
}
前面提到,MetaServerBootstrap 在启动时,会启动三个 Bolt Server,并且注册 Processor Handler,处理对应的请求:
然后启动 HttpServer, 用于处理 Admin 请求,提供推送开关,集群数据查询等 Http 接口。
最后启动 Raft 服务, 每个节点同时作为 RaftClient 和 RaftServer, 用于集群间的变更和数据同步。为支持高可用特性,对于 MetaServer 来说,存储了 SOFARegistry 的元数据,为了保障 MetaServer 集群的一致性,其采用了 Raft 协议来进行选举和复制。
具体代码参见:
public void start() {
openSessionRegisterServer();
openDataRegisterServer();
openMetaRegisterServer();
openHttpServer();
initRaft();
}
private void openHttpServer() {
if (httpStart.compareAndSet(false, true)) {
bindResourceConfig();
httpServer = jerseyExchange.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getHttpServerPort()), new ResourceConfig[] { jerseyResourceConfig });
}
}
private void initRaft() {
raftExchanger.startRaftServer(executorManager);
raftExchanger.startRaftClient();
raftExchanger.startCliService();
}
Raft 和 Bolt 是SOFA所特殊实现,所以我们暂不介绍其底层机制,以后有机会单独开篇,下面提一下三个BoltServer。
private void openSessionRegisterServer() {
if (sessionStart.compareAndSet(false, true)) {
sessionServer = boltExchange
.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getSessionServerPort()), sessionServerHandlers
.toArray(new ChannelHandler[sessionServerHandlers.size()]));
}
}
private void openDataRegisterServer() {
if (dataStart.compareAndSet(false, true)) {
dataServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getDataServerPort()), dataServerHandlers
.toArray(new ChannelHandler[dataServerHandlers.size()]));
}
}
private void openMetaRegisterServer() {
if (metaStart.compareAndSet(false, true)) {
metaServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getMetaServerPort()), metaServerHandlers
.toArray(new ChannelHandler[metaServerHandlers.size()]));
}
}
这几个Server的handler就是我们前面配置的
@Resource(name = "sessionServerHandlers")
private Collection<AbstractServerHandler> sessionServerHandlers;
@Resource(name = "dataServerHandlers")
private Collection<AbstractServerHandler> dataServerHandlers;
@Resource(name = "metaServerHandlers")
private Collection<AbstractServerHandler> metaServerHandlers;
具体参见下图:
+------------------------+
+-----------------+ +---> |sessionConnectionHandler|
+----> | metaServerConfig| | +------------------------+
| +-----------------+ | +-------------------+
| +--------------+ +---> |sessionNodeHandler |
+----> | boltExchange | +-->+ +-------------------+
| +--------------+ | | +------------------------+
| +--------------+ | +---> |renewNodesRequestHandler|
+----> |jerseyExchange| | | +------------------------+
| +--------------+ | | +----------------------+
| +---------------+ | +---> |getNodesRequestHandler|
+----> |executorManager| | | +----------------------+
| +---------------+ +---------------------+ | | +------------------------------+
+-------------------------------------> |sessionServerHandlers+---+ +---> |fetchProvideDataRequestHandler|
| +-------------+ +---------------------+ +------------------------------+
+----> |sessionServer| +------------------^
| +-------------+ +---------------------+
| ----> |dataConnectionHandler|
| +------------------+ | +---------------------+
+---------------------+ +-------------------------------------> |dataServerHandlers+----------+ +----------------------+
| MetaServerBootstrap | +---+ +-----------+ +------------------+ +---> |getNodesRequestHandler|
+---------------------+ +----> |dataServer +----------------------^ | +----------------------+
| +-----------+ | +---------------+
| +---> |dataNodeHandler|
| +------------------+ | +---------------+
+------------------------------------> |metaServerHandlers+------+ | +------------------------+
| +-----------+ +------------------+ | +---> |renewNodesRequestHandler|
+----> |metaServer +----------------------^ | | +------------------------+
| +-----------+ | | +------------------------------+
| | +---> |fetchProvideDataRequestHandler|
| +-------------+ | +------------------------------+
+----> |raftExchanger| |
| +-------------+ |
| |
| +----------+ |
+----> |httpServer| | +---------------------+
| +----------+ | +--> |metaConnectionHandler|
| +---------------------+ +----> +---------------------+
+----> |jerseyResourceConfig | | +---------------------+
| +---------------------+ +--> |getNodesRequestHandle|
| +-------------------+ +---------------------+
+----> |applicationContext |
+-------------------+
手机上
在初始化时候,大致堆栈如下 :
interest:55, RenewNodesRequestHandler (com.alipay.sofa.registry.server.meta.remoting.handler)
interest:61, SyncUserProcessorAdapter (com.alipay.sofa.registry.remoting.bolt)
registerUserProcessor:42, UserProcessorRegisterHelper (com.alipay.remoting.rpc.protocol)
registerUserProcessor:376, RpcServer (com.alipay.remoting.rpc)
registerUserProcessorHandler:159, BoltServer (com.alipay.sofa.registry.remoting.bolt)
initHandler:139, BoltServer (com.alipay.sofa.registry.remoting.bolt)
startServer:92, BoltServer (com.alipay.sofa.registry.remoting.bolt)
open:65, BoltExchange (com.alipay.sofa.registry.remoting.bolt.exchange)
open:36, BoltExchange (com.alipay.sofa.registry.remoting.bolt.exchange)
openSessionRegisterServer:149, MetaServerBootstrap (com.alipay.sofa.registry.server.meta.bootstrap)
start:108, MetaServerBootstrap (com.alipay.sofa.registry.server.meta.bootstrap)
start:51, MetaServerInitializerConfiguration (com.alipay.sofa.registry.server.meta.bootstrap)
以SessionServer为例,在构建过程中,调用到 BoltExchange . open。
private void openSessionRegisterServer() {
if (sessionStart.compareAndSet(false, true)) {
sessionServer = boltExchange
.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getSessionServerPort()), sessionServerHandlers
.toArray(new ChannelHandler[sessionServerHandlers.size()]));
}
}
BoltExchange中有
@Override
public Server open(URL url, ChannelHandler... channelHandlers) {
BoltServer server = createBoltServer(url, channelHandlers);
setServer(server, url);
server.startServer();
return server;
}
BoltServer中有
public void startServer() {
if (isStarted.compareAndSet(false, true)) {
boltServer = new RpcServer(url.getPort(), true);
initHandler();
boltServer.start();
}
}
private void initHandler() {
if (initHandler.compareAndSet(false, true)) {
boltServer.addConnectionEventProcessor(ConnectionEventType.CONNECT,
new ConnectionEventAdapter(ConnectionEventType.CONNECT,
getConnectionEventHandler(), this));
boltServer.addConnectionEventProcessor(ConnectionEventType.CLOSE,
new ConnectionEventAdapter(ConnectionEventType.CLOSE, getConnectionEventHandler(),
this));
boltServer.addConnectionEventProcessor(ConnectionEventType.EXCEPTION,
new ConnectionEventAdapter(ConnectionEventType.EXCEPTION,
getConnectionEventHandler(), this));
registerUserProcessorHandler();
}
}
最后调用,会设定同步和异步的handler。
private void registerUserProcessorHandler() {
if (channelHandlers != null) {
for (ChannelHandler channelHandler : channelHandlers) {
if (HandlerType.PROCESSER.equals(channelHandler.getType())) {
if (InvokeType.SYNC.equals(channelHandler.getInvokeType())) {
boltServer.registerUserProcessor(new SyncUserProcessorAdapter(
channelHandler));
} else {
boltServer.registerUserProcessor(new AsyncUserProcessorAdapter(
channelHandler));
}
}
}
}
}
以使用 Jetty 的 openHttpServer 为例
启动 HttpServer, 用于处理 Admin 请求,提供推送开关,集群数据查询等 Http 接口。
public class JerseyJettyServer implements Server {
public static org.eclipse.jetty.server.Server createServer(final URI uri,
final ResourceConfig resourceConfig,
final boolean start) {
JettyHttpContainer handler = ContainerFactory.createContainer(JettyHttpContainer.class,
resourceConfig);
int defaultPort = Container.DEFAULT_HTTP_PORT;
final int port = (uri.getPort() == -1) ? defaultPort : uri.getPort();
final org.eclipse.jetty.server.Server server = new org.eclipse.jetty.server.Server(
new JettyConnectorThreadPool());
final ServerConnector http = new ServerConnector(server, new HttpConnectionCustomFactory());
http.setPort(port);
server.setConnectors(new Connector[] { http });
if (handler != null) {
server.setHandler(handler);
}
if (start) {
try {
// Start the server.
server.start();
}
}
return server;
}
}
其堆栈如下:
<init>:72, JerseyJettyServer (com.alipay.sofa.registry.remoting.jersey)
open:73, JerseyExchange (com.alipay.sofa.registry.remoting.jersey.exchange)
open:40, JerseyExchange (com.alipay.sofa.registry.remoting.jersey.exchange)
openHttpServer:205, MetaServerBootstrap (com.alipay.sofa.registry.server.meta.bootstrap)
start:114, MetaServerBootstrap (com.alipay.sofa.registry.server.meta.bootstrap)
start:51, MetaServerInitializerConfiguration (com.alipay.sofa.registry.server.meta.bootstrap)
doStart:173, DefaultLifecycleProcessor (org.springframework.context.support)
access$200:50, DefaultLifecycleProcessor (org.springframework.context.support)
start:350, DefaultLifecycleProcessor$LifecycleGroup (org.springframework.context.support)
startBeans:149, DefaultLifecycleProcessor (org.springframework.context.support)
onRefresh:112, DefaultLifecycleProcessor (org.springframework.context.support)
finishRefresh:880, AbstractApplicationContext (org.springframework.context.support)
refresh:546, AbstractApplicationContext (org.springframework.context.support)
refresh:693, SpringApplication (org.springframework.boot)
refreshContext:360, SpringApplication (org.springframework.boot)
run:303, SpringApplication (org.springframework.boot)
run:1118, SpringApplication (org.springframework.boot)
run:1107, SpringApplication (org.springframework.boot)
main:42, MetaApplication (com.alipay.sofa.registry.server.meta)
如下存储由Raft来保证数据一致性,后文针对此有详细讲解。
@RaftService(uniqueId = "sessionServer")
public class SessionVersionRepositoryService
@RaftService(uniqueId = "metaServer")
public class MetaRepositoryService
@RaftService(uniqueId = "dataServer")
public class DataRepositoryService
@RaftService(uniqueId = "sessionServer")
public class SessionRepositoryService
@RaftService(uniqueId = "dataServer")
public class DataConfirmStatusService
@RaftService(uniqueId = "sessionServer")
public class SessionConfirmStatusService
是一个启动各种管理线程的地方,都是定期常规管理任务。
public class ExecutorManager {
private ScheduledExecutorService scheduler;
private ThreadPoolExecutor heartbeatCheckExecutor;
private ThreadPoolExecutor checkDataChangeExecutor;
private ThreadPoolExecutor getOtherDataCenterChangeExecutor;
private ThreadPoolExecutor connectMetaServerExecutor;
private ThreadPoolExecutor checkNodeListChangePushExecutor;
private ThreadPoolExecutor raftClientRefreshExecutor;
private MetaServerConfig metaServerConfig;
@Autowired
private Registry metaServerRegistry;
@Autowired
private MetaClientExchanger metaClientExchanger;
@Autowired
private RaftExchanger raftExchanger;
public void startScheduler() {
init();
scheduler.schedule(new TimedSupervisorTask("HeartbeatCheck", scheduler, heartbeatCheckExecutor,
metaServerConfig.getSchedulerHeartbeatTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerHeartbeatExpBackOffBound(), () -> metaServerRegistry.evict()),
metaServerConfig.getSchedulerHeartbeatFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("GetOtherDataCenterChange", scheduler, getOtherDataCenterChangeExecutor,
metaServerConfig.getSchedulerGetDataChangeTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerGetDataChangeExpBackOffBound(), () -> {
metaServerRegistry.getOtherDataCenterNodeAndUpdate(NodeType.DATA);
metaServerRegistry.getOtherDataCenterNodeAndUpdate(NodeType.META);
}), metaServerConfig.getSchedulerGetDataChangeFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(new TimedSupervisorTask("ConnectMetaServer", scheduler, connectMetaServerExecutor,
metaServerConfig.getSchedulerConnectMetaServerTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerConnectMetaServerExpBackOffBound(),
() -> metaClientExchanger.connectServer()), metaServerConfig.getSchedulerConnectMetaServerFirstDelay(),
TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("CheckSessionNodeListChangePush", scheduler, checkNodeListChangePushExecutor,
metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(),
() -> metaServerRegistry.pushNodeListChange(NodeType.SESSION)),
metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("CheckDataNodeListChangePush", scheduler, checkNodeListChangePushExecutor,
metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(),
() -> metaServerRegistry.pushNodeListChange(NodeType.DATA)),
metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(new TimedSupervisorTask("RaftClientRefresh", scheduler, raftClientRefreshExecutor,
metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(),
() -> raftExchanger.refreshRaftClient()),
metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS);
}
}
ExecutorManager 的启动设置是在 MetaServerBootstrap.initRaft 之中,分别启动RaftServer,RaftClient,CliService。
private void initRaft() {
raftExchanger.startRaftServer(executorManager);
raftExchanger.startRaftClient();
raftExchanger.startCliService();
}
当 Raft 选出 Leader 之后,会调用到 ExecutorManager # startScheduler。
TimedSupervisorTask 实现了 TimerTask。
public class TimedSupervisorTask extends TimerTask {
private final ScheduledExecutorService scheduler;
private final ThreadPoolExecutor executor;
private final long timeoutMillis;
private final Runnable task;
private String name;
private final AtomicLong delay;
private final long maxDelay;
@Override
public void run() {
Future future = null;
try {
future = executor.submit(task);
// block until done or timeout
future.get(timeoutMillis, TimeUnit.MILLISECONDS);
delay.set(timeoutMillis);
} catch {
.....
} finally {
if (future != null) {
future.cancel(true);
}
scheduler.schedule(this, delay.get(), TimeUnit.MILLISECONDS);
}
}
}
可以看到管理任务大致有:
TimedSupervisorTask 会定期执行,比如 CheckDataNodeListChangePush 这个线程会定期执行 metaServerRegistry.pushNodeListChange(NodeType.DATA)) 来看看是否有变化。这里就会用到DataNode注册时候,Confirm的消息。
@Override
public void pushNodeListChange() {
NodeOperator<DataNode> fireNode;
if ((fireNode = dataConfirmStatusService.peekConfirmNode()) != null) {
NodeChangeResult nodeChangeResult = getNodeChangeResult();
Map<String, Map<String, DataNode>> map = nodeChangeResult.getNodes();
Map<String, DataNode> addNodes = map.get(nodeConfig.getLocalDataCenter());
if (addNodes != null) {
Map<String, DataNode> previousNodes = dataConfirmStatusService.putExpectNodes(
fireNode.getNode(), addNodes);
if (!previousNodes.isEmpty()) {
firePushDataListTask(fireNode, nodeChangeResult, previousNodes, true);
}
}
firePushSessionListTask(nodeChangeResult, fireNode.getNodeOperate().toString());
}
}
再比如定期去除过期的Node:
public class MetaServerRegistry implements Registry<Node> {
@Override
public void evict() {
for (NodeType nodeType : NodeType.values()) {
StoreService storeService = ServiceFactory.getStoreService(nodeType);
if (storeService != null) {
Collection<Node> expiredNodes = storeService.getExpired();
if (expiredNodes != null && !expiredNodes.isEmpty()) {
storeService.removeNodes(expiredNodes);
}
}
}
}
}
ServiceFactory 需要特殊说明,它提供了系统所需要的一系列服务。特殊之处在于,ServiceFactory 不是由 MetaServerBootstrap 负责启动,而是由 Spring 负责启动。因为 ServiceFactory 继承了ApplicationContextAware,所以启动时候生成。
在Web应用中,Spring容器通常采用声明式方式配置产生:开发者只要在web.xml中配置一个Listener,该Listener将会负责初始化Spring容器,MVC框架可以直接调用Spring容器中的Bean,无需访问Spring容器本身。在这种情况下,容器中的Bean处于容器管理下,无需主动访问容器,只需接受容器的依赖注入即可。
但在某些特殊的情况下,Bean需要实现某个功能,但该功能必须借助于Spring容器才能实现,此时就必须让该Bean先获取Spring容器,然后借助于Spring容器实现该功能。为了让Bean获取它所在的Spring容器,可以让该Bean实现ApplicationContextAware接口。
下面代码可以看出来,启动了一系列服务。
public class ServiceFactory implements ApplicationContextAware {
private static Map<NodeType, StoreService> storeServiceMap = new HashMap<>();
private static Map<NodeType, NodeConnectManager> connectManagerMap = new HashMap<>();
private static Map<NodeType, NodeService> nodeServiceMap = new HashMap<>();
}
storeServiceMap = {HashMap@5107} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionStoreService@5526}
key = {Node$NodeType@5525} "SESSION"
value = {SessionStoreService@5526}
{Node$NodeType@4815} "DATA" -> {DataStoreService@5527}
key = {Node$NodeType@4815} "DATA"
value = {DataStoreService@5527}
{Node$NodeType@5528} "META" -> {MetaStoreService@5529}
key = {Node$NodeType@5528} "META"
value = {MetaStoreService@5529}
connectManagerMap = {HashMap@5532} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionConnectionHandler@5548}
key = {Node$NodeType@5525} "SESSION"
value = {SessionConnectionHandler@5548}
{Node$NodeType@4815} "DATA" -> {DataConnectionHandler@5549}
key = {Node$NodeType@4815} "DATA"
value = {DataConnectionHandler@5549}
{Node$NodeType@5528} "META" -> {MetaConnectionHandler@5550}
key = {Node$NodeType@5528} "META"
value = {MetaConnectionHandler@5550}
nodeServiceMap = {HashMap@5533} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionNodeServiceImpl@5540}
key = {Node$NodeType@5525} "SESSION"
value = {SessionNodeServiceImpl@5540}
{Node$NodeType@4815} "DATA" -> {DataNodeServiceImpl@5541}
key = {Node$NodeType@4815} "DATA"
value = {DataNodeServiceImpl@5541}
{Node$NodeType@5528} "META" -> {MetaNodeServiceImpl@5542}
key = {Node$NodeType@5528} "META"
value = {MetaNodeServiceImpl@5542}
至此,MetaServer的架构和启动介绍完成,我们下文将会介绍基本功能,比如注册,存储,续约等。
服务注册中心 MetaServer 功能介绍和实现剖析 | SOFARegistry 解析
服务注册中心如何实现 DataServer 平滑扩缩容 | SOFARegistry 解析
服务注册中心数据一致性方案分析 | SOFARegistry 解析
服务注册中心如何实现秒级服务上下线通知 | SOFARegistry 解析
服务注册中心 Session 存储策略 | SOFARegistry 解析
服务注册中心数据分片和同步方案详解 | SOFARegistry 解析
服务注册中心 SOFARegistry 解析 | 服务发现优化之路
海量数据下的注册中心 - SOFARegistry 架构介绍
详解蚂蚁金服 SOFAJRaft | 生产级高性能 Java 实现
怎样打造一个分布式数据库——rocksDB, raft, mvcc,本质上是为了解决跨数据中心的复制