kafka-eagle是一款不错的kafka监控平台,笔者觉得比kafka-manager可用性要好些,官网地址为www.kafka-eagle.org,下面给出几个使用页面:
官网仅给出了非docker部署方式,但是docker部署已经成为了一种趋势,且能让kafka集群不对外暴露端口的前提下完成对kafka的监控,下面给出kafka-eagle构建流程:
1、下载需要版本的kafka-eagle,笔者使用的是2.0.3版本,对应文件为kafka-eagle-web-2.0.3-bin.tar.gz
2、编写Dockerfile,给出笔者使用的Dockerfile,Dockerfile与kafka-eagle-web-2.0.3-bin.tar.gz在同一目录
from openjdk:8u282-slim
ENV KE_HOME=/opt/kafka-eagle
ENV EAGLE_VERSION=2.0.3
#ADD system-config.properties /tmp
COPY kafka-eagle-web-${EAGLE_VERSION}-bin.tar.gz /tmp
RUN mkdir -p /opt/kafka-eagle/conf
RUN tar zxvf /tmp/kafka-eagle-web-${EAGLE_VERSION}-bin.tar.gz -C /opt/kafka-eagle --strip-components 1 && \
chmod +x /opt/kafka-eagle/bin/ke.sh && \
mkdir -p /hadoop/kafka-eagle/db
EXPOSE 8048 8080
COPY system-config.properties /opt/kafka-eagle/conf/system-config.properties
COPY entrypoint.sh /opt/kafka-eagle/bin/entrypoint.sh
RUN chmod +x /opt/kafka-eagle/bin/entrypoint.sh
WORKDIR /opt/kafka-eagle
CMD ["sh", "/opt/kafka-eagle/bin/entrypoint.sh"]
3、编写entrypoint.sh文件,参考如下:
#!/bin/bash
# 实现通过KAFKA_EAGLE_XX前缀来替代system-config.properties对应配置的功能
echo "" >> "$KE_HOME/conf/system-config.properties"
(
function updateConfig() {
key=$1
value=$2
file=$3
# Omit $value here, in case there is sensitive information
echo "[Configuring] '$key' in '$file'"
# If config exists in file, replace it. Otherwise, append to file.
if grep -E -q "^#?$key=" "$file"; then
sed -r -i "s@^#?$key=.*@$key=$value@g" "$file" #note that no config values may contain an '@' char
else
echo "$key=$value" >> "$file"
fi
}
# Fixes #312
# KAFKA_VERSION + KAFKA_HOME + grep -rohe KAFKA[A-Z0-0_]* /opt/kafka/bin | sort | uniq | tr '\n' '|'
EXCLUSIONS="|KAFKA_VERSION|KAFKA_HOME|KAFKA_DEBUG|KAFKA_GC_LOG_OPTS|KAFKA_HEAP_OPTS|KAFKA_JMX_OPTS|KAFKA_JVM_PERFORMANCE_OPTS|KAFKA_LOG|KAFKA_OPTS|"
# Read in env as a new-line separated array. This handles the case of env variables have spaces and/or carriage returns. See #313
IFS=$'\n'
for VAR in $(env)
do
env_var=$(echo "$VAR" | cut -d= -f1)
if [[ "$EXCLUSIONS" = *"|$env_var|"* ]]; then
echo "Excluding $env_var from broker config"
continue
fi
if [[ $env_var =~ ^KAFKA_EAGLE ]]; then
kafka_name=$(echo "$env_var" | cut -d_ -f3- | tr '[:upper:]' '[:lower:]' | tr _ .)
updateConfig "$kafka_name" "${!env_var}" "$KAFKA_HOME/config/server.properties"
fi
done
)
/opt/kafka-eagle/bin/ke.sh start
tail -f /dev/null
4、启动kafka-eagle以及mysql服务,参考如下:
version: '3'
networks:
cluster_default:
external: true
services:
kafka-eagle:
restart: always
image: repos.jtl3d.com/kafka-eagle:1.0.0
deploy:
placement:
constraints: # 添加条件约束
- node.labels.server==tb
networks:
- cluster_default
ports:
- "8148:8048"
environment:
KAFKA_EAGLE_CLUSTER1_ZK_LIST: zk1:2181,zk2:2181,zk3:2181
KAFKA_EAGLE_KAFKA_EAGLE_URL: jdbc:mysql://mysql:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
KAFKA_EAGLE_KAFKA_EAGLE_USERNAME: root
KAFKA_EAGLE_KAFKA_EAGLE_PASSWORD: 123456
TZ: Asia/Shanghai
mysql:
image: "mysql:5.7"
deploy:
placement:
constraints: # 添加条件约束
- node.labels.server==tb
restart: always
environment:
MYSQL_ROOT_PASSWORD: "123456"
MYSQL_USER: 'root'
MYSQL_PASS: '123456'
networks:
- cluster_default
ports:
- "3306:3306"
PS:
笔者在部署时使用的portainer,将kafka集群与kafka-eagle放在一个网络下,这样就可以通过域名访问的kafka broker以及zookeeper服务。