Kafka 服务器 PLAINTEXT 和 SASL_PLAINTEXT 配置 及 consumer/producer 脚本连接配置

就是我的经历 stackoverflow question, 分享一下

更多安全配置，参见 https://kafka.apache.org/documentation/#security

• Setup the server to have a PLAINTEXT at port 9092 and SASL_PLAINTEXT at 9093

listeners=PLAINTEXT://localhost:9092, SASL_PLAINTEXT://localhost:9093
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

# we can also specify the sasl config information instead of using the followign cinfig file 'kafka_jaas.conf'
listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_alice="alice-secret";

• Create a file 'kafka_jaas.conf' under the foler 'config' with the following content

sasl_plaintext.KafkaServer{
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_alice="alice-secret";
};

• Modify the kafka-server-start.bat script to set the JVM parameter 'java.security.auth.login.config' by environment JAAS_OPTS

set JAAS_OPTS=-Djava.security.auth.login.config=file:%~dp0../../config/kafka_jaas.conf

1. Modify the kafka-run-class.bat script to create java command with parameter %JAAS_OPTS%

set COMMAND=%JAVA% %KAFKA_HEAP_OPTS% %KAFKA_JVM_PERFORMANCE_OPTS% %KAFKA_JMX_OPTS% %KAFKA_LOG4J_OPTS% %JAAS_OPTS% -cp "%CLASSPATH%" %KAFKA_OPTS% %*

• Then start the zookeeper and kafka-server in different consoles, now the kafka server is setup correctly and running

zookeeper-server-start.bat .\config\zookeeper.properties
kafka-server-start.bat .\config\server.properties

• Next we need to modify the consumer.properties/producer.properties to allow connecting to port 9092 and 9093

bootstrap.servers=localhost:9092, localhost:9093

• Next we are going to make a test with the consumer and producer script with the unsecure port 9092, first let us create a topic 'gaming-events' (you don't have to create the topic for port 9093 again)

kafka-topics.bat --create --topic gaming-events --bootstrap-server localhost:9092

• Then we start a cosumer with the unsecure port 9092, the consumer is waitting message there :-)

kafka-console-consumer.bat --topic gaming-events --from-beginning --bootstrap-server localhost:9092

• Then we start a producer with the unsecure port 9092, we can input some messages and verify the messages are received by cosumer in the previous terminal

kafka-console-producer.bat --topic gaming-events --bootstrap-server localhost:9092

• Next let's play with the secure port 9093, it is more interesting :-), the configuration is easy for the server, but it is a little bit confusing for the client scripts, we can config through the user/password through the .properties file or through the .conf file, but we always NEED to add the following settings in the consumer.properties/producer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

• Then we will firstly config user/password by the .properties file, add the following content in the consumer.properties/producer.properties

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="alice" password="alice-secret";

• Next we start the consumer and producer as below, DO NOT FORGET the option --consumer.config/--producer.config, the consumer.properties/producer.properties will not be read automatically, this is the BIGGEST PIT that confused me a few days, I hate it. Now you are free to get the message flow from producer to consumer :-)

kafka-console-consumer.bat --topic gaming-events --from-beginning --bootstrap-server localhost:9093 --consumer.config .\config\consumer.properties
kafka-console-producer.bat --topic gaming-events --bootstrap-server localhost:9093 --producer.config .\config\producer.properties

• Finally, let's do something more interesting, let us config the user/password through the .conf (of course you can name the file with any suffix as you want) config file, add the following content in the file 'kafka_jaas.conf'

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="alice"
    password="alice-secret";
};

• Then remove the following setting from the consumer.properties/producer.properties file

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="alice"   password="alice-secret";

• Next modify the kafka-console-producer.bat/kafka-console-consumer.bat by setting jvm

set JAAS_OPTS=-Djava.security.auth.login.config=file:%~dp0../../config/kafka_jaas.conf

• Then we start the consumer and producer as usual, DO NOT FORGET the option --consumer.config/--producer.config as always, and the consumer/producer connect to the server in a secure way and you can send/receive safely :-)

kafka-console-consumer.bat --topic gaming-events --from-beginning --bootstrap-server localhost:9093 --consumer.config .\config\consumer.properties
kafka-console-producer.bat --topic gaming-events --bootstrap-server localhost:9093 --producer.config .\config\producer.properties