Maven配置
添加Avro依赖
org.apache.avro
avro
1.11.1
添加Avro插件
org.apache.avro
avro-maven-plugin
1.11.1
generate-sources
schema
${project.basedir}/src/main/avro/
${project.basedir}/src/main/java/
org.apache.maven.plugins
maven-compiler-plugin
1.8
1.8
使用Avro插件可以自动将Avro Schema转化为Java POJO。
定义Schema
定义Schema文件
注意在 avro 插件的依赖中定义的两个路径:
${project.basedir}/src/main/avro/
${project.basedir}/src/main/java/
该配置的意思是,根据/src/main/avro/下的schema文件,生成对应的类文件到/src/main/java/下,所以我们先创建一个资源文件夹/src/main/avro ,然后再在该资源文件夹下创建 schema 文件,这里定义一个简单的schema文件user.avsc,注意,后缀一定是avsc,其中的内容如下:
namespace: 定义了根据 schema 文件生成的类的包名
生成User类
执行 maven compile 后生成User类。
当然你也可以手动执行,命令如下:
java -jar /path/to/avro-tools-1.11.1.jar compile schema
通过生成代码的方式使用Avro
序列化
package org.example.avro;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.io.DatumWriter;
import org.apache.avro.specific.SpecificDatumWriter;
import java.io.File;
import java.io.IOException;
/**
* @description: 使用 avro 对 org.example.avro.User 类的对象进行序列化
* @author songxt1990
* @date 2023-12-11 14:32
* @version 1.0
*/
public class AvroSerWithCodeGeneration {
public static void main(String[] args) throws IOException {
// Leave favorite color null
User user1 = new User();
user1.setName("Alyssa");
user1.setFavoriteNumber(256);
// Alternate constructor
User user2 = new User("Ben", 7, "red");
// Construct via builder
User user3 = User.newBuilder()
.setName("Charlie")
.setFavoriteColor("blue")
.setFavoriteNumber(null)
.build();
// Serialize user1, user2 and user3 to disk
DatumWriter userDatumWriter = new SpecificDatumWriter(User.class);
DataFileWriter dataFileWriter = new DataFileWriter(userDatumWriter);
dataFileWriter.create(user1.getSchema(), new File("users.avro"));
dataFileWriter.append(user1);
dataFileWriter.append(user2);
dataFileWriter.append(user3);
dataFileWriter.close();
}
}
反序列化
import org.apache.avro.file.DataFileReader;
import org.apache.avro.io.DatumReader;
import org.apache.avro.specific.SpecificDatumReader;
import java.io.File;
import java.io.IOException;
/**
* @description: 使用 avro 对 org.example.avro.User 类的对象进行反序列化
* @author Songxt1990
* @date 2023-12-11 14:40
* @version 1.0
*/
public class AvroDesWithCodeGeneration {
public static void main(String[] args) throws IOException {
// Deserialize Users from disk
DatumReader userDatumReader = new SpecificDatumReader(User.class);
DataFileReader dataFileReader = new DataFileReader(new File("users.avro"), userDatumReader);
User user = null;
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
System.out.println(user);
}
}
}
运行结果如下:
{"name": "Alyssa", "favorite_number": 256, "favorite_color": null}
{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
{"name": "Charlie", "favorite_number": null, "favorite_color": "blue"}
通过不生成代码的方式使用Avro
序列化
/**
* @description: 通过不生成代码的方式使用avro序列化User对象
* @author Songxt1990
* @date 2023-12-11 14:55
* @version 1.0
*/
public class AvroWithoutCodeGeneration {
public static void main(String[] args) throws IOException {
Schema schema = new Schema.Parser().parse(new File("src/main/avro/user.avsc"));
// Leave favorite color null
GenericRecord user1 = new GenericData.Record(schema);
user1.put("name", "Alyssa");
user1.put("favorite_number", 256);
GenericRecord user2 = new GenericData.Record(schema);
user2.put("name", "Ben");
user2.put("favorite_number", 7);
user2.put("favorite_color", "red");
// Serialize user1 and user2 to disk
File file = new File("users.avro");
DatumWriter datumWriter = new GenericDatumWriter(schema);
DataFileWriter dataFileWriter = new DataFileWriter(datumWriter);
dataFileWriter.create(schema, file);
dataFileWriter.append(user1);
dataFileWriter.append(user2);
dataFileWriter.close();
}
}
反序列化
/**
* @description: 通过不生成代码的方式使用avro反序列化User对象
* @author Songxt1990
* @date 2023-12-11 14:55
* @version 1.0
*/
public class AvroDesWithoutCodeGeneration {
public static void main(String[] args) throws IOException {
Schema schema = new Schema.Parser().parse(new File("src/main/avro/user.avsc"));
// Deserialize users from disk
DatumReader datumReader = new GenericDatumReader(schema);
DataFileReader dataFileReader = new DataFileReader(new File("users.avro"), datumReader);
GenericRecord user = null;
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
System.out.println(user);
}
}
}
运行结果如下:
{"name": "Alyssa", "favorite_number": 256, "favorite_color": null}
{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
参考链接
Getting Started (Java) | Apache Avro
领取专属 10元无门槛券
私享最新 技术干货