0

I'm new to Avro messaging format.

Done so far

  • I've json messages
  • Have created Json's avro schema, and saved that schema in .avsc file in resources folder
  • Created Json message based Avro data structure classes using Avro maven plugin .

Now, Have seen so many examples online, but mostly are using schema registry for avro message schema.

Is there any reference or example where I could create avro message from Data structure (classes) created from Avro maven plugin out of .avsc file ?

Can anyone share smallest example of doing it or any reference online link ? will be really helpful.

Update:

From Avro Data structure classes, here is the schema I've got

public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"group_topics_record\",\"namespace\":\"com.main.avro\",\"fields\":[{\"name\":\"urlkey\",\"type\":\"string\"},{\"name\":\"topic_name\",\"type\":\"string\"}]}");

And for example, I've string json message for which I've created above Avro schema classes, then how can I convert that into avro message (not in file, just a string) ? any code snippet ?

Tim
  • 69
  • 7
  • Code you're looking for is in the duplicate, but maybe you should explain your use case for making your Kafka messages larger than they need to be (including the plaintext schema in every single message is required for deserialization) – OneCricketeer May 13 '21 at 12:46
  • @OneCricketeer, in this code snippet DatumWriter datumWriter = new GenericDatumWriter<>(data.getSchema());, from where getSchema() is coming ? – Tim May 13 '21 at 12:59
  • I didn't write the answer, so not completely sure, but I think the class would be something like `Serializer` – OneCricketeer May 13 '21 at 13:02
  • @OneCricketeer, in this snippet, DataFileReader dataFileReader = new DataFileReader<>(new File("file.avro"), datumReader); , is there any way I could mention string json message instead of File? – Tim May 13 '21 at 13:05
  • I don't understand. That line isn't in the post I linked to – OneCricketeer May 13 '21 at 13:17
  • sorrry, Took it from here, but its reading data from file and creating schema out of that. https://stackoverflow.com/questions/45496786/how-to-extract-schema-from-an-avro-file-in-java, but is there any way I could use string json message directly ? – Tim May 13 '21 at 13:23
  • You're using Kafka and ByteArrays, not Files. If you're referring to the AVSC json file, then that would be compiled into a Java class which has that `getSchema` method you're looking for (since it would be an IndexedRecord subclass). Outside of that, you wouldn't need any other strings – OneCricketeer May 13 '21 at 13:26
  • @OneCricketeer, please check my update in question – Tim May 13 '21 at 13:37
  • Avro is a binary format, not a string. `result = byteArrayOutputStream.toByteArray();` in the linked question does exactly what you want, regarding the purpose of using Avro in Kafka without the registry – OneCricketeer May 13 '21 at 14:30
  • If you're saying your input is a JSON string, and you want Avro from that, then I'd recommend looking at this Confluent class that does exactly that https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/formatter/AvroMessageReader.java#L118 – OneCricketeer May 13 '21 at 14:33

0 Answers0