Questions tagged [avro-tools]
112 questions
27
votes
3 answers
How to extract schema from an avro file in Java
How do you extract first the schema and then the data from an avro file in Java? Identical to this question except in java.
I've seen examples of how to get the schema from an avsc file but not an avro file. What direction should I be looking…
mba12
- 2,702
- 6
- 37
- 56
15
votes
2 answers
Storing null values in avro files
I have some json data that looks like this:
{
"id": 1998983092,
"name": "Test Name 1",
"type": "search string",
"creationDate": "2017-06-06T13:49:15.091+0000",
"lastModificationDate": "2017-06-28T14:53:19.698+0000",
…
mba12
- 2,702
- 6
- 37
- 56
11
votes
4 answers
Is there a way to programmatically convert JSON to AVRO Schema?
I need to create AVRO file but for that I need 2 things:
1) JSON
2) Avro Schema
From these 2 requirements - I have JSON:
{"web-app": {
"servlet": [
{
"servlet-name": "cofaxCDS",
"servlet-class": "org.cofax.cds.CDSServlet",
…
Joe
- 11,983
- 31
- 109
- 183
11
votes
2 answers
Avro Schema format Exception - "record" is not a defined name
I'm trying to use this avro shcema
{
"namespace": "nothing",
"name": "myAvroSchema",
"type": "record",
"fields": [
{
"name": "checkInCustomerReference",
"type": "string"
},
{
"name": "customerContacts",
…
Ignacio Alorre
- 7,307
- 8
- 57
- 94
10
votes
0 answers
How to convert json schema to avro schema
My application has been using json schema (org.everit.json.schema.Schema ) to validate JSON messages whether they comply to a particular format. We are now thinking of moving to the Avro schema. This involves converting previously-stored schema.json…
user2206366
- 461
- 3
- 6
- 17
10
votes
2 answers
Concat Avro files using avro-tools
Im trying to merge avro files into one big file, the problem is concat command does not accept the wildcard
hadoop jar avro-tools.jar concat /input/part* /output/bigfile.avro
I get:
Exception in thread "main" java.io.FileNotFoundException: File…
54l3d
- 3,913
- 4
- 32
- 58
9
votes
2 answers
org.apache.avro.AvroTypeException: Unknown union branch
I'm using this Avro schema:
prices-state.avsc
{
"namespace": "com.company.model",
"name": "Product",
"type": "record",
"fields": [
{
"name": "product_id",
"type": "string"
},
{
…
Francesco Casula
- 26,184
- 15
- 132
- 131
6
votes
1 answer
java.io.IOException Not a data file after converting JSON to Avro with Avro Tools
I have a JSON file and an avro schema file, which correctly describes it's structure.
I then convert the JSON file with the Avro tools into an avro file, without getting an error, like this:
java -jar .\avro-tools-1.7.7.jar fromjson --schema-file…
jasie
- 2,192
- 10
- 39
- 54
6
votes
1 answer
How do you serialize a union field in Avro using Python when attributes match
Say you have this AVDL as a simplified example:
@namespace("example.avro")
protocol User {
record Man {
int age;
}
record Woman {
int age;
}
record User {
union {
Man,
Woman
} user_info;
…
tonicebrian
- 4,715
- 5
- 41
- 65
5
votes
0 answers
Invalid sync error while reading avro file using spark or hive
I have an avro file which is created using JAVA api, when the writer was writing data in file the program shut down ungracefully due to machine reboot.
Now when I am trying to read this file using spark/hive, it reads some data and then throws…
User_qwerty
- 375
- 1
- 2
- 10
5
votes
1 answer
create json representation for decimal logical type and byte types for avro schema
I am trying to create JSON string as per below avro schema, for decimal value.
https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types
{
"name": "score",
"type": "bytes",
"logicalType": "decimal",
"precision": 10,
"scale": 5
}
value…
Kalpesh
- 694
- 2
- 8
- 28
4
votes
0 answers
Resolving Avro schema order when generating sources using avro-maven-plugin
We're using avro-maven-plugin to generate java classes for our Avro schemas, the schemas nest each other to reuse common elements, all fairly vanilla. The piece I can't figure out is how to elegantly make the pom file for this.
To get the schemas to…
Matthew Laird
- 51
- 3
4
votes
0 answers
Splitting an Avro file?
The Avro-Tools package provides an easy way to concatenate multiple avro files together, however there doesn't seem to be an easy way to split files.
Does anyone know of a simple command-line tool that allows one to split an Avro file?
Ryan Stull
- 1,056
- 14
- 35
3
votes
2 answers
Concat Avro files in Google Cloud Storage
I have some big .avro files in the Google Cloud Storage and I want to concat all of them in a single file.
I got
java -jar avro-tools.jar concat
However, as my files are in the google storage path: gs://files.avro I can't concat them by using…
Marcus Sandri
- 65
- 8
3
votes
0 answers
Avro is not able to deserialize Union with Logical Types in fields
Avro Tools Version - 1.9.2
This is the problem that I am facing while deserializing Avro data. Looks like Avro is not able to generate correct POJO class when field is Union with nulls and logical types.
Current Behavior: When avro schema has a…
abb
- 352
- 1
- 10
- 23