Step 1. Add the JitPack repository to your build file
Add it in your root settings.gradle at the end of repositories:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
mavenCentral()
maven { url 'https://jitpack.io' }
}
}
Add it in your settings.gradle.kts at the end of repositories:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
mavenCentral()
maven { url = uri("https://jitpack.io") }
}
}
Add to pom.xml
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>
Add it in your build.sbt at the end of resolvers:
resolvers += "jitpack" at "https://jitpack.io"
Add it in your project.clj at the end of repositories:
:repositories [["jitpack" "https://jitpack.io"]]
Step 2. Add the dependency
dependencies {
implementation 'com.github.ryanworsley:avro4s:v1.6.4'
}
dependencies {
implementation("com.github.ryanworsley:avro4s:v1.6.4")
}
<dependency>
<groupId>com.github.ryanworsley</groupId>
<artifactId>avro4s</artifactId>
<version>v1.6.4</version>
</dependency>
libraryDependencies += "com.github.ryanworsley" % "avro4s" % "v1.6.4"
:dependencies [[com.github.ryanworsley/avro4s "v1.6.4"]]
<img src="https://img.shields.io/maven-central/v/com.sksamuel.avro4s/avro4s-core_2.11.svg?label=latest%20release%20for%202.11"/>
<img src="https://img.shields.io/maven-central/v/com.sksamuel.avro4s/avro4s-core_2.12.svg?label=latest%20release%20for%202.12"/>
Avro4s is a schema/class generation and serializing/deserializing library for Avro written in Scala. The objective is to allow seamless use with Scala without the need to to write boilerplate conversions yourself, and without the runtime overhead of reflection. Hence, this is a macro based library and generates code for use with Avro at compile time.
The features of the library are:
Avro4s allows us to generate schemas directly from classes in a totally straightforward way. Let's define some classes.
case class Ingredient(name: String, sugar: Double, fat: Double)
case class Pizza(name: String, ingredients: Seq[Ingredient], vegetarian: Boolean, vegan: Boolean, calories: Int)
Next is to invoke the apply
method of AvroSchema
passing in the top level type. This will return an org.apache.avro.Schema
instance, from which you can output, write to a file etc.
import com.sksamuel.avro4s.AvroSchema
val schema = AvroSchema[Pizza]
Which will output the following schema:
{
"type" : "record",
"name" : "Pizza",
"namespace" : "com.sksamuel.avro4s.json",
"fields" : [ {
"name" : "name",
"type" : "string"
}, {
"name" : "ingredients",
"type" : {
"type" : "array",
"items" : {
"type" : "record",
"name" : "Ingredient",
"fields" : [ {
"name" : "name",
"type" : "string"
}, {
"name" : "sugar",
"type" : "double"
}, {
"name" : "fat",
"type" : "double"
} ]
}
}
}, {
"name" : "vegetarian",
"type" : "boolean"
}, {
"name" : "vegan",
"type" : "boolean"
}, {
"name" : "calories",
"type" : "int"
} ]
}
You can see that the schema generator handles nested case classes, sequences, primitives, etc. For a full list of supported object types, see the table later.
Avro4s supports recursive schemas, but you will have to manually force the SchemaFor
instance, instead of letting it be generated.
case class Recursive(payload: Int, next: Option[Recursive])
implicit val schemaFor = SchemaFor[Recursive]
val schema = AvroSchema[Recursive]
Avro4s allows us to easily serialize case classes using an instance of AvroOutputStream
which we write to, and close, just like you would any regular output stream. An AvroOutputStream
can be created from a File
, Path
, or by wrapping another OutputStream
. When we create one, we specify the type of objects that we will be serializing. Eg, to serialize instances of our Pizza class:
import java.io.File
import com.sksamuel.avro4s.AvroOutputStream
val pepperoni = Pizza("pepperoni", Seq(Ingredient("pepperoni", 12, 4.4), Ingredient("onions", 1, 0.4)), false, false, 98)
val hawaiian = Pizza("hawaiian", Seq(Ingredient("ham", 1.5, 5.6), Ingredient("pineapple", 5.2, 0.2)), false, false, 91)
val os = AvroOutputStream.data[Pizza](new File("pizzas.avro"))
os.write(Seq(pepperoni, hawaiian))
os.flush()
os.close()
With avro4s we can easily deserialize a file back into Scala case classes. Given the pizzas.avro
file we generated in the previous section on serialization, we will read this back in using the AvroInputStream
class. We first create an instance of the input stream specifying the types we will read back, and the file. Then we call iterator which will return a lazy iterator (reads on demand) of the data in the file. In this example, we'll load all data at once from the iterator via toSet
.
import com.sksamuel.avro4s.AvroInputStream
val is = AvroInputStream.data[Pizza](new File("pizzas.avro"))
val pizzas = is.iterator.toSet
is.close()
println(pizzas.mkString("\n"))
Will print out:
Pizza(pepperoni,List(Ingredient(pepperoni,12.2,4.4), Ingredient(onions,1.2,0.4)),false,false,500) Pizza(hawaiian,List(Ingredient(ham,1.5,5.6), Ingredient(pineapple,5.2,0.2)),false,false,500)
You can serialize to Binary using the AvroBinaryOutputStream
which you can create directly or by using AvroOutputStream.binary
.
The difference with binary serialization is that the schema is not included in the output.
Simply
case class Composer(name: String, birthplace: String, compositions: Seq[String])
val ennio = Composer("ennio morricone", "rome", Seq("legend of 1900", "ecstasy of gold"))
val baos = new ByteArrayOutputStream()
val output = AvroOutputStream.binary[Composer](baos)
output.write(ennio)
output.close()
val bytes = baos.toByteArray
You can deserialize Binary using the AvroBinaryInputStream
which you can create directly or by using AvroInputStream.binary
The difference with binary serialization is that the schema is not included in the data.
val in = new ByteArrayInputStream(bytes)
val input = AvroInputStream.binary[Composer](in)
val result = input.iterator.toSeq
result shouldBe Vector(ennio)
You can serialize to JSON using the AvroJsonOutputStream
which you can create directly or by using AvroOutputStream.json
Simply
case class Composer(name: String, birthplace: String, compositions: Seq[String])
val ennio = Composer("ennio morricone", "rome", Seq("legend of 1900", "ecstasy of gold"))
val baos = new ByteArrayOutputStream()
val output = AvroOutputStream.json[Composer](baos)
output.write(ennio)
output.close()
println(baos.toString("UTF-8"))
You can deserialize JSON using the AvroJsonInputStream
which you can create directly or by using AvroInputStream.json
val json = "{\"name\":\"ennio morricone\",\"birthplace\":\"rome\",\"compositions\":[\"legend of 1900\",\"ecstasy of gold\"]}"
val in = new ByteInputStream(json.getBytes("UTF-8"), json.size)
val input = AvroInputStream.json[Composer](in)
val result = input.singleEntity
result shouldBe Success(ennio)
To interface with the Java API it is sometimes desirable to convert between your classes and the Avro GenericRecord
type. You can do this easily in avro4s using the RecordFormat
typeclass (this is what the input/output streams use behind the scenes). Eg,
To convert from a class into a record:
case class Composer(name: String, birthplace: String, compositions: Seq[String])
val ennio = Composer("ennio morricone", "rome", Seq("legend of 1900", "ecstasy of gold"))
val format = RecordFormat[Composer]
// record is of type GenericRecord
val record = format.to(ennio)
And to go from a record back into a type:
// given some record from earlier
val record = ...
val format = RecordFormat[Composer]
// is an instance of Composer
val ennio = format.from(record)
Bring an implicit ScaleAndPrecision
into scope before using AvroSchema
.
import com.sksamuel.avro4s.ScaleAndPrecision
case class MyDecimal(d: BigDecimal)
implicit val sp = ScaleAndPrecision(8, 20)
val schema = AvroSchema[MyDecimal]
{
"type":"record",
"name":"MyDecimal",
"namespace":"$iw",
"fields":[{
"name":"d",
"type":{
"type":"bytes",
"logicalType":"decimal",
"scale":"8",
"precision":"20"
}
}]
}
Avro supports generalised unions, eithers of more than two values. To represent these in scala, we use shapeless.:+:
, such that A :+: B :+: C :+: CNil
represents cases where a type is A
OR B
OR C
. See shapeless' documentation on coproducts for more on how to use coproducts.
Scala sealed traits/classes are supported both when it comes to schema generation and conversions to/from GenericRecord
. Generally sealed hierarchies are encoded as unions - in the same way like Coproducts. Under the hood, shapeless Generic
is used to derive Coproduct representation for sealed hierarchy.
When all descendants of sealed trait/class are singleton objects, optimized, enum-based encoding is used instead.
import scala.collection.{Array, List, Seq, Iterable, Set, Map, Option, Either}
import shapeless.{:+:, CNil}
|Scala Type|Avro Type| |----------|---------| |Boolean|boolean| |Array[Byte]|bytes| |String|string or fixed| |Int|int| |Long|long| |BigDecimal|decimal with default scale 2 and precision 8| |Double|double| |Float|float| |java.util.UUID|string| |Java Enums|enum| |sealed trait T|union| |sealed trait with only case objects|enum| |Array[T]|array| |List[T]|array| |Seq[T]|array| |Iterable[T]|array| |Set[T]|array| |Map[String, T]|map| |Option[T]|union:null,T| |Either[L, R]|union:L,R| |A :+: B :+: C :+: CNil|union:A,B,C| |Option[Either[L, R]]|union:null,L,R| |Option[A :+: B :+: C :+: CNil]|union:null,A,B,C| |T|record|
It is very easy to add custom type mappings. To do this, you need to create instances of ToSchema
, ToValue
and FromValue
typeclasses.
ToSchema
is used to generate an Avro schema for a given JVM type. ToValue
is used to convert an instance of a JVM type into an instance of the Avro type. And FromValue
is used to convert an instance of the Avro type into the JVM type.
For example, to create a mapping for org.joda.time.DateTime
that we wish to store as an ISO Date string, then we can do the following:
implicit object DateTimeToSchema extends ToSchema[DateTime] {
override val schema: Schema = Schema.create(Schema.Type.STRING)
}
implicit object DateTimeToValue extends ToValue[DateTime] {
override def apply(value: DateTime): String = ISODateTimeFormat.dateTime().print(value)
}
implicit object DateTimeFromValue extends FromValue[DateTime] {
override def apply(value: Any, field: Field): DateTime = ISODateTimeFormat.dateTime().parseDateTime(value.toString())
}
These typeclasses must be implicit and in scope when you invoke AvroSchema
or create an AvroInputStream
/AvroOutputStream
.
You can selectively customise the way Avro4s generates certain parts of your hierarchy, thanks to implicit precedence. Suppose you have the following classes:
case class Product(name: String, price: Price, litres: BigDecimal)
case class Price(currency: String, amount: BigDecimal)
And you want to selectively use different scale/precision for the price
and litres
quantities. You can do this by forcing the implicits in the corresponding companion objects.
object Price {
implicit val sp = ScaleAndPrecision(10,2)
implicit val schema = SchemaFor[Price]
}
object Product {
implicit val sp = ScaleAndPrecision(8,4)
implicit val schema = SchemaFor[Product]
}
This will result in a schema where both BigDecimal
quantities have their own separate scale and precision.
compile 'com.sksamuel.avro4s:avro4s-core_2.11:xxx'
libraryDependencies += "com.sksamuel.avro4s" %% "avro4s-core" % "xxx"
<dependency>
<groupId>com.sksamuel.avro4s</groupId>
<artifactId>avro4s-core_2.11</artifactId>
<version>xxx</version>
</dependency>
Check the latest released version on Maven Central
This project is built with SBT. So to build
sbt compile
And to test
sbt test
Contributions to avro4s are always welcome. Good ways to contribute include:
The MIT License (MIT)
Copyright (c) 2015 Stephen Samuel
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.