§Serialization

Out of the box, Lagom will use JSON for request and response message format for the external API of the service, using Play JSON to serialize and deserialize messages. The messages that are sent within the cluster of the service must also be serializable and so must the events that are stored by Persistent Entities. We recommend JSON for these as well and Lagom makes it easy to add Play JSON serialization support to such classes.

Do not depend on Java serialization for production deployments. It is inefficient both in serialization size and speed. It is very difficult to evolve the classes when using Java serialization, which is especially important for the persistent state and events, since you must be able to deserialize old objects that were stored.

Runtime overhead is avoided by not basing the serialization on reflection. Transformations to and from JSON are defined either manually or by using a built in macro - essentially doing what reflection would do, but at compile time instead of during runtime. This comes with one caveat, each top level class that can be serialized needs an explicit serializer defined. To use the Play JSON support in Lagom you need to provide such serializers for each type.

The Play JSON abstraction for serializing and deserializing a class into JSON is the Format which in turn is a combination of Reads and Writes. The library parses JSON into a JSON tree model, which is what the Formats work with.

§Enabling JSON Serialization

To enable JSON Serialization, there are three steps you need to follow.

The first step is to define your Format for each class that is to be serialized. This can be done using automated mapping or manual mapping.

implicit val format: Format[AddPost] = Json.format

Best practice is to define the Format as an implicit in the classes companion object, so that it can be found by implicit resolution.

The second step is to implement JsonSerializerRegistry and have all the service formats returned from its serializers method.

import com.lightbend.lagom.scaladsl.playjson.JsonSerializer
import com.lightbend.lagom.scaladsl.playjson.JsonSerializerRegistry

object MyRegistry extends JsonSerializerRegistry {
  override val serializers = Vector(
    JsonSerializer[AddComment],
    JsonSerializer[AddPost]
  )
}

Having done that, you can provide the serializer registry by overriding the jsonSerializerRegistry component method in your application cake. For example:

import com.lightbend.lagom.scaladsl.server._
import com.lightbend.lagom.scaladsl.cluster.ClusterComponents

abstract class MyApplication(context: LagomApplicationContext)
    extends LagomApplication(context)
    with ClusterComponents {

  override lazy val jsonSerializerRegistry = MyRegistry
}

If you need to use the registry outside of a Lagom application (for example, in tests), this can be done by customising the creation of the actor system. For example:

import akka.actor.ActorSystem
import akka.actor.setup.ActorSystemSetup
import com.lightbend.lagom.scaladsl.playjson.JsonSerializerRegistry

val system = ActorSystem(
  "my-actor-system",
  ActorSystemSetup(
    JsonSerializerRegistry.serializationSetupFor(MyRegistry)
  )
)

§Compression

Compression, as described here, is only used for persistent events, persistent snapshots and remote messages with the service cluster. It is not used for messages that are serialized in the external API of the service.

JSON can be rather verbose and for large messages it can be beneficial to enable compression. That is done by using the JsonSerializer.compressed[T] builder method instead of the JsonSerializer.apply[T] (as shown in the example snippet above):

import com.lightbend.lagom.scaladsl.playjson.JsonSerializer
import com.lightbend.lagom.scaladsl.playjson.JsonSerializerRegistry

object RegistryWithCompression extends JsonSerializerRegistry {
  override val serializers = Vector(
    // 'AddComment' uses `apply[T]()` .
    JsonSerializer[AddComment],
    // The AddPost message is usually rather big, so we want it compressed
    // when it's too large.
    JsonSerializer.compressed[AddPost]
  )
}

The serializer will by default only compress messages that are larger than 1024 bytes. This threshold can be changed with configuration property:

lagom.serialization.json {

  # The serializer will compress the payload if the message class
  # was registered using JsonSerializer.compressed and the payload
  # is larger than this value. Only used for remote messages within
  # the cluster of the service.
  compress-larger-than = 1024b

}

§Automated mapping

The Json.format[MyClass] macro will inspect a case class for what fields it contains and produce a Format that uses the field names and types of the class in the resulting JSON.

The macro allows for defining formats based on the exact structure of the class, which is handy, and avoids spending development time on explicitly defining the format. On the other hand, it tightly couples the structure of the JSON with the structure of the class. This means that refactoring unexpectedly leads to the format being unable to read JSON that was serialized before the change. There are tools in place to deal with this (see schema evolution), but care must be taken.

If the class contains fields of complex types, it pulls those in from implicit marked Formats in the scope. This means that you must provide such implicit formats for all the complex types used inside a class before calling the macro.

case class UserMetadata(twitterHandle: String)
object UserMetadata {
  implicit val format: Format[UserMetadata] = Json.format
}
case class AddComment(userId: String, comment: String, userMetadata: UserMetadata)
object AddComment {
  implicit val format: Format[AddComment] = Json.format
}

§Manual mapping

Defining a Format can be done in several ways using the Play JSON APIs, either using JSON Combinators, or by manually implementing functions that turn a JsValue into a JsSuccess(T) or a JsFailure().

case class AddOrder(productId: String, quantity: Int)

import play.api.libs.functional.syntax._
import play.api.libs.json._

object AddOrder {
  implicit val format: Format[AddOrder] =
    (JsPath \ "product_id")
      .format[String]
      .and((JsPath \ "quantity").format[Int])
      .apply(AddOrder.apply, unlift(AddOrder.unapply))
}

§Special mapping considerations

§Mapping options

The automatic mapping will handle Option fields, for manual mapping of optional fields you can use (JsPath \ "optionalField").formatNullable[A]. This will treat missing fields as None allowing for adding of new fields without providing an explicit schema migration step.

§Mapping singletons

For toplevel singletons (Scala objects) you can use com.lightbend.lagom.scaladsl.playjson.Serializers.emptySingletonFormat to get a Format that outputs empty JSON (as the type is also encoded along side the data).

case object GetOrders {
  implicit val format: Format[GetOrders.type] =
    JsonSerializer.emptySingletonFormat(GetOrders)
}

§Mapping hierarchies

When mapping a hierarchy of types, for example an ADT, or a trait or abstract class you will need to provide a Format for the supertype, that based on some information in the JSON decides which subtype to deserialize.

import play.api.libs.json._

sealed trait Fruit
case object Pear                 extends Fruit
case object Apple                extends Fruit
case class Banana(ripe: Boolean) extends Fruit

object Banana {
  implicit val format: Format[Banana] = Json.format
}

object Fruit {
  implicit val format = Format[Fruit](
    Reads { js =>
      // use the fruitType field to determine how to deserialize
      val fruitType = (JsPath \ "fruitType").read[String].reads(js)
      fruitType.fold(
        errors => JsError("fruitType undefined or incorrect"), {
          case "pear"   => JsSuccess(Pear)
          case "apple"  => JsSuccess(Apple)
          case "banana" => (JsPath \ "data").read[Banana].reads(js)
        }
      )
    },
    Writes {
      case Pear  => JsObject(Seq("fruitType" -> JsString("pear")))
      case Apple => JsObject(Seq("fruitType" -> JsString("apple")))
      case b: Banana =>
        JsObject(
          Seq(
            "fruitType" -> JsString("banana"),
            "data"      -> Banana.format.writes(b)
          )
        )
    }
  )
}

§Schema Evolution

When working on long running projects using Persistence, or any kind of Event Sourcing, schema evolution becomes an important aspect of developing your application. The requirements as well as our own understanding of the business domain may (and will) change over time.

Lagom provides a way to perform transformations of the JSON tree model during deserialization. To do those transformations you can either modify the json imperatively or use the Play JSON transformers.

We will look at a few scenarios, detailing how the classes may be evolved.

§Remove Field

Removing a field can be done without any migration code. Both manual and automatic mappings will ignore properties that do not exist in the class.

§Add Field

Adding an optional field can be done without any migration code if automated mapping is used. You can also add an optional field if manual mapping is used, and you have made sure a missing field is read as a None by your format (see mapping options).

Old class:

case class ItemAdded(shoppingCartId: String, productId: String, quantity: Int)

New class with a new optional discount property:

case class ItemAdded(shoppingCartId: String, productId: String, quantity: Int, discount: Option[BigDecimal])

Let’s say we want to have a mandatory discount property without default value instead:

case class ItemAdded(shoppingCartId: String, productId: String, quantity: Int, discount: Double)

To add a new mandatory field we have to use a JSON migration adding a default value to the JSON.

This is how a migration logic would look like for adding a discount field using imperative code:

class ShopSerializerRegistry extends JsonSerializerRegistry {

  import play.api.libs.json._

  override val serializers = ShopCommands.serializers ++ ShopEvents.serializers

  private val itemAddedMigration = new JsonMigration(2) {
    override def transform(fromVersion: Int, json: JsObject): JsObject = {
      if (fromVersion < 2) {
        json + ("discount" -> JsNumber(0.0d))
      } else {
        json
      }
    }
  }

  override def migrations = Map[String, JsonMigration](
    classOf[ItemAdded].getName -> itemAddedMigration
  )
}

Create a concrete subclass of JsonMigration, handing it the current version of the schema as a parameter. Next, implement the transformation logic on the JsObject in the transform method when an older fromVersion is passed in.

Then provide your JsonMigration together with the classname of the class that it migrates in the migrations map from your JsonSerializerRegistry.

Alternatively you can use the Play JSON transformers API. This API is more concise, but arguably has a much higher learning curve.

class ShopSerializerRegistry extends JsonSerializerRegistry {

  import play.api.libs.json._

  override val serializers = ShopCommands.serializers ++ ShopEvents.serializers

  val addDefaultDiscount = JsPath.json.update((JsPath \ "discount").json.put(JsNumber(0.0d)))

  override def migrations = Map[String, JsonMigration](
    JsonMigrations.transform[ItemAdded](
      immutable.SortedMap(
        1 -> addDefaultDiscount
      )
    )
  )
}

In this case we give the JsonMigrations.transform method the type it is for, and a sorted map of transformations that has happened leading up to the current version of the schema.

§Rename Field

Let’s say that we want to rename the productId field to itemId in the previous example.

case class ItemAdded(shoppingCartId: String, itemId: String, quantity: Int)

The imperative migration code would look like:

private val itemAddedMigration = new JsonMigration(2) {
  override def transform(fromVersion: Int, json: JsObject): JsObject = {
    if (fromVersion < 2) {
      val productId = (JsPath \ "productId").read[JsString].reads(json).get
      json + ("itemId" -> productId) - "productId"
    } else {
      json
    }
  }
}

override def migrations = Map[String, JsonMigration](
  classOf[ItemAdded].getName -> itemAddedMigration
)

And alternatively the transformer based migration:

val productIdToItemId =
  JsPath.json
    .update(
      (JsPath \ "itemId").json.copyFrom((JsPath \ "productId").json.pick)
    )
    .andThen((JsPath \ "productId").json.prune)

override def migrations = Map[String, JsonMigration](
  JsonMigrations.transform[ItemAdded](
    immutable.SortedMap(
      1 -> productIdToItemId
    )
  )
)

§Structural Changes

In a similar way we can do arbitrary structural changes.

Old class:

case class Customer(name: String, street: String, city: String, zipCode: String, country: String)

New classes:

case class Address(street: String, city: String, zipCode: String, country: String)

case class Customer(name: String, address: Address, shippingAddress: Option[Address])

The migration code could look like:

import play.api.libs.json._
import play.api.libs.functional.syntax._

val customerMigration = new JsonMigration(2) {

  // use arbitrary logic to parse an Address
  // out of the old schema
  val readOldAddress: Reads[Address] = {
    (JsPath \ "street")
      .read[String]
      .and(
        (JsPath \ "city").read[String])
      .and(
        (JsPath \ "zipCode").read[String])
      .and(
        (JsPath \ "country").read[String])(Address)
  }

  override def transform(fromVersion: Int, json: JsObject): JsObject = {
    if (fromVersion < 2) {
      val address           = readOldAddress.reads(json).get
      val withoutOldAddress = json - "street" - "city" - "zipCode" - "country"

      // use existing formatter to write the address in the new schema
      withoutOldAddress + ("address" -> Customer.addressFormat.writes(address))
    } else {
      json
    }
  }
}

override def migrations: Map[String, JsonMigration] = Map(
  classOf[Customer].getName -> customerMigration
)

§Rename Class

It is also possible to rename the class. For example, let’s rename OrderAdded to OrderPlaced.

Old class:

case class OrderAdded(shoppingCartId: String)

New class:

case class OrderPlaced(shoppingCartId: String)

The migration code would look like:

override def migrations: Map[String, JsonMigration] = Map(
  JsonMigrations
    .renamed(fromClassName = "com.lightbend.lagom.shop.OrderAdded", inVersion = 2, toClass = classOf[OrderPlaced])
)

When a class has both been renamed and had other changes over time, the name change is added separately (as in the example) and the class name transformations are defined in the migrations map. The Lagom serialization logic will first look for name changes, and then use the changed name to resolve any schema migrations that will be done using the changed name.