How To Setup CI Build Pipeline With Travis CI, Heroku and sbt

This post covers all steps that are required to setup a Continuous Integration (CI) build pipeline using Travis CI as a main driver for deploying our Play application written in Scala to Heroku cloud.

Expected end result:

After each commit to the master branch of my Github project, I’d like to run full test suite. Following that, each successful build, should trigger a deployment to Heroku. If the tests fail, application should not be deployed.

I’ll use my own project as an example, but this tutorial is suitable for any other project that is uses sbt as a built as well. I’ll be covering following aspects of the work that needs to be performed:

  1. Deploy Play application to Heroku
  2. Integrate Travis CI with Play app
  3. Setup Travis to deploy to Heroku

Deploy Play application to Heroku

Deploying a Play application (build with Activator or sbt) can be done very simply with the help of sbt-heroku plugin.

Those are the steps I had to perform:

  1. Install heroku toolbelt and login with your heroku account
  2. Run heroku create which will generate new application name for you
  3. Add the sbt-heroku plugin – setup application name from the previous step
  4. Run sbt stage deployHeroku
  5. The application will be deployed in around 1 or 2 minutes
  6. If you go to your app activity log (in my case here: https://dashboard.heroku.com/apps/warm-hamlet-57324/activity) you should see all actions that were performed, including your latest deployment

Integrate Travis CI with Play app

This step is even simpler, it’s enough to login to Travis CI with your Github account, then from the list of discovered applications select the one you are interested in.

Travis CI will do everything else automatically and to get started that’s enough. Travis will get notified by Github on each commit you make, and will run your tests for you.

To allow for more customization and control over the build and test process it’s recommend to add .travis.yml configuration file. This is the simple one to get started.


language: scala

jdk:
  - oraclejdk8

scala:
  - 2.11.8

cache:
  directories:
    - $HOME/.m2/repository
    - $HOME/.sbt
    - $HOME/.ivy2

Setup Travis to deploy to Heroku

Next step is the combination of the work we did so far.

To get started you need your Heroku API key which can be found on your account page.

This API key needs to be configured on Travis CI project configuration page, in my case it is: https://travis-ci.org/wlk/game-arena/settings, you need to setup a new environment variable called HEROKU_API_KEY and set the API key as a value.

Additionally we need to update .travis.yml file to describe deployment steps:

language: scala

jdk:
  - oraclejdk8

scala:
  - 2.11.8

cache:
  directories:
    - $HOME/.m2/repository
    - $HOME/.sbt
    - $HOME/.ivy2

deploy:
  provider: script
  script: sbt stage deployHeroku

As you can see I have decided to configure Travis to run my own deployment script which is sbt stage deployHeroku. It’s exactly the same one I have used when deploying from localhost (this time the Heroku API key is not read from heroku toolbelt, but from the environment variable we configured one step above).

Note: Travis comes with build in Heroku deployment capabilities, but I decided no to use them, because I wanted to be able to reuse the same deployment code for both automated and manual deployments.

Summary

As you can see, setting up a simple CI build pipeline is quite a straightforward thing to do, after that the whole process of testing and deploying will happen automatically, and new version of your app can be live within few minutes after your last commit.

I have been using Travis CI for all my Github projects with good results, but this is the first time I have been deploying application automatically to Heroku, so there is still much more to learn how to do this effectively.

BTW. Did you know that I’m available for hire?

JSON in Play Framework – Advanced Libraries

This is a followup post to my previous one covering JSON in Play framework. I’d like to show how the manual work I did before in trying to make JSON mapping compatible with external API can be done by using 2 small but useful libraries:

play-json-naming

This is a very simple library that can be used to convert from camelCase formatting (the default one that we use in Scala) to snake_case formatting that is common in various different languages (for example PHP or Ruby). There is a vast amount of APIs that follow that convention, examples include GitHub or Twitter APIs.

This is how you can use it:


import com.github.tototoshi.play.json.JsonNaming

implicit val playerWrites = JsonNaming.snakecase(Json.writes[Player])

This is enough to provide a mapping from a case class like this:


case class Player(
  id: Int,
  name: String,
  status: String,
  version: String,
  stack: Int,
  bet: Int,
  holeCards: List[Card])

to a JSON in the following format:


{
  "id": 1,
  "name": "wojtek test",
  "status": "active",
  "version": "test version",
  "stack": 3,
  "bet": 32,
  "hole_cards": [
    {
      "rank": "A",
      "suit": "spades"
    },
    {
      "rank": "10",
      "suit": "spades"
    }
  ]
}

As you can see holeCards was replaced with hole_cards. That’s pretty much all that library does.

This is the project page: https://github.com/tototoshi/play-json-naming. Over there you can find information how to include it in your project.

This is the commit I did to introduce the change in my own project, as you can see I have removed a bunch of type-unsafe code, and replaced it with 2 method calls.

play-json extensions

This library is definitely more comprehensive than previous one, it consists of few (de)serializers which can be used in your project. It’s main feature is case class serializer that can handle more than 22 fields in one class (common limitation in Scala), but I’ll show one other feature I had to use to make my API compatible with external service.

Unfortunately it’s documentation isn’t really comprehensive and there are few mistakes (probably related to recent migration to new package name).

I’d like to show you one particular use case in which I was able to use it, namely: “De-/Serialize single value classes”

The problem is that in a setup like this:


sealed abstract class Suit
case object Clubs extends Suit
case object Spades extends Suit
case object Hearts extends Suit
case object Diamonds extends Suit

sealed abstract class Rank
case object Two extends Rank 
case object Three extends Rank
...

case class Card(rank: Rank, suit: Suit)

...
implicit val suitFormat = Json.format[Suit]

implicit val rankFormat = Json.format[Rank]

implicit val cardFormat = Json.format[Card]

The resulting JSON for class Card will look like this:


{
  "suit": {
    "suit": "Spades"
  },
  "rank": {
    "rank": "A"
  }
}

And our expected format is as follows (scroll up to see complete example):


{
  "rank": "A",
  "suit": "Spades"
}

In the previous post I have shown how this can be done by writing manual mapping from case class to JsValue (JsString in this case).

The same result can be achieved with the help of play-json-extensions in the following way:


import ai.x.play.json.Jsonx

implicit val suitFormat = Jsonx.formatInline[Suit]
implicit val rankFormat = Jsonx.formatInline[Rank]
implicit val cardFormat = Json.format[Card]

One note though: There is a problem in the library itself and I had to make to my domain objects definitions, the Suit was a sealed abstract class, now it has to be just sealed class. In my opinion this is a small workaround that is acceptable in my scenario, and hopefully the library will be fixed/extended to allow for this case.

Take a look at the commit I did to introduce the change, as you can see I have removed a bunch of lines which were responsible for manual JSON mappings.

This is the library page: https://github.com/xdotai/play-json-extensions

Summary

I have show how you can use 2 more advanced Play JSON libraries to achieve your goal of making a compatible APIs. This is a more reasonable approach than writing mappings yourself and also it should be less error prone.

JSON in Play Framework – Techniques For Making Compatible Mappings

I’ll show 2 slightly advanced techniques for working with JSON in Play Framework (play-json) that are useful especially when you need to control the mappings yourself. For example when you have to make sure that your API is compatible with existing applications. The examples are based on my project Game Arena (which is in very early stages of development)

One suggestion, before we start, take a look at Play Framework JSON documentation which is truly quite comprehensive and provides a very good introduction to JSON usage in Play.

Even in a very simple project I managed to use few methods for JSON parsing and this post is primary written as a way for me to show how you can work with play-json in the context of Scala case classes.

JSON in Play Framework

Play comes with it’s own JSON parsing library called play-json. Even though it’s sources (github) are in the same repository as the framework itself, it’s published to the Maven repository so you can reuse this knowledge and library in the non-Play projects. Whether or not it’s a good idea is out of the scope for today.

Working With play-json – Basics

I usually put all my JSON mappings into separate Trait. In my case JsonMarshalling and then I extend it in all my Controllers, like that:

@Singleton
class DeckController @Inject() extends Controller with JsonMarshalling {
  def shuffled = Action {
    Ok(Json.toJson(Deck.fullDeck.shuffle))
  }
}

This allows for a very clear separation of concerns, case classes are and don’t have to be aware of their mappings. What’s more I could also provide different JSON mappings in different traits (for example to support multiple API versions).

Case classes

The simplest use case for JSON mapping are case classes (this is true for almost all Scala JSON libraries), in case of play-json you can use a macro to provide mapping for you. For case classes it’s enough to write:

case class User(id: Int, name: String)

implicit val userFormat = Json.format[User]

This will provide mapping to and from JSON. This use case is quite obvious. We automatically will map all fields from a case class to a JSON object. The fields will be called exactly the same.

This is how sample User looks when mapped to JSON:


{
 "id": "1337",
 "name": "Wojtek"
}

Play-json: Slightly Advanced Techniques

Case classes with custom mapping

In case of my project there is a (small) problem. I’m designing my API to be fully compatible with the existing applications, which already use defined field names for all JSON properties. I want to keep using Scala name conventions in my case classes, but for the JSON format I want to follow what is required.

This means that I have to write custom mapping function, from object to JSON, those functions are called Writes[T].

This is an example:

GameState class declaration

case class GameState(
  tournamentId: String,
  gameId: String,
  round: Int,
  betIndex: Int,
  smallBlind: Int,
  currentBuyIn: Int,
  pot: Int,
  minimumRaise: Int,
  dealer: Int,
  orbits: Int,
  inAction: Int,
  players: List[Player],
  communityCards: List[Card]
)

As you can see, I’m able to follow Scala naming conventions here.

Now, the JSON formatting:


 implicit val gameStateWrites = new Writes[GameState] {
   def writes(g: GameState) = Json.obj(
     "tournament_id" -> g.tournamentId,
     "game_id" -> g.gameId,
     "round" -> g.round,
     "bet_index" -> g.betIndex,
     "small_blind" -> g.smallBlind,
     "current_buy_in" -> g.currentBuyIn,
     "pot" -> g.pot,
     "minimum_raise" -> g.minimumRaise,
     "dealer" -> g.dealer,
     "orbits" -> g.orbits,
     "in_action" -> g.inAction,
     "players" -> g.players, //intellij complains here, but it's correct and project compiles OK
     "community_cards" -> g.communityCards
   )
 }

This is a longer snippet and let’s more closely. My writes function maps GameState to JsObject. JsObject is basically a mapping from String to JsValue (which can also contain additional JsObjects). This allows me to change output style from Scala’s camel case format to underscore. It’s actually very simple 1 to 1 mapping, but it’s good technique to use when you want to rename field names when outputting JSON.

If you are interested in having all that work done by a library, take a look at play-json-naming (In case API you are communicating with is requiring snake case formatting)

Mapping Case Classes as JsValues

Consider following snippet and case class at the end:


sealed abstract class Suit
case object Clubs extends Suit
case object Spades extends Suit
case object Hearts extends Suit
case object Diamonds extends Suit

sealed abstract class Rank
case object Two extends Rank 
case object Three extends Rank
...

case class Card(rank: Rank, suit: Suit)

The problem is that if we were using default mapping for case classes in case of Card class we would end up with output like this:


{
  "suit": {
    "suit": "Spades"
  },
  "rank": {
    "rank": "Ace"
  }
}

This is not very convenient because our expected format is following one:


{
  "rank": "Ace",
  "suit": "Spades"
}

The first one is why we want to write a custom writes function (I’ll show how this can be done for Suits, and Ranks are done very similarly):

  implicit val suitFormat = new Format[Suit] {
    def writes(s: Suit) = JsString(s.toString)

    def reads(json: JsValue) = json match {
      case JsNull => JsError()
      case _ => JsSuccess(Suit(json.as[String]))
    }
  }

This custom writes function makes a mapping from Suit to JsString (line 2). It prevents additional layer of wrapping into objects (simply said, it prevents from adding additional brackets)

Note: I have shown an example how you might want to do this yourself, but in the real project a play-json-extensions library is probably a better choice.

Summary and Followup

I was able to show two play-json techniques that are very useful when dealing with APIs where you need to keep compatibility and cannot rely on default JSON mappings in Play Framework

The followup to this post is here – I show how to remove the code I wrote manually and replace it with 2 clever libraries (thanks to Mikołaj Wielocha for suggesting this approach)

BTW. Did you know that I’m available for hire?