Scala Patterns To Avoid: Implicit Arguments With Default Values

There is a tendency for the Scala projects to prefer more explicit programming style. The biggest aspect of that is in my opinion the type system of the Scala language, programmers often start writing their functions by defining types of the arguments and type of the result, only to write the body of the function as last step. That’s also because we have the Scala compiler to help us.

I recently stumbled on a snippet that contradicts this rule and can be a source of hard to spot bugs for the person unfamiliar with the code.

The problem happens when you try to mix both features of the Scala language that make sense if used on their own (although I’ll argue with that a little in a moment), but when used together they create a very serious problem.

The features (as you probably guessed) are:

  • Implicit arguments
  • Default function arguments values

Implicit arguments

Implicit arguments when used in separation make a lot of sense, they allow for context-like arguments to be passed through the whole call stack. Furthermore they are the major building block for more advanced features like typeclasses.
Many major libraries or projects would not exist without it, but because this is a very powerful feature, it’s use should be limited.

Default function arguments

In my option this feature should be avoided almost everywhere – they make it very hard to use functions as arguments and pass function around.
It’s better to use either currying (multiple parameter lists) or just define function few times taking different set of parameters in each case.

I think it makes sense to use default arguments in 2 cases:

  • Backwards compatibility – you have an existing case class and want do add another attribute to it without changing your code or underlying database structures
  • Complicated, builder-like constructors – you have constructor that takes a ton of arguments, and users would like to configure only a limited set each time, using defaults for everything else

The problematic code

Here’s a snippet of the code I stumbled on:

case class User(name: String)

object User {
  val Default = User(name = `unknown`)
}

def updateCampaign(c: Campaign)(implicit user: User = User.Default) {
  // save updated campaign
  // add audit log entry indicating `user` as a person performing action
}

Here’s what I think is wrong here:

  • Caller is not aware that updateCampaign actually takes any argument unless he/she reads the source or looks up documentation (if there’s any).
    You can perfectly well write code that looks like this:
val user = User(`test`)
val campaign = Campaign(...)
updateCampaign(campaign)

And you will not see anything wrong – the compiler won’t complain. Unless the caller is aware there’s another parameter expected, there’s no straightforward way to learn it.

  • If someone overrides updateCampaign function this information will be lost forever, all calls to the overridden version will use the default argument.

The above snippet contradicts the explicitness rule, also Scala compiler will not help you spotting a bug.

Of course the code will still run – the only problem is that every update to the campaign will be attributed to User.Default which is not what we are expecting – but there’s no way to express that intend when using default arguments.

Fixing the code

To fix it, simply remove the default value for the user argument:

def updateCampaign(c: Campaign)(implicit user: User) {
  // save updated campaign
  // add audit log entry indicating `user` as a person performing action
}

After that change you will get the compilation error forcing you to fix the issue.

Scala compiler is now able to detect this issue right away, you don’t have to remember which functions take which arguments because you can leverage the compiler.

Summary

The outlined scenario is one of many examples where a programmer can use Scala compiler to his/her benefit. A simple change to the code will result in a compiler pointing out the problem right away, forcing you to address it before code is released.

It’s also one of the many logic-related mistakes that are very hard to spot in testing – the “audit logging” in this case is a side effect to the regular responsibility of the code.

Simple way to create Scala scripts

This post is a description of a small project idea developed by a friend of mine: Przemysław Pokrywka, I’m just writing down the idea as a blog post.

There are many ways one can execute Scala code, most people use sbt to create a some kind of build, for example fat jar or something similar or just sbt-native-packager to build the application in more native formats.

But what options do you have in case you want to write Scala scripts?

You can use the scala command to execute something, for example like this:

[email protected] ~/ $ cat hello.scala 
println("Hello from scala")

[email protected] ~/ $ scala hello.scala
Hello from scala

This is quite useful when you are using only the standard library, but when your script requires more dependencies you have to figure out how to properly manage them, and very quickly this becomes troublesome

You can achieve similar results by using Ammonite, especially with the “Scala Scripts” extensions. I find this a little bit troublesome, especially because it makes it harder to edit files inside IDE.

The option I’m suggesting allows you to take this one step further:

  • Only single file, that’s both valid Bash and Scala
  • Manage dependencies via coursier
  • IDE support with sbt
  • only Bash and JVM required to run

The whole script is available here: https://github.com/przemek-pokrywka/play-framework-app-in-a-single-file

You can execute it now and you should see following output:

[email protected] ~/ $ bash play-app.scala 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Server started! Please go to http://127.0.0.1:9000/hello/world to see the result

After going to the url you should see this:

Hello world 127.0.0.1:9000

So by just running a simple Bash script we were able to start very simple Play application and accept HTTP requests!

I’m going to go over it step by step and explain everything.

Line 1:

trait valid_both_in_bash_and_in_scala /* 2>/dev/null

First line of the file added because Bash attempts to execute it

Lines 5-7

# Making sure Coursier is available
cr=~/.coursier
test -e $cr/cr || (mkdir $cr && wget -q -O $cr/cr https://git.io/vgvpD && chmod +x $cr/cr)

This downloads coursier library if it’s not available

Lines 9-14

dependencies=(
  com.typesafe.play:play-netty-server_2.11:2.5.0
  com.typesafe.play:play_2.11:2.5.0
  com.lihaoyi:ammonite-repl_2.11.7:0.5.5    # Mandatory for running Scala script
)

This is a normal Bash array that stores Scala dependencies, we’ll be using it later. Note that we are actually using ammonite-repl here to execute the script.

Lines 16-20

# Generate simple build.sbt for editing in IDEs locally. Run with `--build.sbt` (now also on Macs)
test "$1" == "--build.sbt" && \
  printf '%s\n' "${dependencies[@]}" | \
  sed 's/\(.*\):\(.*\):\(.*\)/libraryDependencies += "\1" % "\2" % "\3"/g' > build.sbt && \
  exit

This allows you to generate build.sbt file with libraryDependencies section so that you are able to import and edit the script inside IDE and all dependencies will be resolved correctly.

Lines 22-24

# Small enhancement to the Scalapolis version. Enabling Ammonite to cache compilation output:
just_scala_file=${TMPDIR:-/tmp}/$(basename $0)
(sed -n '/^object script/,$ p' $0; echo "script.run()") > $just_scala_file

Ammonite uses caching mechanism to prevent unnecessary recompilation, also we are saving evertyhing after object script into the temporary file

Line 26

CLASSPATH="$($cr/cr fetch -q -p ${dependencies[*]} )" \

This is where the magic happens: The cr fetch command downloads all dependencies (if they were not downloaded before) listed in the previously defined Bash array. The output of the command is a list of those JARs, this output is captured in the CLASSPATH variable, which we’ll be using in the next step.

Lines 27-31

  java \
    -Dplay.crypto.secret=foo.bar.baz \
    -Dconfig.resource=reference.conf \
    ammonite.repl.Main $just_scala_file # hide Bash part from Ammonite
                                        # and make it run the Scala part

We are executing straightforward Java process (with CLASSPATH from the previous step), we are starting Ammonite REPL feeding it the scala part of the script

Line 33

exit $?

This prevents Bash from processing the rest of the file.

Lines 36-57

This is the actual Scala code that we are running.

Applications

The most immediate application for a script like that is some sort of a quick start guide where by just running one Bash command you can run some Scala code or setup environment for development.
Also it should be suitable for running more advanced Scala CLI scripts that fetch multiple dependencies and the scripts are not intended to be used for a long time (if you happen to have that scenario, you might be better of by generating a fat jar)

Testing Akka Performance

Few weeks ago I attended a workshop called “Understanding Mechanical Sympathy” ran by Martin Thompson. During that workshop we written and tested few concurrent programming techniques and as a first exercise we have written a simple Ping-Pong program:

package uk.co.real_logic;

import static java.lang.System.out;

/*
Original exercise did during "Lock Free Workshop" by Martin Thompson: http://www.real-logic.co.uk/training.html
 */
public final class PingPong {
    private static final int REPETITIONS = 100_000_000;

    private static volatile long pingValue = -1;
    private static volatile long pongValue = -1;

    public static void main(final String[] args) throws Exception {
        final Thread pongThread = new Thread(new PongRunner());
        final Thread pingThread = new Thread(new PingRunner());
        pongThread.setName("pong-thread");
        pongThread.setName("ping-thread");
        pongThread.start();
        pingThread.start();

        final long start = System.nanoTime();

        pingThread.join();
        pongThread.join();

        final long duration = System.nanoTime() - start;

        out.printf("duration %,d (ns)%n", duration);
        out.printf("%,d ns/op%n", duration / (REPETITIONS * 2L));
        out.printf("%,d ops/s%n", (REPETITIONS * 2L * 1_000_000_000L) / duration);
        out.println("pingValue = " + pingValue + ", pongValue = " + pongValue);

        main(args);
    }

    public static class PingRunner implements Runnable {
        public void run() {
            for(int i = 0; i < REPETITIONS; ++i){
                pingValue = i;
                while(i != pongValue){
                }
            }
        }
    }

    public static class PongRunner implements Runnable {
        public void run() {
            for(int i = 0; i < REPETITIONS; ++i) {
                while (i != pingValue) {
                }
                pongValue = i;
            }
        }
    }
}


As you can see we have 2 threads that need to synchronize on the single variable, this is achieved by checking the value of the variable and if it matches expected value, the thread can continue. What’s worth to note is that those 2 threads are busy spinning (inside while loops).

Testing Original Code

After running the code for few minutes the test results stabilize in my case (i7-3610QM) around those values:

duration 10,756,457,024 (ns)
53 ns/op
18,593,482 ops/s

After pinning threads to specific CPU cores the results are slightly better:

duration 10,147,866,952 (ns)
50 ns/op
19,708,575 ops/s

This pretty trivial optimization, helped to gain about 7% of performance gain, so overall not much (I suspect that my Linux system is performing some sort of optimizations at this level as well).

Akka version

After playing around with the pure Java version of the code, I decided to see how I could design a close enough Akka version of this exercise and what optimizations I could use to improve it’s original performance.

This is my code:

case object Ping
case object Pong

object PingPongAkkaApp extends App {
  override def main(args: Array[String]): Unit = {
    val t = new Tester
    t.run()
    main(args)
  }
}

class Tester {
  val REPETITIONS: Int = 100000000

  val startTime: Long = System.nanoTime
  val pongValue: AtomicInteger = new AtomicInteger(0)

  def run() = {
    val system = ActorSystem("PingPongSystem")
    val pongActor = system.actorOf(Props(new PongActor(REPETITIONS, pongValue)), name = "Pong")
    val pingActor = system.actorOf(Props(new PingActor(REPETITIONS)), name = "Ping")

    system.registerOnTermination {
      val duration: Long = System.nanoTime - startTime
      printf("duration %,d (ns)%n", duration)
      printf("%,d ns/op%n", duration / (REPETITIONS * 2L))
      printf("%,d ops/s%n", (REPETITIONS * 2L * 1000000000L) / duration)
      println("pongValue = " + pongValue)
    }

    pongActor.tell(Ping, pingActor)

    Await.result(system.whenTerminated, Duration.Inf)
  }
}

class PingActor(repetitions: Int) extends Actor {

  override def receive = {
    case Pong =>
      sender ! Ping
  }
}

class PongActor(repetitions: Int, pongValue: AtomicInteger) extends Actor {
  var counter: Int = 0

  override def receive = {
    case Ping =>
      counter = counter + 1
      if (counter >= repetitions) {
        pongValue.set(counter)
        context.system.terminate()
      } else {
        sender ! Pong
      }
  }
}

In this case instead of explicitly synchronizing on the variable we use 2 akka actors to send messages between each other. The PingActor just replies with Ping to each Pong it receives, but PongActor stores the counter and can stop the actor system when it reaches expected number of iterations. In this case we also need a little bit more boilerplate code to setup actor system and wait for termination.

Testing Akka Version

Let’s look at performance results after running this code without any additional configuration:

After running test for couple of minutes the results stabilized around those values:

duration 222,328,154,119 (ns)
1,111 ns/op
899,571 ops/s


As you can see it’s roughly 20x slower than Java version, that’s quite bad.

During test execution I noticed that it was using all my CPU cores at 100% which was undesired so I decided to add 2 things:

Setup proper Akka configuration, in this case something like that was enough:

akka {
  actor {
    default-dispatcher {
      type = Dispatcher
      executor = "thread-pool-executor"
      throughput = 1
      fork-join-executor {
        parallelism-min = 2
        parallelism-factor = 0.5
        parallelism-max = 3
      }
    }
  }
}

Additional I’m pinning Java process to use only 2 CPU cores:

taskset -c 1,2 java -server -jar target/scala-2.11/akka-ping-pong-assembly-1.0.jar

This gave much better test results:

duration 60,581,436,734 (ns)
302 ns/op
3,301,341 ops/s

Summary

As you can see the overhead is now 6x compared to Java version which is not bad when taking into account all additional work that happens in Akka to provide all additional features we don’t have when using pure threads.

I’m sure it’s possible to go even further with optimizations and I suspect that this time could be cut by additional 50% given enough experimentation, but I’ll leave as an exercise 🙂

Here is the git repository with full code: https://github.com/wlk/akka-ping-pong