Simple way to create Scala scripts

This post is a description of a small project idea developed by a friend of mine: Przemysław Pokrywka, I’m just writing down the idea as a blog post.

There are many ways one can execute Scala code, most people use sbt to create a some kind of build, for example fat jar or something similar or just sbt-native-packager to build the application in more native formats.

But what options do you have in case you want to write Scala scripts?

You can use the scala command to execute something, for example like this:

[email protected] ~/ $ cat hello.scala 
println("Hello from scala")

[email protected] ~/ $ scala hello.scala
Hello from scala

This is quite useful when you are using only the standard library, but when your script requires more dependencies you have to figure out how to properly manage them, and very quickly this becomes troublesome

You can achieve similar results by using Ammonite, especially with the “Scala Scripts” extensions. I find this a little bit troublesome, especially because it makes it harder to edit files inside IDE.

The option I’m suggesting allows you to take this one step further:

  • Only single file, that’s both valid Bash and Scala
  • Manage dependencies via coursier
  • IDE support with sbt
  • only Bash and JVM required to run

The whole script is available here: https://github.com/przemek-pokrywka/play-framework-app-in-a-single-file

You can execute it now and you should see following output:

[email protected] ~/ $ bash play-app.scala 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Server started! Please go to http://127.0.0.1:9000/hello/world to see the result

After going to the url you should see this:

Hello world 127.0.0.1:9000

So by just running a simple Bash script we were able to start very simple Play application and accept HTTP requests!

I’m going to go over it step by step and explain everything.

Line 1:

trait valid_both_in_bash_and_in_scala /* 2>/dev/null

First line of the file added because Bash attempts to execute it

Lines 5-7

# Making sure Coursier is available
cr=~/.coursier
test -e $cr/cr || (mkdir $cr && wget -q -O $cr/cr https://git.io/vgvpD && chmod +x $cr/cr)

This downloads coursier library if it’s not available

Lines 9-14

dependencies=(
  com.typesafe.play:play-netty-server_2.11:2.5.0
  com.typesafe.play:play_2.11:2.5.0
  com.lihaoyi:ammonite-repl_2.11.7:0.5.5    # Mandatory for running Scala script
)

This is a normal Bash array that stores Scala dependencies, we’ll be using it later. Note that we are actually using ammonite-repl here to execute the script.

Lines 16-20

# Generate simple build.sbt for editing in IDEs locally. Run with `--build.sbt` (now also on Macs)
test "$1" == "--build.sbt" && \
  printf '%s\n' "${dependencies[@]}" | \
  sed 's/\(.*\):\(.*\):\(.*\)/libraryDependencies += "\1" % "\2" % "\3"/g' > build.sbt && \
  exit

This allows you to generate build.sbt file with libraryDependencies section so that you are able to import and edit the script inside IDE and all dependencies will be resolved correctly.

Lines 22-24

# Small enhancement to the Scalapolis version. Enabling Ammonite to cache compilation output:
just_scala_file=${TMPDIR:-/tmp}/$(basename $0)
(sed -n '/^object script/,$ p' $0; echo "script.run()") > $just_scala_file

Ammonite uses caching mechanism to prevent unnecessary recompilation, also we are saving evertyhing after object script into the temporary file

Line 26

CLASSPATH="$($cr/cr fetch -q -p ${dependencies[*]} )" \

This is where the magic happens: The cr fetch command downloads all dependencies (if they were not downloaded before) listed in the previously defined Bash array. The output of the command is a list of those JARs, this output is captured in the CLASSPATH variable, which we’ll be using in the next step.

Lines 27-31

  java \
    -Dplay.crypto.secret=foo.bar.baz \
    -Dconfig.resource=reference.conf \
    ammonite.repl.Main $just_scala_file # hide Bash part from Ammonite
                                        # and make it run the Scala part

We are executing straightforward Java process (with CLASSPATH from the previous step), we are starting Ammonite REPL feeding it the scala part of the script

Line 33

exit $?

This prevents Bash from processing the rest of the file.

Lines 36-57

This is the actual Scala code that we are running.

Applications

The most immediate application for a script like that is some sort of a quick start guide where by just running one Bash command you can run some Scala code or setup environment for development.
Also it should be suitable for running more advanced Scala CLI scripts that fetch multiple dependencies and the scripts are not intended to be used for a long time (if you happen to have that scenario, you might be better of by generating a fat jar)

Testing Akka Performance

Few weeks ago I attended a workshop called “Understanding Mechanical Sympathy” ran by Martin Thompson. During that workshop we written and tested few concurrent programming techniques and as a first exercise we have written a simple Ping-Pong program:


package uk.co.real_logic;

import static java.lang.System.out;

/*
Original exercise did during "Lock Free Workshop" by Martin Thompson: http://www.real-logic.co.uk/training.html
 */
public final class PingPong {
    private static final int REPETITIONS = 100_000_000;

    private static volatile long pingValue = -1;
    private static volatile long pongValue = -1;

    public static void main(final String[] args) throws Exception {
        final Thread pongThread = new Thread(new PongRunner());
        final Thread pingThread = new Thread(new PingRunner());
        pongThread.setName("pong-thread");
        pongThread.setName("ping-thread");
        pongThread.start();
        pingThread.start();

        final long start = System.nanoTime();

        pingThread.join();
        pongThread.join();

        final long duration = System.nanoTime() - start;

        out.printf("duration %,d (ns)%n", duration);
        out.printf("%,d ns/op%n", duration / (REPETITIONS * 2L));
        out.printf("%,d ops/s%n", (REPETITIONS * 2L * 1_000_000_000L) / duration);
        out.println("pingValue = " + pingValue + ", pongValue = " + pongValue);

        main(args);
    }

    public static class PingRunner implements Runnable {
        public void run() {
            for(int i = 0; i < REPETITIONS; ++i){
                pingValue = i;
                while(i != pongValue){
                }
            }
        }
    }

    public static class PongRunner implements Runnable {
        public void run() {
            for(int i = 0; i < REPETITIONS; ++i) {
                while (i != pingValue) {
                }
                pongValue = i;
            }
        }
    }
}


As you can see we have 2 threads that need to synchronize on the single variable, this is achieved by checking the value of the variable and if it matches expected value, the thread can continue. What’s worth to note is that those 2 threads are busy spinning (inside while loops).

Testing Original Code

After running the code for few minutes the test results stabilize in my case (i7-3610QM) around those values:

duration 10,756,457,024 (ns)
53 ns/op
18,593,482 ops/s

After pinning threads to specific CPU cores the results are slightly better:

duration 10,147,866,952 (ns)
50 ns/op
19,708,575 ops/s

This pretty trivial optimization, helped to gain about 7% of performance gain, so overall not much (I suspect that my Linux system is performing some sort of optimizations at this level as well).

Akka version

After playing around with the pure Java version of the code, I decided to see how I could design a close enough Akka version of this exercise and what optimizations I could use to improve it’s original performance.

This is my code:


case object Ping
case object Pong

object PingPongAkkaApp extends App {
  override def main(args: Array[String]): Unit = {
    val t = new Tester
    t.run()
    main(args)
  }
}

class Tester {
  val REPETITIONS: Int = 100000000

  val startTime: Long = System.nanoTime
  val pongValue: AtomicInteger = new AtomicInteger(0)

  def run() = {
    val system = ActorSystem("PingPongSystem")
    val pongActor = system.actorOf(Props(new PongActor(REPETITIONS, pongValue)), name = "Pong")
    val pingActor = system.actorOf(Props(new PingActor(REPETITIONS)), name = "Ping")

    system.registerOnTermination {
      val duration: Long = System.nanoTime - startTime
      printf("duration %,d (ns)%n", duration)
      printf("%,d ns/op%n", duration / (REPETITIONS * 2L))
      printf("%,d ops/s%n", (REPETITIONS * 2L * 1000000000L) / duration)
      println("pongValue = " + pongValue)
    }

    pongActor.tell(Ping, pingActor)

    Await.result(system.whenTerminated, Duration.Inf)
  }
}

class PingActor(repetitions: Int) extends Actor {

  override def receive = {
    case Pong =>
      sender ! Ping
  }
}

class PongActor(repetitions: Int, pongValue: AtomicInteger) extends Actor {
  var counter: Int = 0

  override def receive = {
    case Ping =>
      counter = counter + 1
      if (counter >= repetitions) {
        pongValue.set(counter)
        context.system.terminate()
      } else {
        sender ! Pong
      }
  }
}

In this case instead of explicitly synchronizing on the variable we use 2 akka actors to send messages between each other. The PingActor just replies with Ping to each Pong it receives, but PongActor stores the counter and can stop the actor system when it reaches expected number of iterations. In this case we also need a little bit more boilerplate code to setup actor system and wait for termination.

Testing Akka Version

Let’s look at performance results after running this code without any additional configuration:

After running test for couple of minutes the results stabilized around those values:


duration 222,328,154,119 (ns)
1,111 ns/op
899,571 ops/s


As you can see it’s roughly 20x slower than Java version, that’s quite bad.

During test execution I noticed that it was using all my CPU cores at 100% which was undesired so I decided to add 2 things:

Setup proper Akka configuration, in this case something like that was enough:

akka {
  actor {
    default-dispatcher {
      type = Dispatcher
      executor = "thread-pool-executor"
      throughput = 1
      fork-join-executor {
        parallelism-min = 2
        parallelism-factor = 0.5
        parallelism-max = 3
      }
    }
  }
}

Additional I’m pinning Java process to use only 2 CPU cores:

taskset -c 1,2 java -server -jar target/scala-2.11/akka-ping-pong-assembly-1.0.jar

This gave much better test results:

duration 60,581,436,734 (ns)
302 ns/op
3,301,341 ops/s

Summary

As you can see the overhead is now 6x compared to Java version which is not bad when taking into account all additional work that happens in Akka to provide all additional features we don’t have when using pure threads.

I’m sure it’s possible to go even further with optimizations and I suspect that this time could be cut by additional 50% given enough experimentation, but I’ll leave as an exercise 🙂

Here is the git repository with full code: https://github.com/wlk/akka-ping-pong

How To Setup Garmin 310XT To Work With Linux

In this post I intent to provide a overview of the steps that need to be performed to setup Garmin 310XT GPS Sports Watch to work with Linux (Ubuntu).

(This tutorial should also apply to other similar Garmin GPS Watches -Garmin Forerunner 60 – 405CX – 310XT – 610 – 910XT)

Install Required Packages

sudo apt-get install python-pip python-qt

sudo pip install pyusb

Install GFrun

GFrun is the program that you can use to download recorded workouts from your watch. This program has it’s own installation script:

wget -N https://github.com/xonel/GFrun/raw/GFrun/GFrun/GFrun.sh && chmod a+x GFrun.sh && sudo bash ./GFrun.sh


But I have discovered that running it as root is not required.

Configure udev Rules

At first you need to plug in your ANT+ stick and run lsusb |grep ANTUSB

In my case this was the result:

Bus 001 Device 031: ID 0fcf:1009 Dynastream Innovations, Inc. ANTUSB-m Stick

Now you just need to create file `/etc/udev/rules.d/51-garmin.rules` and set following content:

ATTRS{idVendor}=="0fcf", ATTRS{idProduct}=="1009", MODE="666"

After that you need to restart udev by running

/etc/init.d/udev restart

And re-plug your ANT+ stick.

Running GFrun

To simply extract workouts from device I only run command:

/home/w/GFrun/GFrun.sh -el

This will download FIT files from the device to

/home/w/GFrun/forerunners/{ID}/activities

The downloaded files are ready to be uploaded anywhere you like (for example Endomondo or Strava – both services accept them without issues).