Just today I’ve got a request to review changes to introduce Jekyll as the documentation framework. I was earlier proposing it myself so I knew what the outcome of the review could be - APPROVED.

But…

I also knew that the guy who proposed the change was fighting the installation of Ruby gems and Jekyll to have a complete working environment for the documentation system on his own laptop. He was on Linux while I’m on Mac OS. He finally got it sorted out, but the final solution was not satisfactory to me - he installed additional dependencies onto his local machine directly and suggested the very same steps in README so others could follow his steps. I simply couldn’t approve it. Sorry.

The alternative was to use Docker and the docker-jekyll image. It makes Jekyll running in a self-contained container with no system-wide interaction with the local machine. And it’s officially supported and maintained by the Jekyll project itself.

Using Jekyll inside Docker

The steps to use Jekyll without installing Ruby, gems and other dependencies locally is to use the docker-jekyll image. It’s very safe because it’s a self-contained image and I’m not “infecting” my local working machine in any way (except installing Docker and pulling the image, but that’s acceptable).

Install Docker on your platform and do the following.

Go to the docs directory where your documentation lives and execute:

docker run --rm -v $(pwd):/srv/jekyll -t -p 4000:4000 jekyll/stable jekyll build

It will generate the docs from the sources and save the output into the current working directory (under _site).

Serve the docs using jekyll s which is the default command while spinning up a new docker-jekyll container.

docker run --rm -v $(pwd):/srv/jekyll -t --name=jekyll -p 4000:4000 jekyll/stable

Mind the name for the container - jekyll - so it’s easier to work with it later on.

➜  docs git:(39fd9c9) ✗ docker run --rm --volume=$(pwd):/srv/jekyll -t --name=jekyll -p 4000:4000 jekyll/stable
Configuration file: /srv/jekyll/_config.yml
            Source: /srv/jekyll
       Destination: /srv/jekyll/_site
      Generating...
                    done.
 Auto-regeneration: enabled for '/srv/jekyll'
Configuration file: /srv/jekyll/_config.yml
    Server address: http://0.0.0.0:4000/
  Server running... press ctrl-c to stop.

Open http://0.0.0.0:4000/ and have fun!

Stop the container using docker stop jekyll.

➜  docs git:(39fd9c9) ✗ docker ps
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                    NAMES
b6bb07d8b8aa        jekyll/stable       "/usr/bin/startup"   26 seconds ago      Up 26 seconds       0.0.0.0:4000->4000/tcp   jekyll
➜  docs git:(39fd9c9) ✗ docker stop jekyll
jekyll
➜  docs git:(39fd9c9) ✗ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
➜  docs git:(39fd9c9) ✗

Happy Dockering!

Caveats on Mac OS (and possibly on Windows)

You are using Docker Machine to work with Docker. And so the IP address for Jekyll’s generated website is given by docker-machine ip dev where dev is the name of the Docker machine instance.

Also, you may face issues with not regenerating docs after your changes even when jekyll serve alone is supposed to work fine. It seems it does not with Docker on Mac OS. The workaround is to use jekyll serve --force_polling. The complete command line to have Jekyll running inside a Docker container with your changes being picked up is as follows:

docker run --rm -v $(pwd):/srv/jekyll -t --name=jekyll -p 4000:4000 jekyll/stable jekyll serve --force_polling

See https://github.com/jekyll/jekyll/issues/2926.

Let me know what you think about the topic of the blog post in the Comments section below or contact me at jacek@japila.pl. Follow the author as @jaceklaskowski on Twitter, too.

I didn’t know deploying Scala web applications might be so easy until the very recent Warsaw Scala Enthusiasts meetup when Rafal Krzewski introduced me to one of the two sbt plugins for Docker - sbt-native-packager (the other is sbt-docker that they say is even better).

The blog post shows how easy it is to use Docker as a means of deploying Scala web applications using Play Framework (that actually uses sbt-native-packager under the covers).

Creating Play Framework web application

Create a new web application using Typesafe Activator tool using activator new command:

➜  docker-playground  activator new play-dockerized play-scala

Fetching the latest list of templates...

OK, application "play-dockerized" is being created using the "play-scala" template.

To run "play-dockerized" from the command line, "cd play-dockerized" then:
/Users/jacek/dev/sandbox/docker-playground/play-dockerized/activator run

To run the test for "play-dockerized" from the command line, "cd play-dockerized" then:
/Users/jacek/dev/sandbox/docker-playground/play-dockerized/activator test

To run the Activator UI for "play-dockerized" from the command line, "cd play-dockerized" then:
/Users/jacek/dev/sandbox/docker-playground/play-dockerized/activator ui

cd the play-dockerized directory and execute sbt run to start the application:

➜  play-dockerized  sbt run
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Updating {file:/Users/jacek/.sbt/0.13/plugins/}global-plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /Users/jacek/dev/sandbox/docker-playground/play-dockerized/project
[info] Updating {file:/Users/jacek/dev/sandbox/docker-playground/play-dockerized/project/}play-dockerized-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to play-dockerized (in build file:/Users/jacek/dev/sandbox/docker-playground/play-dockerized/)
[info] Updating {file:/Users/jacek/dev/sandbox/docker-playground/play-dockerized/}root...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.

--- (Running the application, auto-reloading is enabled) ---

[info] p.a.l.c.ActorSystemProvider - Starting application default Akka system: application
[info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Ctrl+D to stop and go back to the console...)

You should now be able to access http://localhost:9000. It’s a vanilla Play Framework 2.4.2 web application.

Publishing Docker image - docker:publishLocal

Play comes with sbt-native-packager plugin so stop the previous command (using Ctrl+D) and execute sbt docker:publishLocal:

➜  play-dockerized  sbt docker:publishLocal
...
[info] Digest: sha256:66638b21de4b16af589f54cbd3e2698919efd529583b732a593613f35e813f0b
[info] Status: Downloaded newer image for java:latest
[info]  ---> 49ebfec495e1
[info] Step 1 : WORKDIR /opt/docker
[info]  ---> Running in ac01dbacaf66
[info]  ---> 271ea5c0bd1e
[info] Removing intermediate container ac01dbacaf66
[info] Step 2 : ADD opt /opt
[info]  ---> 9c423c2d2f0c
[info] Removing intermediate container 3087077a2680
[info] Step 3 : RUN chown -R daemon:daemon .
[info]  ---> Running in bd40460a5e7d
[info]  ---> aeec9392fc83
[info] Removing intermediate container bd40460a5e7d
[info] Step 4 : USER daemon
[info]  ---> Running in 461907ca0474
[info]  ---> 4f0ad20b6a7f
[info] Removing intermediate container 461907ca0474
[info] Step 5 : ENTRYPOINT bin/play-dockerized
[info]  ---> Running in 09aa91f09bc5
[info]  ---> 7f2afe7c4918
[info] Removing intermediate container 09aa91f09bc5
[info] Step 6 : CMD
[info]  ---> Running in 99c12a3604a3
[info]  ---> 617942a5bc6f
[info] Removing intermediate container 99c12a3604a3
[info] Successfully built 617942a5bc6f
[info] Built image play-dockerized:1.0-SNAPSHOT
[success] Total time: 101 s, completed Jul 23, 2015 8:31:18 AM

That was the exact moment when I realized how clever the sbt-native-packager plugin is to leverage well-known publishLocal task to publish to Docker repository (that’s merely scoped to docker to change the way it works).

You’ve just created a brand new Docker image play-dockerized:1.0-SNAPSHOT. Use docker images command to check it out:

➜  play-dockerized  docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
play-dockerized     1.0-SNAPSHOT        617942a5bc6f        2 minutes ago       892.7 MB

Docker my time!

You can start a container off the play-dockerized image using docker run command:

➜  play-dockerized  docker run --name play-8080 -p 8080:9000 play-dockerized:1.0-SNAPSHOT
[info] - play.api.libs.concurrent.ActorSystemProvider - Starting application default Akka system: application
[info] - play.api.Play - Application started (Prod)
[info] - play.core.server.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

The other command line options for docker run are -p to expose port 9000 outside Docker’s virtual network (that’s locally available as 8080) and --name to give the container a friendly name (instead of relying on a cryptic hash).

In another terminal execute docker ps to see the container running:

➜  play-dockerized  docker ps -a
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS                    NAMES
511ca96e64a4        play-dockerized:1.0-SNAPSHOT   "bin/play-dockerized   10 minutes ago      Up 5 seconds        0.0.0.0:8080->9000/tcp   play-8080

Stop the container with docker stop play-8080. The Play Framework web app is no longer accessible. To start it again, execute docker start play-8080.

Summary

It’s so easy to have a Docker image of a Play Framework/Scala web application that I hardly believe I could have lived without using it for so long. Once an application becomes a Docker image you can use the other commands to play with it, mainly to deploy the image to any environment to have a consistent and exact environment. Love it so much now. And you can deploy the image to Docker Hub (similarly how you publish the sources of your excellent applications to GitHub).

Let me know what you think about the topic of the blog post in the Comments section below or contact me at jacek@japila.pl. Follow the author as @jaceklaskowski on Twitter, too.

Time flies by very quickly and as the other blog posts may have showed I’m evaluating Apache Kafka as a potential messaging and integration platform for my future projects. A lot is happening in so called big data space and Apache Kafka fits the bill in many dataflows around me so well. I’m very glad it’s mostly all Scala which we all love and are spending our time with. Ain’t we?

From Spark Streaming documentation (Kafka bolded on purpose):

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window.

Since Apache Kafka aims at being the central hub for real-time streams of data (see 1.2 Use Cases and Putting Apache Kafka To Use: A Practical Guide to Building a Stream Data Platform (Part 1)) I couldn’t deny myself the simple pleasure of giving it a go.

Buckle up and ingest some data using Apache Kafka and Spark Streaming! You surely will love the infrastructure (if you haven’t already). Be sure to type fast to see the potential of the platform at your fingertips.

The project (using sbt)

Here is the sbt build file build.sbt for the project:

val sparkVersion = "1.4.1"
scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion
)

It uses the latest released versions of Spark Streaming 1.4.1 and Scala 2.11.7.

Setting up Kafka broker

It assumes you’ve already installed Apache Kafka. You may want to use Docker (see Apache Kafka on Docker) or build Kafka from the sources. Whatever approach you choose, start Zookeeper and Kafka.

Starting Zookeeper

I’m using the version of Kafka built from the sources and ./bin/zookeeper-server-start.sh that comes with it.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/zookeeper-server-start.sh config/zookeeper.properties
...
[2015-07-21 06:51:39,614] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Starting Kafka broker

Once Zookeeper is up and running (in the above case, listening to the port 2181), run a Kafka broker.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-server-start.sh config/server.properties
...
[2015-07-21 06:53:17,051] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(192.168.1.9,9092,PLAINTEXT) (kafka.utils.ZkUtils$)
[2015-07-21 06:53:17,058] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

There are merely two commands to boot the entire environment up and that’s it.

Creating topic - spark-topic

A topic is where you’re going to send messages to and where Spark Streaming is consuming them from later on. It’s the communication channel between producers and consumers and you’ve got to have one.

Create spark-topic topic. The name is arbitrary and pick whatever makes you happy, but use it consistently in the other places where the name’s used.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic spark-topic
Created topic "spark-topic".

You may want to check out the available topics.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-topics.sh --list --zookeeper localhost:2181
spark-topic

You’re now done with setting up Kafka for the demo.

(optional) Sending to and receiving messages from Kafka

Apache Kafka comes with two shell scripts to send and receive messages from topics. They’re kafka-console-producer.sh and kafka-console-consumer.sh, respectively. They both use the console (stdin) as the input and output.

Let’s publish few messages to the spark-topic topic using ./bin/kafka-console-producer.sh.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic spark-topic
hello
hi
^D

You can keep the producer up in one terminal and use another terminal to consume the messages or just send a couple of messages and close the session. Kafka persists messages for a period of time.

Consuming messages is as simple as running ./bin/kafka-console-consumer.sh. Mind the option --zookeeper to point to Zookeeper where Kafka stores its configuration and --from-beginning that tells Kafka to process all persisted messages.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic spark-topic --from-beginning
hello
hi
^DConsumed 2 messages

Spark time!

Remember build.sbt above that sets the Scala/sbt project up with appropriate Scala version and Spark Streaming dependencies?

val sparkVersion = "1.4.1"
scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion
)

To learn a little about the integration between Spark Streaming and Apache Kafka you’re going to use sbt console and type all the integration code line by line in the interactive console. You could have a simple Scala application, but I’m leaving it to you as an exercise.

You may want to read the scaladoc of org.apache.spark.SparkConf and org.apache.spark.streaming.StreamingContext to learn about their purpose in the sample.

scala> import org.apache.spark.SparkConf
import org.apache.spark.SparkConf

scala> val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaReceiver")
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@2f8bf5fc

scala> import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext

scala> import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.Seconds

scala> val ssc = new StreamingContext(conf, Seconds(10))
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/21 09:08:39 INFO SparkContext: Running Spark version 1.4.1
...
ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@2ce5cc3

scala> import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.kafka.KafkaUtils

// Note the name of the topic in use - spark-topic
scala> val kafkaStream = KafkaUtils.createStream(ssc, "localhost:2181","spark-streaming-consumer-group", Map("spark-topic" -> 5))
kafkaStream: org.apache.spark.streaming.dstream.ReceiverInputDStream[(String, String)] = org.apache.spark.streaming.kafka.KafkaInputDStream@4ab601ac

// The very complex BIG data analytics
scala> kafkaStream.print()

// Start the streaming context so Spark Streaming polls for messages
scala> ssc.start
15/07/21 09:11:31 INFO ReceiverTracker: ReceiverTracker started
15/07/21 09:11:31 INFO ForEachDStream: metadataCleanupDelay = -1
...
15/07/21 09:11:31 INFO StreamingContext: StreamingContext started

Spark Streaming is now connected to Apache Kafka and consumes messages every 10 seconds. Leave it running and switch to another terminal.

Open a terminal to run a Kafka producer. You may want to use kafkacat (highly recommended) or the producer that comes with Apache Kafka - kafka-console-producer.sh.

➜  kafka_2.11-0.8.3-SNAPSHOT git:(trunk) ✗ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic spark-topic
hey spark, how are you doing today?

Switch to the terminal with Spark Streaming running and see the message printed out.

15/07/21 09:12:00 INFO DAGScheduler: ResultStage 1 (print at <console>:18) finished in 0.016 s
15/07/21 09:12:00 INFO DAGScheduler: Job 1 finished: print at <console>:18, took 0.030530 s
-------------------------------------------
Time: 1437462720000 ms
-------------------------------------------
(null,hey spark, how are you doing today?)

15/07/21 09:12:00 INFO JobScheduler: Finished job streaming job 1437462720000 ms.1 from job set of time 1437462720000 ms

Congratulations! You’ve earned the Spark Streaming and Apache Kafka integration badge! Close Spark Streaming’s context using

scala> ssc.stop

or simply press Ctrl+C. Shut down Apache Kafka and Zookeeper, too. Done.

(bonus) Building Apache Spark from the sources

You could use the very latest version of Spark Streaming in which the latest and greatest development is going on and lives on the master branch.

Following the official documentation Building Spark¹, execute the following two commands. Please note the versions in the build as it uses Scala 2.11.7 and Hadoop 2.7.1.

➜  spark git:(master) ./dev/change-scala-version.sh 2.11
➜  spark git:(master) ✗ ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Dscala-2.11 -DskipTests clean install
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  6.187 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 10.096 s]
[INFO] Spark Project Networking ........................... SUCCESS [  8.650 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  6.085 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 10.308 s]
[INFO] Spark Project Core ................................. SUCCESS [02:08 min]
[INFO] Spark Project Bagel ................................ SUCCESS [  6.750 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 15.942 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 32.429 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 55.389 s]
[INFO] Spark Project SQL .................................. SUCCESS [ 56.297 s]
[INFO] Spark Project ML Library ........................... SUCCESS [01:05 min]
[INFO] Spark Project Tools ................................ SUCCESS [  4.702 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 47.624 s]
[INFO] Spark Project REPL ................................. SUCCESS [  5.686 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [  7.479 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 11.903 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 59.155 s]
[INFO] Spark Project External Twitter ..................... SUCCESS [  7.177 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [  6.205 s]
[INFO] Spark Project External Flume ....................... SUCCESS [  9.151 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  1.896 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 15.044 s]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [  3.593 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [  6.658 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [ 11.207 s]
[INFO] Spark Project Examples ............................. SUCCESS [01:16 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  5.115 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:21 min
[INFO] Finished at: 2015-08-22T15:08:02+02:00
[INFO] Final Memory: 431M/1960M
[INFO] ------------------------------------------------------------------------

The jars for the version are at your command in the Maven local repository. Switch the version of Spark Streaming to 1.5.0-SNAPSHOT and start over.

Summary

As it turns out setting up a working configuration of Apache Kafka and Spark Streaming is just few clicks away. There are a couple of places that need improvement, but what the article has showed could be a good starting point for other real-time big data analytics using Apache Kafka as the central hub for real-time streams of data that are then processed using complex algorithms in Spark Streaming.

Once the data’s processed, Spark Streaming could be publishing results into yet another Kafka topic and/or store in Hadoop for later. It seems I’ve got a very powerful setup I’m not yet fully aware of where I should apply to.

Let me know what you think about the topic² of the blog post in the Comments section below or contact me at jacek@japila.pl. Follow the author as @jaceklaskowski on Twitter, too.

In order to learn Kafka better, I developed a custom producer using the latest Kafka’s Producer API in Scala. I built Kafka from the sources, and so I’m using the version 0.8.3-SNAPSHOT. It was pretty surprising experience, esp. when I ran across java.util.concurrent.Future that seems so limited to what scala.concurrent.Future offers. No map, flatMap or such? So far I consider the switch to using Java for the Client API a big mistake.

Here comes the complete Kafka producer I’ve developed in Scala that’s supposed to serve as a basis for my future development endeavours using the API in what’s going to be in 0.8.3 release.

Custom KafkaProducer in Scala

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import java.util.concurrent.Future

import org.apache.kafka.clients.producer.RecordMetadata

object KafkaProducer extends App {

  val topic = util.Try(args(0)).getOrElse("my-topic-test")
  println(s"Connecting to $topic")

  import org.apache.kafka.clients.producer.KafkaProducer

  val props = new java.util.Properties()
  props.put("bootstrap.servers", "localhost:9092")
  props.put("client.id", "KafkaProducer")
  props.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer")
  props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")

  val producer = new KafkaProducer[Integer, String](props)

  import org.apache.kafka.clients.producer.ProducerRecord

  val polish = java.time.format.DateTimeFormatter.ofPattern("dd.MM.yyyy H:mm:ss")
  val now = java.time.LocalDateTime.now().format(polish)
  val record = new ProducerRecord[Integer, String](topic, 1, s"hello at $now")
  val metaF: Future[RecordMetadata] = producer.send(record)
  val meta = metaF.get() // blocking!
  val msgLog =
    s"""
       |offset    = ${meta.offset()}
       |partition = ${meta.partition()}
       |topic     = ${meta.topic()}
     """.stripMargin
  println(msgLog)

  producer.close()
}

Building Kafka from the sources

In order to run the client you should build Kafka from the sources first and publish the jars to the local Maven repository. The reason to have the build is that the producer uses the very latest Kafka Producer API.

Building Kafka from the sources is as simple as executing gradle -PscalaVersion=2.11.7 clean releaseTarGz in the directory where you git clone https://github.com/apache/kafka.gitd the Kafka repo from GitHub.

➜  kafka git:(trunk) gradle -PscalaVersion=2.11.7 clean releaseTarGz install
Building project 'core' with Scala version 2.11.7
...
BUILD SUCCESSFUL

Total time: 1 mins 23.233 secs

I was building the distro against Scala 2.11.7.

Once done, core/build/distributions/kafka_2.11-0.9.0.0-SNAPSHOT.tgz is where you find the release package.

➜  kafka git:(trunk) ls -l core/build/distributions/kafka_2.11-0.9.0.0-SNAPSHOT.tgz
-rw-r--r--  1 jacek  staff  17813003 29 wrz 08:32 core/build/distributions/kafka_2.11-0.9.0.0-SNAPSHOT.tgz

Unpack it and cd to it.

➜  kafka git:(trunk) tar -zxf core/build/distributions/kafka_2.11-0.9.0.0-SNAPSHOT.tgz
➜  kafka git:(trunk) ✗ cd kafka_2.11-0.9.0.0-SNAPSHOT
➜  kafka_2.11-0.9.0.0-SNAPSHOT git:(trunk) ✗ ls -l
total 32
-rw-r--r--   1 jacek  staff  11358  9 lis  2014 LICENSE
-rw-r--r--   1 jacek  staff    162  9 lis  2014 NOTICE
drwxr-xr-x  26 jacek  staff    884 29 wrz 08:32 bin
drwxr-xr-x  16 jacek  staff    544 29 wrz 08:32 config
drwxr-xr-x  21 jacek  staff    714 29 wrz 08:32 libs

Zookeeper up and running

Running Zookeeper is the very first step you should do (as that’s how Kafka maintains high-availability). Use ./bin/zookeeper-server-start.sh config/zookeeper.properties:

➜  kafka_2.11-0.9.0.0-SNAPSHOT git:(trunk) ✗ ./bin/zookeeper-server-start.sh config/zookeeper.properties
[2015-09-29 12:26:41,011] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2015-09-29 12:26:41,014] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-09-29 12:26:41,014] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-09-29 12:26:41,014] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-09-29 12:26:41,014] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2015-09-29 12:26:41,036] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2015-09-29 12:26:41,036] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2015-09-29 12:26:41,301] INFO Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2015-09-29 12:26:41,301] INFO Server environment:host.name=172.20.36.184 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-09-29 12:26:41,301] INFO Server environment:java.version=1.8.0_60 (org.apache.zookeeper.server.ZooKeeperServer)
...
[2015-09-29 12:26:41,333] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Kafka broker up and running

In another terminal, start a Kafka broker using ./bin/kafka-server-start.sh config/server.properties command:

  ➜  kafka_2.11-0.9.0.0-SNAPSHOT git:(trunk) ✗ ./bin/kafka-server-start.sh config/server.properties
...
[2015-07-20 00:18:33,671] INFO starting (kafka.server.KafkaServer)
[2015-07-20 00:18:33,673] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2015-07-20 00:18:33,684] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-07-20 00:18:33,693] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-07-20 00:18:33,694] INFO Client environment:host.name=192.168.1.9 (org.apache.zookeeper.ZooKeeper)
[2015-07-20 00:18:33,694] INFO Client environment:java.version=1.8.0_45 (org.apache.zookeeper.ZooKeeper)
[2015-07-20 00:18:33,694] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
...
[2015-09-29 13:18:49,919] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(192.168.99.1,9092,PLAINTEXT) (kafka.utils.ZkUtils$)
[2015-09-29 13:18:49,933] INFO Kafka version : 0.9.0.0-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
[2015-09-29 13:18:49,933] INFO Kafka commitId : 4e7db39556ba916c (org.apache.kafka.common.utils.AppInfoParser)
[2015-09-29 13:18:49,934] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-09-29 13:18:49,935] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

Creating topic

You’re now going to create my-topic topic where the custom producer is going to publish messages to. Of course, the name of the topic is arbitrary, but should match what the custom producer uses.

➜  kafka_2.11-0.9.0.0-SNAPSHOT git:(trunk) ✗ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-topic-test
Created topic "my-topic-test".

Check out the topics available using ./bin/kafka-topics.sh --list --zookeeper localhost:2181. You should see one.

➜  kafka_2.11-0.9.0.0-SNAPSHOT git:(trunk) ✗ ./bin/kafka-topics.sh --list --zookeeper localhost:2181
my-topic-test

kafka-publisher - Scala project

Create a Scala project. The project is managed by sbt with the following build.sbt:

val kafkaVersion = "0.9.0.0-SNAPSHOT"
scalaVersion := "2.11.7"

libraryDependencies += "org.apache.kafka" % "kafka-clients" % kafkaVersion
resolvers += Resolver.mavenLocal

Use the following project/build.properties:

sbt.version=0.13.9

Sending messages using KafkaProducer - sbt run

With the setup, you should now be able to run sbt run to run the custom Scala producer for Kafka.

➜  kafka-publisher  sbt run
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/dev/sandbox/kafka-publisher/project
[info] Set current project to kafka-publisher (in build file:/Users/jacek/dev/sandbox/kafka-publisher/)
[info] Compiling 1 Scala source to /Users/jacek/dev/sandbox/kafka-publisher/target/scala-2.11/classes...
[info] Running KafkaProducer
Connecting to my-topic-test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

offset    = 0
partition = 0
topic     = my-topic-test

[success] Total time: 4 s, completed Sep 29, 2015 4:21:14 PM

Executing sbt run again should show a different offset for the sam partition and topic:

➜  kafka-publisher  sbt run
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/dev/sandbox/kafka-publisher/project
[info] Set current project to kafka-publisher (in build file:/Users/jacek/dev/sandbox/kafka-publisher/)
[info] Running KafkaProducer
Connecting to my-topic-test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

offset    = 1
partition = 0
topic     = my-topic-test

[success] Total time: 1 s, completed Sep 29, 2015 4:21:47 PM

Using kafkacat as a Kafka message consumer

If you really would like to see the message on the other, receiving side, I strongly recommend using kafkacat that, once installed, boils down to the following command:

➜  ~  kafkacat -C -b localhost:9092 -t my-topic-test
hello at 20.07.2015 0:29:43
hello at 20.07.2015 0:30:46

It will read all the messages already published to my-topic-test topic and print out others once they come.

That’s it. Congratulations!

Summary

The complete project is on GitHub in kafka-producer repo.

You may also want to read 1.3 Quick Start in the official documentation of Apache Kafka.

Let me know what you think about the topic¹ of the blog post in the Comments section below or contact me at jacek@japila.pl. Follow the author as @jaceklaskowski on Twitter, too.

Apache Kafka has always been high on my list of things to explore, but since there are quite a few things high on my list, Kafka couldn’t actually make it to the very top. Until just recently, when I was asked to give the broker a try and see whether or not it meets a project’s needs. Two projects, to be honest. You should see my face when I heard it.

I compiled Apache Kafka from the sources, connected it to Spark Streaming and even attempted to answer few questions on StackOverflow (How to use Kafka in Flink using Scala? and How to monitor Kafka broker using jmxtrans?), not to mention reading tons of articles and watching videos about the tool. I developed pretty strong confidence what use cases are the sweet spot for Apache Kafka.

With the team in Codilime I’m developing DeepSense.io platform where we have just used Ansible to automate deployment. We’ve also been evaluating Docker and/or Vagrant. All to ease the deployment of DeepSense.io.

That’s the moment when these two needs converged - exploring Apache Kafka and Docker (among the other tools) for three separate projects! Amazing, isn’t it? I could finally explore how Docker might ease exploration of products and deployment. I knew Docker could ease my developer life, but it’s only now when I really saw it. I would now dockerize everything. When I was told about the images wurstmeister/kafka and wurstmeister/zookeeper I couldn’t have been happier. Running Apache Kafka and using Docker finally became a no-brainer and such a pleasant experience.

I then thought I’d share the love so it’s not only mine and others could benefit from it, too.

Since I’m on Mac OS X the steps to run Apache Kafka using Docker rely on boot2docker - a Lightweight Linux for Docker for platforms that don’t natively support Docker - aforementioned Mac OS X and Windows.

You’re going to use the images wurstmeister/kafka and wurstmeister/zookeeper.

You can run containers off the images in background or foreground. Depending on you Unix skills, it means one or two terminals. Let’s use two terminals for each server - Apache Kafka and Apache Zookeeper. I’m going to explain the role of Apache Zookeeper in another blog post.

Here come the steps to run Apache Kafka using Docker. It’s assumed you’ve got boot2docker and docker tools installed.

➜  ~  boot2docker version
Boot2Docker-cli version: v1.7.1
Git commit: 8fdc6f5

➜  ~  docker --version
Docker version 1.7.1, build 786b29d

I’m a big fan of homebrew and highly recommend it to anyone using Mac OS X. Plenty of ready-to-use packages are just brew install away, docker and boot2docker including.

Running Kafka on two Docker images

1. (Mac OS X and Windows users only) Execute boot2docker up to start the tiny Linux core on Mac OS.

    ➜  ~  boot2docker up
    Waiting for VM and Docker daemon to start...
    .o
    Started.
    Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/ca.pem
    Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/cert.pem
    Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/key.pem
   
    To connect the Docker client to the Docker daemon, please set:
        export DOCKER_HOST=tcp://192.168.59.104:2376
        export DOCKER_CERT_PATH=/Users/jacek/.boot2docker/certs/boot2docker-vm
        export DOCKER_TLS_VERIFY=1
   

2. (Mac OS X and Windows users only) Execute $(boot2docker shellinit) to have the terminal set up and let docker know where the tiny Linux core is running (via boot2docker). You have to do the step in any terminal you open to work with Docker so the exports above are set. Should you face communication issues with docker commands, remember the step.

   ➜  ~  $(boot2docker shellinit)
   Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/ca.pem
   Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/cert.pem
   Writing /Users/jacek/.boot2docker/certs/boot2docker-vm/key.pem
   

3. Run docker ps to verify the terminal is configured properly for Docker.

   ➜  ~  docker ps
   CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
   

   No containers are running at this time. It’s going to change soon once you start the containers for Zookeeper first and then Kafka.

4. Create an account on Docker Hub and execute docker login to store the credentials. With the step you don’t have to repeat them for docker pull to pull images off the public hub of Docker images. Think of the Docker Hub as the GitHub for Docker images. Refer to the documentation Using the Docker Hub for more up-to-date information.

5. Run docker pull wurstmeister/zookeeper to pull the Zookeeper image off Docker Hub (might take a few minutes to download):

   ➜  ~  docker pull wurstmeister/zookeeper
   Pulling repository wurstmeister/zookeeper
   a3075a3d32da: Download complete
   ...
   840840289a0d: Download complete
   e7381f1a45cf: Download complete
   5a6fc057f418: Download complete
   Status: Downloaded newer image for wurstmeister/zookeeper:latest
   

   You will see hashes of respective layers printed out to the console. It’s expected.

6. Execute docker pull wurstmeister/kafka to pull the Kafka image off Docker Hub (might take a few minutes to download):

    ➜  ~  docker pull wurstmeister/kafka
    latest: Pulling from wurstmeister/kafka
    428b411c28f0: Pull complete
    ...
    422705fe88c8: Pull complete
    02bb7ca441d8: Pull complete
    0f9a08061516: Pull complete
    24fc32f98556: Already exists
    Digest: sha256:06150c136dcfe6e4fbbf37731a2119ea17a953c75902e52775b5511b3572aa1f
    Status: Downloaded newer image for wurstmeister/kafka:latest
   

7. Verify that the two images - wurstmeister/kafka and wurstmeister/zookeeper - are downloaded. From the command line execute docker images:

   ➜  ~  docker images
   REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
   wurstmeister/kafka       latest              24fc32f98556        3 weeks ago         477.6 MB
   wurstmeister/zookeeper   latest              a3075a3d32da        9 months ago        451 MB
   

8. You can now execute docker run --name zookeeper -p 2181 -t wurstmeister/zookeeper in one terminal to boot Zookeeper up.

   Remember $(boot2docker shellinit) if you’re on Mac OS X or Windows.

    ➜  ~  docker run --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper
    JMX enabled by default
    Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
    2015-07-17 19:10:40,419 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
    ...
    2015-07-17 19:10:40,452 [myid:] - INFO  [main:ZooKeeperServer@773] - maxSessionTimeout set to -1
    2015-07-17 19:10:40,464 [myid:] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
   

   This gives you Zookeeper listening to port 2181. Check it out by telneting to it using docker (or boot2docker on MacOS) ip address.

    ➜  ~  telnet `boot2docker ip` 2181
    Trying 192.168.59.103...
    Connected to 192.168.59.103.
    Escape character is '^]'.
   

9. Execute docker run --name kafka -e HOST_IP=localhost -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_BROKER_ID=1 -e ZK=zk -p 9092 --link zookeeper:zk -t wurstmeister/kafka in another terminal.

   Remember $(boot2docker shellinit) if you’re on Mac OS X or Windows.

    ➜  ~  docker run --name kafka -e HOST_IP=localhost -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_BROKER_ID=1 -e ZK=zk -p 9092 --link zookeeper:zk -t wurstmeister/kafka
    [2015-07-17 19:32:35,865] INFO Verifying properties (kafka.utils.VerifiableProperties)
    [2015-07-17 19:32:35,891] INFO Property advertised.port is overridden to 9092 (kafka.utils.VerifiableProperties)
    [2015-07-17 19:32:35,891] INFO Property broker.id is overridden to 1 (kafka.utils.VerifiableProperties)
    ...
    [2015-07-17 19:32:35,894] INFO Property zookeeper.connect is overridden to 172.17.0.5:2181 (kafka.utils.VerifiableProperties)
    [2015-07-17 19:32:35,895] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
    [2015-07-17 19:32:35,924] INFO [Kafka Server 1], starting (kafka.server.KafkaServer)
    [2015-07-17 19:32:35,925] INFO [Kafka Server 1], Connecting to zookeeper on 172.17.0.5:2181 (kafka.server.KafkaServer)
    [2015-07-17 19:32:35,934] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
    [2015-07-17 19:32:35,939] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
    ...
    [2015-07-17 19:32:36,093] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
    [2015-07-17 19:32:36,095] INFO [Socket Server on Broker 1], Started (kafka.network.SocketServer)
    [2015-07-17 19:32:36,146] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
    [2015-07-17 19:32:36,172] INFO 1 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
    [2015-07-17 19:32:36,253] INFO Registered broker 1 at path /brokers/ids/1 with address 61c359a3136b:9092. (kafka.utils.ZkUtils$)
    [2015-07-17 19:32:36,270] INFO [Kafka Server 1], started (kafka.server.KafkaServer)
    [2015-07-17 19:32:36,318] INFO New leader is 1 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
   

   You’re now a happy user of Apache Kafka on your computer using Docker. Check the status of the containers using docker ps:

    ➜  ~  docker ps
    CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                                 NAMES
    0b34a9927004        wurstmeister/kafka       "/bin/sh -c start-ka   2 minutes ago       Up 2 minutes        0.0.0.0:32769->9092/tcp                               kafka
    14fd32558b1c        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb   4 minutes ago       Up 4 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32768->2181/tcp   zookeeper
   

10. Once you’re done with your journey into Apache Kafka, docker stop the containers using docker stop kafka zookeeper (or docker stop $(docker ps -aq) if the only running containers are kafka and zookeeper).

     ➜  ~  docker stop kafka zookeeper
     kafka
     zookeeper
    

    Running docker ps shows no running containers afterwards:

     ➜  ~  docker ps
     CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    

    There are no running containers since they’re stopped now. They are still ready to be booted up again - use docker ps -a to see the ready-to-use containers:

     ➜  ~  docker ps -a
     CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS                        PORTS               NAMES
     7dde25ff7ec2        wurstmeister/kafka       "/bin/sh -c start-ka   15 hours ago        Exited (137) 16 seconds ago                       kafka
     b7b4b675b9c0        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb   16 hours ago        Exited (137) 5 seconds ago                        zookeeper
    

11. (Mac OS X and Windows users only) Finally, stop boot2docker daemon using boot2docker down.

Summary

With these two docker images - wurstmeister/kafka and wurstmeister/zookeeper - you can run Apache Kafka without much changing your local workstation to install it together with the necessary components like Apache ZooKeeper. You don’t need to worry about upgrading the software and its dependencies except docker itself (and boot2docker if you’re lucky to be on Mac OS). That saves you from spending time on installation and ensures proper functioning of your machine and the software. Moreover, the Docker images could be deployed to other machines and guarantee a consistent environment of the software inside.

Let me know what you think about the topic¹ of the blog post in the Comments section below or contact me at jacek@japila.pl. Follow the author as @jaceklaskowski on Twitter, too.

This time I sank into type classes that is a means of doing ad hoc polymorphism in Scala.

From ad hoc polymorphism article on Wikipedia:

In programming languages, ad hoc polymorphism is a kind of polymorphism in which polymorphic functions can be applied to arguments of different types, because a polymorphic function can denote a number of distinct and potentially heterogeneous implementations depending on the type of argument(s) to which it is applied.

The blog post presents a way to implement the type classes concept in Scala.

p.s. I’m yet to find out how much of it is multimethods in Clojure (that was once of much help to introduce me to functional programming).

Theory

From the article Polymorphism (computer science)) on Wikipedia, you can read about three different kinds of polymorphism:

• subtyping
• parametric polymorphism
• ad hoc polymorphism

The order does matter (and is different from the Wikipedia article’s one) as I think that’s exactly the order of when and how well developers master them (in any programming language).

I reckon the Scala community finds the first two quite often used in code that contributes to how well it’s understood and applied (except variance), and just the last one - ad hoc polymorphism - is proving a major hurdle for many, me including. It didn’t click for me for a long time, either, only until I found the “venues” of very concise and comprehensible material (see References section below).

In programming languages and type theory, polymorphism is the provision of a single interface to entities of different types.

The difference between subtyping and parametric polymorphism and ad hoc polymorphism is that the type hierarchy is expressed explicitly using extends keyword in Scala (for subtyping) or type parameters (for parametric polymorphism) while ad hoc polymorphism bases itself on implicit classes to mixin the behaviour (using traits). And that’s pretty much all, technically.

Let’s see it in a Scala code.

Practice

Algebraic data types (raw data)

Let’s assume you have the following type hierarchy (from Tutorial: Typeclasses in Scala with Dan Rosen):

sealed trait Expression
case class Number(value: Int) extends Expression
case class Plus(lhs: Expression, rhs: Expression) extends Expression
case class Minus(lhs: Expression, rhs: Expression) extends Expression

No behaviour, but algebraic data types or (often called) raw data.

Think about how you’d go about evaluating expressions, i.e. Plus(Minus(Number(5), Number(3)), Number(18)), i.e. how to make the following application compile and print 20?

object Main extends App {
  val expr = Plus(Minus(Number(5), Number(3)), Number(18))
  println(...) // do something with expr so it prints 20
}

The past me would change sealed trait Expression to include def value: Int and force the three case classes Number, Plus and Minus follow. I’d not be surprised if you thought exactly so. You should not, however.

Think about the case where you must not change them as they could be provided as a library or be part of the language or be licensed or…I simply ask you not to.

objects to apply behaviour

Since you’re in Scala, you could easily work it around with an object, say object ExpressionEvaluator, that would pattern match to types and do the heavy lifting, i.e. know what to do with each and every type:

object ExpressionEvaluator {
  def value(expression: Expression): Int = expression match {
    case Number(value) => value
    case Plus(lhs, rhs) => value(lhs) + value(rhs)
    case Minus(lhs, rhs) => value(lhs) - value(rhs)
  }
}

That’s quite inefficient and cumbersome, though. You’d have to know about all of the implementations as well as about what to do for each. It’s nearly impossible to have right, complete and still flexible.

With the above object, you’d write:

object Main extends App {
  val expr = Plus(Minus(Number(5), Number(3)), Number(18))
  println(ExpressionEvaluator.value(expr)) // print the value
}

implicits in Scala

Let’s do it the right way, so it’s not ExpressionEvaluator to know what to do with every Expression, but expressions themselves do what they’re supposed to do instead.

You may have heard about implicits in Scala. Perhaps, you may have used them, too. Think about a solution with implicit machinery so the following is possible:

object Main extends App {
  val expr = Plus(Minus(Number(5), Number(3)), Number(18))
  println(expr.value) // print the value
}

One way in Scala would be to apply Pimp my Library pattern leveraging implicit classes to add necessary methods as follows:

implicit class ExpressionOps(e: Expression) {
  def value = ... // calculate the value
}

And have a implicit class per case class and sealed trait Expression, too. The reason to have the implicits is to add a value method to every type, i.e. implicitly convert types without value to ones that have it.

implicit class ExpressionOps(e: Expression) {
  def value: Int = e match {
    case n : Number => n.value
    case p : Plus => p.value
    case m : Minus => m.value
  }
}

implicit class PlusOps(p: Plus) {
  def value: Int = p.lhs.value + p.rhs.value
}

implicit class MinusOps(m: Minus) {
  def value: Int = m.lhs.value - m.rhs.value
}

Note that since Number had value already, an implicit was not needed.

With the implicits in place, you can write:

println(expr.value)  // prints 20

The implicit-based solution is far much more flexible because calculating a value is the responsibility of an implicit in scope – a change in a single implicit, say MinusOps, would only change how Minus.value works (with no changes to the rest of the “framework”).

You’re halfway to typeclasses!

Partial ad hoc polymorphism

Think about the case where you’d have a library to calculate a value of Valueable values (no pun intended). Say, you have such a library that offers the following “calculator”:

def calculate(v: Valueable) = ... // calculate the value of v

How would you go about converting Expressions to Valueables so a value of Expression type would participate in the contract of def calculate?

You’re right that whenever type conversion is needed in Scala, implicits have their say – you used them already to have def value for Expression type hierarchy. You’re going to use them again with a very small change that has enormous impact. Doing so will introduce the type class design pattern.

The current solution relies on values with def value – it’s like having a library that uses structural typing in Scala that unfortunatelly uses reflection and hence is very costly performance-wise. Happily, you don’t need structural types.

If you had a library that expects values of some type, say trait Valueable, to calculate a value or some other library to JSONify them (using some trait Json), the previous solutions would fall short – you’ve merely met the requirement of being able to call a value method on values, but the values don’t belong to a single type hierarchy of some trait Valueable that the library uses.

Assume you have a library that works with trait Valueable-only values and there’s a calculate method to work with them as follows:

trait Valueable {
  def value: Int
}

def calculate(v: Valueable) = v.value

Note that the library knows nothing about the Expression type hierarchy. The Expression type hierarchy could not even have existed at the time Valueable did.

A more flexible and efficient solution is to somehow meld the Expression type hierarchy with Valueable and create is-a relationship. No, no, you’re not going to change the Expression trait in any way. You must not and you even can’t, remember?

Welcome to typeclasses (also known as type class design pattern)!

In order to have extends-like semantic at runtime with no extends Valueable in the (closed and sealed) Expression type hierarchy you change implicit classes to have the common trait mixed in, instead.

implicit class ExpressionOps(e: Expression) extends Valueable {
  def value = e match {
    case n: Number => n.value
    case p: Plus => p.value
    case m: Minus => m.value
  }
}

implicit class PlusOps(p: Plus) {
  def value: Int = p.lhs.value + p.rhs.value
}

implicit class MinusOps(m: Minus) {
  def value: Int = m.lhs.value - m.rhs.value
}

With the above implicit classes that all extends Valueable, you can safely do expr.value for any Expression value for which an implicit exists in scope and be done with the task at hand. Notice how Scala “executes” value on the different types (that’s based upon using implicit classes for the types when value is needed).

println(calculate(expr))  // prints 20

For what is worth, the calculate method belongs to a library that knows nothing about Expression type and you can’t change it so Expressions would be worked with. That’s exactly where typeclasses shines and are hard to beat.

This is the version of type classes pattern as described in the book Programming Scala, 2nd Edition in “Type Class Pattern” section, page 156.

Final solution: Value[T] type class

You can still do better than that in Scala and that’s what (the real) type class design pattern offers. This is the version of type class pattern as demonstrated in the video Tutorial: Typeclasses in Scala with Dan Rosen. You should really watch it.

Let’s have a parameterized type trait Value[T] that’s supposed to “bridge” the type hierarchies - Expression and Valueable:

trait Value[T] {
  def value(t: T): Valueable
}

The fictitious Calculator library has object Calculator with a single def calculate(v: Valueable): Int method:

object Calculator {
  def calculate(v: Valueable): Int = v.value
}

No Expressions, just Valueables. That’s where the type class pattern shines.

Create object CalculatorEx as follows:

object CalculatorEx {
  def calculate[T : Value](t: T): Int =
    Calculator.calculate(implicitly[Value[T]].value(t))
}

It says that for any type T there’s an implicit conversion to a Value[_] type hierarchy using implicits (that are used in the method via implicitly).

All in all, you need to throw in three more implicits for Number, Plus and Minus so they can be seen as Valueable and participate in def calculate(v: Valueable): Int-based library:

implicit val number2Value = new Value[Number] {
  def value(n: Number): Valueable = new Valueable {
    override def value: Int = n.value
  }
}

implicit val plus2Value = new Value[Plus] {
  def value(p: Plus): Valueable = new Valueable {
    override def value: Int = p.lhs.value + p.rhs.value
  }
}

implicit val minus2Value = new Value[Minus] {
  def value(m: Minus): Valueable = new Valueable {
    override def value: Int = m.lhs.value - m.rhs.value
  }
}

These make it possible to calculate the value of the expression leveraging the fictitious Calculator library:

println(CalculatorEx.calculate(expr))  // prints 20

That’s it! You’re done. If you’ve followed along closely and have developed the “framework” on your own, you should have a complete understanding of the type class design pattern in Scala. Without the type class pattern, blending the Expression type hierarchy with Valueable would not have been possible! And that was the goal of the exercise.

Follow up - All Valueable

Think about using other types, say Int, with the fictitious Calculator library so the following is possible:

println(CalculatorEx.calculate(1))  // prints 1

As the Scala compiler says:

could not find implicit value for evidence parameter of type Value[Int]

All, we need to do to blend Ints as Valueables is to have an implicit that does the conversion.

implicit val int2Value = new Value[Int] {
  override def value(t: Int) = new Valueable {
    override def value: Int = t
  }
}

And that’s it!

As a final exercise would be to reuse implicits and create more complex ones to support “sum” types, like Tuple. I’m leaving it as a homework. Have fun!

Let me know how the homework and the whole blog post went out in the Comments section below. I’d appreciate any comments to improve upon.

References

There are plenty of very good sources on the topic of type classes in general and in Scala, in particular, but the following have done wonders for me:

• ad hoc polymorphism article on Wikipedia
• Polymorphism (computer science)) article on Wikipedia
• Tutorial: Typeclasses in Scala with Dan Rosen video
• Programming Scala, 2nd Edition book

I can now ditch Guice’s @Singleton as I’ve got a trait and the companion object combo (thanks to Scala).

This time the blog post is without a complete working example. Not yet. It’s to remind myself to prepare one (or be given one after the blog post is published – whatever comes first). I just think it needs to be said aloud to be heard and think about.

And then, quite unexpectedly to me, Play Framework - “The High Velocity Web Framework For Java and Scala” - that’s the web framework supported by Typesafe in their Reactive Platform has announced in What’s new in Play 2.4:

In the Scala ecosystem, the approach to dependency injection is not generally agreed upon, with many competing compile time and runtime dependency injection approaches out there.

Play’s philosophy in providing a dependency injection solution is to be unopinionated in what approaches we allow, but to be opinionated to the approach that we document and provide out of the box. For this reason, we have provided the following:

An implementation that uses Guice out of the box

I’m very lucky to be able to pursue my understanding of Scala the programming language not only in my free time, but also in commercial projects as a full-time Scala developer and a technical leader (in DeepSense.io) as well as supporting companies making the most out of Scala and open source software (in HCore) not to mention leading Warsaw Scala Enthusiasts in Warsaw, Poland. The technical part of my life simply can’t be any better! I’m learning as well as teaching people using Scala as an object-oriented and functional programming language on JVM, and am also meeting up lots of Scala developers. I really wish I had more time to publish all the major breakthroughs in blog posts here.

Enough praising. On to real matters.

In DeepSense.io we’re using Guice as a dependency framework. It’s also used in few other projects where Scala is used. That often leads to my questioning the need for Guice or any dependency injection framework since Scala the programming language itself offers enough features to let Guice leave.

I think there’s the issue of how most Scala developers (I’m meeting) think – they are former Java developers who see no reason to adapt to new approaches of tackling development problems. They simply have no more courage to dig deeper, and…let the past rest and welcome new solutions for the brighter future. And since Scala alone is still shaping itself and the Scala community is not clear on what to follow along and what to forget about, that all makes the leave-past-welcome-new approach harder.

Just this week I had a pleasure to meet two teams that both use Guice because nobody introduced viable alternatives (even when they use one already - Scala!). I don’t consider myself skilled enough, either, since I’m pretty much a Guice newbie and hence just unable to counter the Guice’s way of solving problems. It’s that something inside me that is telling me that Guice is dragged along unnecessarily and gives a false perception of being the right fit to dependency injection-looking problems (even when it introduces more problems, mostly related to learning a yet another framework, than it solves). I’m glad Spring is far heavier as it would likely have found its way in the projects, too.

It has worked fine for Arek and I’m hoping it’s going to work fine for me soon, too. I’ll simply have to find it out and promote the right approach.

I’ve recently been assigned to a task to create a solution to share common settings across projects in a multi-project build in a Scala project managed by sbt. With the new feature of sbt - autoplugins - it was very easy to implement from the day one.

The solution = project/CommonSettingsPlugin.scala

import sbt._
import Keys._

object CommonSettingsPlugin extends AutoPlugin {
  override def trigger = allRequirements
  override lazy val projectSettings = Seq(
    organization  := "io.deepsense",
    version       := "0.1.0",
    scalaVersion  := "2.11.6",
    scalacOptions := Seq(
      "-unchecked", "-deprecation", "-encoding", "utf8", "-feature",
      "-language:existentials", "-language:implicitConversions"
    )
  )
}

Since the autoplugin is automatically triggered for all the projects in the multi-project build – override def trigger = allRequirements – all the projects have organization, version, scalaVersion, scalacOptions set. No other work is needed.

Simple and easy, and, what’s tracked under another issue, Jenkins should now be able to execute coverageReport and coverageAggregate without troubles for the project!

As a bonus, you don’t have to change anything in your sbt project to leverage the simplicity - just drop the code as project/CommonSettingsPlugin.scala and do show scalaVersion. From that moment on, all the (sub)projects in your multi-project build should have the same scalaVersion (unless you use a separate build.sbt for a project and set scalaVersion explicitly).

Lessons learnt

The initial mistake was to create a separate sbt project just to keep the autoplugin’s code.

From the very first day of the plugin’s life it was so much different from the other projects in the build – it almost yelled out in pain at me not to place it where it was initially. It was a sbt plugin and as such it only meant to enhance sbt itself (not be a part of the commercial project).

What’s even worse, up to the latest version of sbt 0.13.8 all plugins (are doomed to) use Scala 2.10.4 that’s an issue for sbt-scoverage plugin as it kept refusing to work with the version of Scala.

All in all, I seemed reluctant for a long time to have even thought of a better place for the plugin even though I had actually known it.

A few discussions on gitter on the sbt channel made my day after the very helpful and nice people from the channel lent me a helping hand to find the final solution - the autoplugin went under project and…my live was so much bright again!

What was later pointed out to me when we discussed the change in the team, the final solution was a mere 24 line removal (!)

It got +2 from a teammate and has been merged to master without much hussle.

p.s. Shhh, the team thinks it was just me to have figured it out myself.

I’m yet to appreciate gerrit as a code review tool worth to learn (after having heard bad and good stories about its features and how it complements development workflows), but in my new team at Codilime where we develop…a revolutionary machine learning engine enabling your team to use state-of-the-art algorithms in a fraction of time! that’s the tool to conduct code reviews.

The blog post presents how I discovered a way to contribute to a patch set with my own changes. Use with caution as I’m not really sure that’s how gerrit should be used in a team.

Very (lame) intro to gerrit

From the website of gerrit:

Web based code review and project management for Git based projects.

The take away from it is that it’s the tool for code reviews for git project. You git push your changes to master early and often – I’d say it’s even earlier and more often than I did following the github workflow with pull requests or the gitlab workflow with merge requests.

At GitHub and GitLab I usually git rebase changes to squash them all into one and send a pull/merge request.

In gerrit, you git commit --amend as a way to keep changes under a single code review request. The point is to keep Change-Id and you should be fine in a single code review session. Every git push following git commit --amend with a single Change-Id creates a new Patch Set until it’s ready to be merged with master. These Patch Sets (under a single Change-Id) establish a sort of branch from which you send changes for code review.

Once the code looks good, i.e. it’s +1 twice by the team and verified (possibly by Jenkins), it can be submitted to the central git repository - preferably (private) one at GitHub.

Drafts as branches - not for review

There’s a feature of gerrit called drafts that…will be for a change that is not meant for review (yet). It’s a good candidate for changes that other developers can contribute to (I might be mistaken here since I’m new to gerrit).

Contributing to a Patch Set

DISCLAIMER: What follows assumes that a patch set in draft status can be considered a branch to develop collaboratively with a few developers. I’m not so sure it’s a correct assumption.

In gerrit, go to All > Open where you see all the available changes for code review. Pick one, preferrably in Draft status (but others should work fine, too).

Expand the latest Patch Set and select checkout tab in Download. Copy the git fetch ... && git checkout FETCH_HEAD command using the icon on the right (see the screenshot above).

Do your changes and once satisfied git checkout -b new_branch_name to create a new branch as new_branch_name.

git add the changes and very important git commit --amend them with Change-Id untouched. The Change-Id is the way for gerrit to keep track what belongs to what development stream as a new patch set. You may also want to change the author with git commit --amend --author to mark the changes as yours.

Push the commit to origin (or whatever remote repository you use) with git push origin <new_branch_name>:refs/drafts/master. The git push command assumes that you’re on master branch.

You’re done!

See How to change a patchset and push it as a new one? where the knowledge was initially developed.

In this blog post I’m introducing you to two modes in the recently-shipped IntelliJ IDEA 14.1 – Full Screen and Distraction Free modes – and the few keystrokes I use in the development environment to have a comfortable place to learn Scala. I’m sure you’ll have found few ideas to improve your way into your own personal Scala nirvana.

Let’s go minimalistic, full screen, distraction-free, mouse- and touchpad-less!

You may find the blog post What to Check Out in Scala Plugin 1.4.x for IntelliJ IDEA 14 & 14.1 helpful, too.

Side note It came as a complete surprise to me to have noticed that I’ve been writing the blog post exactly a month after the last one.

Why I’m using IntelliJ IDEA to learn Scala?

I’m using IntelliJ IDEA daily.

I begin a day switching to the desktop where the IDE awaits my attention and keep it open (until a mandatory reboot following a system update). I was using other IDEs – NetBeans IDE or Eclipse IDE – in the past to develop applications in Java or Java EE, but things have changed since I switched focus on Scala entirely.

The reason for the switch was to master the Scala language not the other available IDEs, and given IntelliJ IDEA have always been receiving positive marks it’s with me nowadays. When I need a full-blown IDE, it’s IntelliJ IDEA with the Scala plugin. Period.

There’s another tool that supports learning Scala beautifully – Scala REPL. However it’s often too rudimentary and limiting, for quick rendezvous I prefer it with Sublime Text 3 and sbt. For more advanced sessions nothing beats the beloved IDE - IntelliJ IDEA.

I think it was Tomasz Nurkiewicz – an IntelliJ IDEA expert – who first showed the beauty of using IntelliJ IDEA mouse- and touchpad-less. Thanks Tomek!

Minimalistic workspace

I remember the tweet from Adrian Gruntkowski very well when the need to go minimalistic was first planted in my head. Adrian mentioned a user guide to set up a minimalistic workspace in IntelliJ IDEA (though it was for CursiveClojure) and the story began.

It took me a while to get used to it, but it was worth it! I just needed a mixture of Sublime Text 3 and IntelliJ IDEA as the Scala development environment and think I’ve found mine already.

A day with Scala and IntelliJ IDEA

Be warned that I’m working on Mac OS X with Mac OS X 10.5+ keymap so your milleage may vary.

The blog post assumes you’ve got a Scala project imported or created from scratch already. I don’t bother explaining how to do it. In either case, IntelliJ IDEA should open with a Scala project so switching between files makes sense.

Minimalistic IntelliJ IDEA

Follow CursiveClojure UI to have a minimalistic, clutter-free workspace. It’s a good start and boils down to turning off the toolbars, deselecting Toolbar and Navigation Bars in the View menu and finally disabling the Editor Tabs. That’s a very good start.

Start by pressing Cmd + Ctrl + F to enter full screen.

You should have the IDE looked like as in the following screenshot.

Press Shift key twice to open Search everywhere popup.

You may also want to use Cmd + Shift + A to search for actions only.

Type in pre mod or (better) preMod to execute Enter Presentation Mode action. In IDEA 14.1 there’s far more productive mode - Distraction Free Mode so type in distMod instead.

Either way – Presentation or Distraction mode – you’ve got a clean desk and should concentrate on Scala much easier (with the goodies of IntelliJ IDEA at your fingertips).

Following the advice in the empty workspace of IntelliJ IDEA, use Cmd + E to switch between files or Cmd + Shift + E to switch between files that were recently edited.

Use Cmd + O to open traits or classes. Use Cmd + Shift + O to open any file like build.sbt or project resources.

And the last but not least, install BashSupport plugin to have a fully-supported terminal inside IntelliJ IDEA. Use Alt + F12 to open a terminal session. I use it to have sbt shell open for ~ test to have the tests executed every time the main and test sources change.

I use the terminal to open Scala REPL when I need to try out a new API.

There are also Cmd + KeyUp to select files from other directories in a project, or just switch to Project view with Cmd + 1. You could use Alt + F1 to select the target to view the currently open file.

Summary

There are plenty of ways and tools to help you learn Scala in a pleasant environment. I’ve been using Sublime Text 3 and sbt for quite some time, and found myself very productive with this combo. In my case, though, enough turned out to be lazy to learn more advanced developer tools, i.e. IntelliJ IDEA.

Once I switched to IntelliJ IDEA and started using the features like Full Screen and Distraction Free modes with proper keystrokes, it became the development environment of choice to get full steam ahead into Scala.

There ain’t no such thing as a free lunch and it does take time to hop onto a new tooling and change habits. New things can often be getting into your way until you find them useful. As much as habits can help (speeding things up), they should not rule out others and mark them worse by default. As Learn, Unlearn And Relearn: How To Stay Current And Get Ahead says:

Whatever the reasons, once the basics are covered, many people tend to stick with what they know and avoid situations or challenges where they may mess up or be forced to learn something new, thus creating a safe, secure and comfortable (and confining) world for themselves.

Give the tips from the blog post a try and a month later you will have found they’re as pleasant as yours. Or bring you even more joy! Let me know how it’s worked out in the Comments section below.

Oh-My-Zsh is an open source, community-driven framework for managing your ZSH configuration. It comes bundled with a ton of helpful functions, helpers, plugins, themes, and a few things that make you shout…“Oh My ZSH!”

As I’m using the ZSH configuration framework so often that I can’t believe I could’ve lived without it for so long here’s a collection of the plugins to cut the number of keystrokes while working with git repositories. To my great surprise there are quite a few plugins for git and the list below is my humble summary to get it remembered as much as to spread a word (as I’ve been meeting people unaware of Oh-My-Zsh far too often).

FIXME Consider the blog post complete after the line’s gone. It’s always in readable state, though.

Oh Mom, there are 10 plugins for git!

I was very surprised to have been presented with 10 plugins for git when I hit TAB to complete the ~/.oh-my-zsh/plugins/git path:

• git = (quoting README.md) this plugin adds several git aliases and increase the completion function provided by zsh
• git-extras = ??? (aka unsure what it does - yet to be discovered)
• git-flow = (quoting the plugin’s sources) git-flow completion nirvana
• git-flow-avh = unsure how it’s different from the git-flow plugin
• git-hubflow = unsure how it’s different from the git-flow plugin
• git-prompt = (quoting the plugin’s sources) ZSH Git Prompt Plugin
• git-remote-branch = ??? (aka unsure what it does - yet to be discovered)
• gitfast = ??? (aka unsure what it does - yet to be discovered)
• github = ??? (aka unsure what it does - yet to be discovered)
• gitignore = ??? (aka unsure what it does - yet to be discovered)

I just assume they are all plugins to ease my work with git repositories and am going to review them all one by one and amend the blog post afterwards.

The commands I’ve managed to master so far:

• gaa = git add --all
• gco = git checkout
• glo = git log --oneline --decorate --color
• glgga = git log --graph --decorate --all
• gcm = git checkout master
• gd = git diff
• gdc = git diff --cached
• gc! = git commit -v --amend
• gcp = git cherry-pick
• gl = git pull
• gp = git push
• gst = git status
• grba = git rebase --abort
• grbi = git rebase -i
• grbc = git rebase --continue
• grh = git reset HEAD
• grhh = git reset HEAD --hard
• grv = git remote -v
• gwc = git whatchanged -p --abbrev-commit --pretty=medium

Read the official Plugins page to learn how to enable plugins in your configuration.

The configuration of mine includes the following plugins, git including:

➜  ~  grep -e "plugins=(" ~/.zshrc | grep -e "^[^#]"
plugins=(git osx brew common-aliases)

I’m totally aware that there’s plenty of room for improvement here. Let me know what I’m missing (and where I’m wasting my time still). I’d wholeheartedly appreciate any time savings.

As there are a lot of REST interactions – sending requests to and receiving them from REST microservices – as much as I’m not really in need to comprehend every bit of the communication format, I do need tools to work with them effectively, i.e. be able to create and consume HTTP packets with JSON payload with little to no effort.

HTTPie (pronounced aych-tee-tee-pie)

I’ve been using curl for so long that I hardly remember when it all began. It’s a very handy tool, alas too complex at times.

Not so long ago I stumbled upon another command-line tool called HTTPie. Installation on MacOS X with brew was just brew install httpie away.

➜  ~  brew install httpie
Warning: httpie-0.9.1 already installed

HTTPie describes itself as:

HTTPie (pronounced aych-tee-tee-pie) is a command line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. It provides a simple http command that allows for sending arbitrary HTTP requests using a simple and natural syntax, and displays colorized responses.

The tool’s so simple and intuitive that I’m still fighting with my thoughts to believe that people could still be using curl. I for one was using curl for so long that I almost lost the ability to sense how much pain I was experiencing with curl. I did get used to it and didn’t notice I’m sufferring an enormous pain.

GET requests are pretty similar in both tools (still think httpie is simpler).

http -v --json localhost:9000/api/person

Where the difference shows up is to send POST requests with JSON payload as application/json (-v is to verbose the communication):

➜  ~  http -v localhost:9000/api/person name=jacek
POST /api/person HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 17
Content-Type: application/json; charset=utf-8
Host: localhost:9000
User-Agent: HTTPie/0.9.1

{
    "name": "jacek"
}

HTTP/1.1 200 OK
Content-Length: 21
Content-Type: application/json; charset=UTF-8
Date: Thu, 12 Feb 2015 09:35:46 GMT
Server: tmprl

{
    "name": "jacek"
}

First, I don’t have to express it explicitly that I’m going to send a POST request since the key=value pair (in the example earlier name=jacek, and there could be more) implies so. Less typing, less errors.

Second, I don’t have to build the JSON payload myself, key=value pairs are enough to inform the tool to create one for me. Less typing, again.

And the last but not least, colorized responses (aka formatted and colorized terminal output). That could’ve made a difference and it did!

Just these three features made httpie an indispensable tool for me.

httpie has changed how I test my REST microservices using JSON now. As is JSON to REST microservices, so is httpie to the new architecture.

Postman

It appears that command-line tools are not always at a premium, and having few GUI tools might be of help at times.

Just a couple of days ago I saw Postman - REST Client being used by a teammate. And it wouldn’t have been any special if I had not heard about the tool from another person during my presentation about REST microservices with JSON – just a day after I’d seen it the very first time (!) Coincidence? Don’t think so!

Postman is a plugin for Google Chrome so the installation went smoothly. Just a click and it’s done. You should give it a go, too (unless you’ve already done and have few remarks to share in the comments to the blog post - I’d appreciate it very much).

Advanced REST client

What made the situation more interesting was that even though Postman was mentioned few times here and there, it seems that Advanced REST client beats Postman by the number of stars - 6791 vs 2635. Why is so?! It’s over 200%! How could that be that Postman is more famous (in my circles) without Advanced REST client being even mentioned?!

It’s installed and I’m going to give it a whirl, too.

What are your tools to work with REST microservices with JSON as the data format? Let me know in the comments!

Edit ~/.sbt/0.13/plugins/sbt-dependency-graph.sbt so it looks as follows:

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")

Edit ~/.sbt/0.13/global.sbt so it looks:

net.virtualvoid.sbt.graph.Plugin.graphSettings

With these two files, open sbt or activator and execute dependencyGraph (I used hello-slick-specs2 project):

> dependencyGraph
[info] Updating {file:/Users/jacek/dev/oss/hello-slick-specs2/}hello-slick-specs2...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info]                             +---------------------------+
[info]                             |hello-slick-specs2_2.11 [S]|
[info]                             |          default          |
[info]                             |            1.0            |
[info]                             +---------------------------+
[info]                                    |     |   |    |
[info]                ---------------------     |   |    ----------------------------
[info]                |                         |   -----------------               |
[info]                v                         v                   |               |
[info]           +---------+          +------------------+          |               |
[info]           |slf4j-nop|          |  slick_2.11 [S]  |          |               |
[info]           |org.slf4j|          |com.typesafe.slick|          |               |
[info]           |  1.7.7  |          |      2.1.0       |          |               |
[info]           +---------+          +------------------+          |               |
[info]                |                   |   ||      |             |               |
[info]      -----------                   |   ||      ---------     |               |
[info]      |  ----------------------------   ||              |     |               |
[info]      |  |             ------------------|              |     |               |
[info]      |  |             |                 |              |     |               |
[info]      v  v             v                 v              v     v               v
[info]  +---------+ +-----------------+ +------------+ +------------------+ +--------------+
[info]  |slf4j-api| |    slf4j-api    | |   config   | |  scala-library   | |      h2      |
[info]  |org.slf4j| |    org.slf4j    | |com.typesafe| |  org.scala-lang  | |com.h2database|
[info]  |  1.7.7  | |      1.6.4      | |   1.2.1    | |      2.11.1      | |   1.4.182    |
[info]  +---------+ |evicted by: 1.7.7| +------------+ |evicted by: 2.11.4| +--------------+
[info]              +-----------------+                +------------------+
[info] Note: The old tree layout is still available by using `dependency-tree`
[success] Total time: 0 s, completed Nov 29, 2014 11:19:30 PM

Neat, isn’t it?

You may also want to execute dependencyGraphMl:

> dependencyGraphMl
[info] Wrote dependency graph to '/Users/jacek/dev/oss/hello-slick-specs2/target/dependencies-compile.graphml'
[success] Total time: 0 s, completed Nov 29, 2014 11:21:46 PM

Install yEd and open the graph:

> eval "open target/dependencies-compile.graphml" !
[info] ans: Int = 0

I really wish I’d known it earlier. It’d surely have saved me a lot of time.

So here they are, loose notes about Cassandra to understand the value propositions of the database and where it could fit well. Should I ever be faced the question of using Cassandra or not, I may some day find an answer here (or know where to look for it).

My story with Cassandra began with the Spark Cassandra Connector project that enables Apache Spark to use Cassandra to have lightning-fast cluster computing with Spark and Cassandra.

The latest version of Cassandra is 2.1.0.

Step 1. Installation

I installed Cassandra using brew on Mac OS X. It was a mere brew install cassandra and took a few secs.

➜  ~  brew info cassandra
cassandra: stable 2.1.0
http://cassandra.apache.org
/usr/local/Cellar/cassandra/2.1.0 (3899 files, 92M) *
  Built from source
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/cassandra.rb
==> Caveats
If you plan to use the CQL shell (cqlsh), you will need the Python CQL library
installed. Since Homebrew prefers using pip for Python packages, you can
install that using:

  pip install cql

To have launchd start cassandra at login:
    ln -sfv /usr/local/opt/cassandra/*.plist ~/Library/LaunchAgents
Then to load cassandra now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist

Execute sudo pip install cql to install the Python packages.

➜  ~  sudo pip install cql
Downloading/unpacking cql
  Downloading cql-1.4.0.tar.gz (76kB): 76kB downloaded
  Running setup.py (path:/private/tmp/pip_build_root/cql/setup.py) egg_info for package cql

Requirement already satisfied (use --upgrade to upgrade): thrift in /Library/Python/2.7/site-packages (from cql)
Installing collected packages: cql
  Running setup.py install for cql

Successfully installed cql
Cleaning up...

That should be all needed to get you going.

Step 2. Running the server

Start an instance using cassandra -f.

➜  ~  cassandra -f
objc[43878]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/Current/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
INFO  22:49:39 Hostname: japila.local
INFO  22:49:39 Loading settings from file:/usr/local/etc/cassandra/cassandra.yaml
...
INFO  22:47:52 Using Netty Version: [netty-buffer=netty-buffer-4.0.20.Final.1709113, netty-codec=netty-codec-4.0.20.Final.1709113, netty-codec-http=netty-codec-http-4.0.20.Final.1709113, netty-codec-socks=netty-codec-socks-4.0.20.Final.1709113, netty-common=netty-common-4.0.20.Final.1709113, netty-handler=netty-handler-4.0.20.Final.1709113, netty-transport=netty-transport-4.0.20.Final.1709113, netty-transport-rxtx=netty-transport-rxtx-4.0.20.Final.1709113, netty-transport-sctp=netty-transport-sctp-4.0.20.Final.1709113, netty-transport-udt=netty-transport-udt-4.0.20.Final.1709113]
INFO  22:47:52 Starting listening for CQL clients on localhost/127.0.0.1:9042...
INFO  22:47:52 Binding thrift service to localhost/127.0.0.1:9160
INFO  22:47:52 Listening for thrift clients...

Step 3. Using cqlsh

In another terminal, start the Cassandra shell using cqlsh:

➜  ~  cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh>

You may face the issue Why does cqlsh fail with LookupError: unknown encoding? that can be easily solved with setting up LC_ALL appropriately. Mine’s pl_pl.utf-8:

export LC_ALL=pl_pl.utf-8

Copying Step 4: Using cqlsh from the official Getting Started documentation:

CREATE KEYSPACE mykeyspace
WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

USE mykeyspace;

CREATE TABLE users (
  user_id int PRIMARY KEY,
  fname text,
  lname text
);

INSERT INTO users (user_id,  fname, lname)
  VALUES (1745, 'john', 'smith');
INSERT INTO users (user_id,  fname, lname)
  VALUES (1744, 'john', 'doe');
INSERT INTO users (user_id,  fname, lname)
  VALUES (1746, 'john', 'smith');

SELECT * FROM users;

I’m currently using Play’s Anorm and writing the queries myself didn’t seem to be something I much liked. I’ve been seeing Slick has had a good press so it was a no-brainer to pick it as an alternative.

Mistake #1. Hurting myself with cutting edge versions

Reading the documentation at http://slick.typesafe.com/ was not very informative as there was no word about how to embed it in a Play Framework application. There should really be a “Getting started with Slick and Play Framework” tutorial since both are part of The Typesafe Reactive Platform.

I added Slick to build.sbt as follows:

"com.typesafe.slick" %% "slick" % “2.1.0"

And then I realised that I’d have had to do:

Database.forURL("jdbc:h2:mem:test1", driver = "org.h2.Driver") withSession {
  implicit session =>
  // <- write queries here
}

but Play Framework gives me:

DB.withConnection("sayenedb") { implicit c =>
    ...
}

It was certainly not the way to follow. I needed a solution that would read database configuration from Play’s.

Mistake #2. Using play-slick with Play 2.4.0-M1

Time has come to give play-slick a serious try. The project aims to make “Slick a first-class citizen of Play 2.x.”

It took me a while to abandon the idea of running play-slick with Play 2.4.0-M1 since this:

The purpose of this release is to get feedback about the approach to dependency injection that we’re implementing in Play 2.4. The old Play plugins mechanism is going to be deprecated. For a detailed overview of the different styles of DI available in Play 2.4, please read here:

https://www.playframework.com/documentation/2.4.x/ScalaDependencyInjection https://www.playframework.com/documentation/2.4.x/ScalaCompileTimeDependencyInjection https://www.playframework.com/documentation/2.4.x/JavaDependencyInjection

I reported an issue for play-slick hoping that the developers notice the missing integration point and the support for 2.4 will come…sooner (?)

Mistake #3. Using custom db configuration in Play

The application uses no db.default configuration in application.conf in Play - just a custom one. The result?

[error] application -

! @6jg39b0pk - Internal server error, for (GET) [/tips/wgcategories] ->

play.api.Configuration$$anon$1: Configuration error[Slick error : jdbc driver not defined in application.conf for db.default.driver key]
     at play.api.Configuration$.play$api$Configuration$$configError(Configuration.scala:94) ~[play_2.11-2.3.4.jar:2.3.4]
     at play.api.Configuration.reportError(Configuration.scala:743) ~[play_2.11-2.3.4.jar:2.3.4]
     at play.api.db.slick.Config$$anonfun$1.apply(Config.scala:64) ~[play-slick_2.11-0.8.0.jar:0.8.0]
     at play.api.db.slick.Config$$anonfun$1.apply(Config.scala:64) ~[play-slick_2.11-0.8.0.jar:0.8.0]
     at scala.Option.getOrElse(Option.scala:120) ~[scala-library-2.11.2.jar:na]

I had to add the following entries to work it around:

db.default.driver=org.postgresql.Driver
db.default.url=${?DB_CONN}

DB_CONN is the property I set up at command line at Play startup.

Using two datasources required following Advanced drivers configuration and applying the (in)famous Cake pattern.

Mistake #4. Copying examples to production code - PostgreSQL is case sensitive for field names in quotes

I had the following in my Component:

def id = column[Int]("ID", O.PrimaryKey, O.AutoInc)

That generated a query with "ID" in select clause that in turn resulted in the following error:

STATEMENT:  select s13."ID", s13."name" from “xxx" s13;
ERROR:  column s13.ID does not exist at character 8

As Rene pointed out in his tweet:

@jaceklaskowski not true - #PostgreSQL follows SQL standard. Columns names are case insensitive unless created with quotation marks

He was right - changing ID to id has indeed fixed the issue.

def id = column[Int]("id", O.PrimaryKey, O.AutoInc)

Mistake #5. [SI-3664] Explicit case class companion does not extend Function / override toString

There’s an issue reported against the Scala compiler - [SI-3664] Explicit case class companion does not extend Function / override toString - that stands in a way for the * projection, i.e. def * = ..., in your table description:

class Users(tag: Tag) extends Table[User](tag, "users") {
    def id    = column[Int]   ("id", O.PrimaryKey, O.AutoInc)
    def login = column[String]("login")

    def * = (id, login) <> (User.tupled, User.unapply)
}

For the Users class above the Scala compiler fails reporting:

value tupled is not a member of object model.User

A solution is described in Mapped Tables section of UPGRADE GUIDES document in the Slick manual:

Note that .tupled is only available for proper Scala functions.
When using a case class, the companion object extends the correct function type by default, but only if you do not define the object yourself. In that case you should provide the right supertype manually.

The mapping definition can look as follows:

class Users(tag: Tag) extends Table[User](tag, "users") {
    def id    = column[Int]   ("id", O.PrimaryKey, O.AutoInc)
    def login = column[String]("login")

    def * = (id, login) <> ((User.apply _).tupled, User.unapply)
}

Note (User.apply _).tupled.

Mistake #6. JodaTime support

For cases where you need to use JodaTime types in Slick you should use slick-joda-mapper.

Else you have to stick to java.sql.Date, java.sql.Time, java.sql.Timestamp as described in Table Rows in the Slick documentation.

Summary

Using a non-default configuration is always a kind of minefield. Stay away from it unless you’re adventurous and have enough time and patience to fix issues along the way. You’ve been warned.

So, here it is. The yet another tutorial series about developing web applications in Play Framework (with Scala) and AngularJS (in JavaScript). Let’s get rolling!

NOTE It’s a work in progress. Watch this space until the note has disappeared and the blog post become feature-complete.

Step 1. Installing Typesafe Activator

Download Typesafe Activator from http://typesafe.com/activator and install it in a directory. Add the directory to PATH so you’ll be able to execute activator from any place in your file system without having to use the fully-qualified path.

Execute activator ui to open up Activator UI in a browser. Go to http://localhost:8888.

See what’s available in the UI and once you’re satisfied, go to the command line where activator ui is running and press Ctrl+C to stop the UI process.

Step 2. Creating hello-play-tutorial web application

In a directory of your choice, execute activator new hello-play-tutorial play-scala to create a web application using the Play Scala Seed template.

Pro-tip: Read the output from the command so you learn what you can do with activator that I’m not going to cover in the article.

➜  sandbox  activator new hello-play-tutorial play-scala

Fetching the latest list of templates...

OK, application "hello-play-tutorial" is being created using the "play-scala" template.

To run "hello-play-tutorial" from the command line, "cd hello-play-tutorial" then:
/Users/jacek/sandbox/hello-play-tutorial/activator run

To run the test for "hello-play-tutorial" from the command line, "cd hello-play-tutorial" then:
/Users/jacek/sandbox/hello-play-tutorial/activator test

To run the Activator UI for "hello-play-tutorial" from the command line, "cd hello-play-tutorial" then:
/Users/jacek/sandbox/hello-play-tutorial/activator ui

Change the working directory to hello-play-tutorial and run activator run. Wait until the message (Server started, use Ctrl+D to stop and go back to the console...) has showed up.

➜  sandbox  cd hello-play-tutorial
➜  hello-play-tutorial  activator run
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/sandbox/hello-play-tutorial/project
[info] Updating {file:/Users/jacek/sandbox/hello-play-tutorial/project/}hello-play-tutorial-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to hello-play-tutorial (in build file:/Users/jacek/sandbox/hello-play-tutorial/)
[info] Updating {file:/Users/jacek/sandbox/hello-play-tutorial/}root...
[info] Resolving jline#jline;2.11 ...
[info] Done updating.

--- (Running the application from SBT, auto-reloading is enabled) ---

[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Ctrl+D to stop and go back to the console...)

Visit http://localhost:9000 to open up the web application in a browser.

Pro-tip: Read the content of the welcome page so you know what the sample application offers beyond what is included in the article.

Step 3. (optional) Deploying to a cloud service - CloudBees

According to the official documentation of Play Framework in Deploying to Cloudbees:

CloudBees support Play dists natively - with Jenkins and continuous deployment

that is in turn confirmed in the official documentation of CloudBees in RUN@cloud » Play Framework:

CloudBees includes first-class support for running Play! applications in the Cloud.

Install CloudBees SDK. On Mac OS X it’s just that easy as brew install cloudbees-sdk.

Start with bees init and provide necessary configuration information before moving on to deploying the Play application.

Execute bees app:create hello-play-tutorial -t play2 to configure the application on CloudBees.

➜  hello-play-tutorial git:(master) bees app:create hello-play-tutorial -t play2
Application: jaceklaskowski/hello-play-tutorial
    url: hello-play-tutorial.jaceklaskowski.eu.cloudbees.net

Create a new git repository under Repos in the CloudBees Administrative Console and execute git init followed by git add -am 'Initial commit' in the directory.

➜  hello-play-tutorial git:(master) git remote add cloudbees https://git.cloudbees.com/jaceklaskowski/hello-play-tutorial.git
➜  hello-play-tutorial git:(master) git push --mirror cloudbees
Counting objects: 30, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (25/25), done.
Writing objects: 100% (30/30), 1010.69 KiB | 0 bytes/s, done.
Total 30 (delta 0), reused 0 (delta 0)
To https://git.cloudbees.com/jaceklaskowski/hello-play-tutorial.git
 * [new branch]      master -> master
➜  hello-play-tutorial git:(master) activator clean dist
...
➜  hello-play-tutorial git:(master) bees app:deploy -t play2 -a hello-play-tutorial target/universal/hello-play-tutorial-1.0-SNAPSHOT.zip
Deploying application jaceklaskowski/hello-play-tutorial (environment: ): target/universal/hello-play-tutorial-1.0-SNAPSHOT.zip
Application parameters: {containerType=play2}
........................uploaded 25%
........................uploaded 50%
........................uploaded 75%
........................upload completed
deploying application to server(s)...
Application jaceklaskowski/hello-play-tutorial deployed: http://hello-play-tutorial.jaceklaskowski.eu.cloudbees.net

Access http://hello-play-tutorial.jaceklaskowski.eu.cloudbees.net (mind that yours is different) to see the application deployed and running.

Set up a Jenkins job so the application’s deployed every git push cloudbees.

In this installment, I’m presenting a sbt build for learning the JSON API from the play-json module in the play.api.libs.json package in Play Framework 2.4.0-M1.

It’s a code-mainly version of the article JSON Macro Inception from the official documentation of Play Framework.

Start a new activator/sbt project with the following build definition in build.sbt:

scalaVersion := "2.11.2"

val playVersion = "2.4.0-M1"

libraryDependencies += "com.typesafe.play" %% "play-json" % playVersion

On the command line execute sbt and then, while in the sbt shell, console.

> console
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.11.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_20).
Type in expressions to have them evaluated.
Type :help for more information.

When you see the above (the Java version might be different depending on your configuration), you’re in an interactive development environment - Scala REPL - with play-json library and Scala 2.11.2. The world of JSON’s (almost) yours!

:pa (a shortcut for :paste) enters paste mode so you can copy and paste entire Scala statements.

scala> :pa
// Entering paste mode (ctrl-D to finish)

import play.api.libs.functional.syntax._
import play.api.libs.json._

// Exiting paste mode, now interpreting.

import play.api.libs.functional.syntax._
import play.api.libs.json._

scala> case class Person(name: String, age: Int, lovesChocolate: Boolean)
defined class Person

scala> :pa
// Entering paste mode (ctrl-D to finish)

implicit val personReads = (
  (__ \ 'name).read[String] and
  (__ \ 'age).read[Int] and
  (__ \ 'lovesChocolate).read[Boolean]
)(Person)

// Exiting paste mode, now interpreting.

personReads: play.api.libs.json.Reads[Person] = play.api.libs.json.Reads$$anon$8@5c2b898d

scala> val jsonStr = """{ "name" : "Jacek", "age" : 41, "lovesChocolate": true }"""
jsonStr: String = { "name" : "Jacek", "age" : 41, "lovesChocolate": true }

scala> val json = play.api.libs.json.Json.parse(jsonStr)
json: play.api.libs.json.JsValue = {"name":"Jacek","age":41,"lovesChocolate":true}

scala> val jacek: Person = json
<console>:18: error: type mismatch;
 found   : play.api.libs.json.JsValue
 required: Person
       val jacek: Person = json
                           ^

scala> val jacek: Person = json.as[Person]
jacek: Person = Person(Jacek,41,true)

scala> implicit val personReads = Json.reads[Person]
personReads: play.api.libs.json.Reads[Person] = play.api.libs.json.Reads$$anon$8@5e930aa2

With the playground you can play with JSON types in Play however you like. Start with the trait play.api.libs.json.JsValue and then learn what play.api.libs.json.Reads[T] offers. They’re the cornerstone of the JSON API in Play.

The entire code to paste (:pa or :paste in sbt console) follows. Note the simplifications codenamed JSON Inception.

import play.api.libs.json._

case class Person(name: String, age: Int, lovesChocolate: Boolean)

val jsonStr = """{ "name" : "Jacek", "age" : 41, "lovesChocolate": true }"""

val json = play.api.libs.json.Json.parse(jsonStr)

implicit val personReads = Json.reads[Person]

val jacek: Person = json.as[Person]

Once you’re done, press Ctrl+D twice to exit console and the sbt shell afterwards.

In order to pursue my understanding of sbt (and hence Scala itself) I’ve been reading the sources that honestly keep surprising me so much often. It’s almost every minute when I find myself scratching my head to digest a piece of sbt code. It’s akin to when I was reading the source code of Clojure to learn the language. People can write complicated code and I wouldn’t be surprised to hear sbt’s sources belong to the category. I don’t care, though. I’m fine with the complexity hoping the mental pain brings me closer to master Scala.

Today I picked the trait sbt.Init believing it’d be an important step in my journey.

NOTE It becomes feature-complete when the note disappears. Live with the few mistakes for now. Let me know what you think in the Comments section. The site is on GitHub so pull requests are warmly welcome, too. Thanks!

There’s the trait sbt.Init. I don’t really know what its purpose is and I hope to find it out after few Scala snippets. There’s just enough hope to master Scala while pursuing my understanding of sbt with the trait.

Goal

Create an instance of trait Init[Scope].

Solution

1
2
3
4
5

val init = new Init[Int] {
  def showFullKey: Show[ScopedKey[_]] = Show { (sk: ScopedKey[_]) => 
    s"${sk.scope}:${sk.key}...${sk.scopedKey}"
  }
}

Run sbt and then execute the command consoleProject to open sbt’s Scala REPL with all the necessary types of sbt loaded.

Mental issues encountered

1. I’m far from being able to distinguish easily type parameters, e.g. Scope, in parameterised types, e.g. Init[Scope], from types themselves. When I see Init[Scope] my Java-trained eyes see Scope type within Init type and although it doesn’t make sense after a moment that’s my initial thought.

2. The type constructor Show[ScopedKey[_]] in the return value type of showFullKey is another trait Show that comes with apply that is supposed to return a String instance from ScopedKey[_]. But hey, ScopedKey[_] is another type constructor, and things get more complex for me again. Happily, Show has a companion object with apply method. The story ends as ScopedKey is a final parameterized case class and the function parameter f: T => String in Show returns a String so I’ve just merely followed the types and it happened to work fine. The Scala compiler happy and so am I.

Summary

Show is a function type (with apply) that accepts T and returns String. In our case, T is ScopedKey[_] that’s…well…it’s yet to be understood.

consoleProject in sbt

If you happened to want to see the code in action, execute sbt consoleProject and give the following a try:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// (attribute) key that points at Int value
scala> val number = AttributeKey[Int]("number", "number stringified")
number: sbt.AttributeKey[Int] = number
 
scala> val init = new Init[Int] {
     |   def showFullKey: Show[ScopedKey[_]] = Show { (sk: ScopedKey[_]) =>
     |     s"${sk.scope}:${sk.key}...${sk.scopedKey}"
     |   }
     | }
init: sbt.Init[Int] = $anon$1@1f95802
 
scala> val sfk: Show[init.ScopedKey[_]] = init.showFullKey
sfk: sbt.Show[init.ScopedKey[_]] = sbt.Show$$anon$1@7f54be72

scala> val s = sfk(init.ScopedKey[Int](scope=5, key=number))
s: String = 5:number...ScopedKey(5,number)

I don’t want to keep the techniques for myself so the git/GitHub cheat sheet is supposed to help me remember the commands and others to learn from my mistakes (aka experience). It’s so easy on GitHub that I keep wondering why it took me so long to learn it. It must not for you.

Have fun contributing to open source projects as much as I do! Pro publico bono.

NOTE It becomes feature-complete when the note disappears. Live with the few mistakes for now. Let me know what you think in the Comments section. Pull requests are welcome, too. Thanks!

Picking project

You start your day hunting down the project you want to contribute to. Be adventurous and pick the one you’ve always been dreaming about. This might be the day when the dream comes true.

I’m more into Scala/sbt lately so I’m with projects under control of the project build tool - sbt as I can learn both contributing.

Cloning project

Learning a project can take different approaches and reading the source code or just building it and staying on the cutting edge are a few examples.

In the project’s repository on GitHub, on the right-hand side, there’s this clone URL field. Select the protocol to use (HTTPS or SSH) and click the Copy to clipboard button.

In the terminal, execute the following command:

git clone [clone URL]

It creates a directory with the project. The sources are yours now, master.

Forking project

Your very first step is to fork a project. Forking means creating your own copy of the project. On GitHub it’s so easy with the Fork button in the upper-right corner. Click it and select the account you want the fork go.

In the terminal, go to the project’s directory and add the repository as a remote repository.

git remote add [remote-name] [clone URL]

I tend to use my first name for remote-name so I know that my personal repository copy is under jacek nick.

Branching project

Developing a change for a project is the real thing. It can be a documentation page, a fix for an issue or whatever else the project holds.

The following command

git checkout -b [branch-name]

creates and changes your current branch from master (usually) to branch-name. Use wip/ in the branch-name to denote that the work is in progress so people can review the changes before they get squashed and merged with the master.

Committing changes to project

On a branch, go the following to commit the changes of yours:

git commit -am [commit-message]

There are some strict rules on how to write a proper commit-message. For now, don’t worry about it too much. There are tougher things you will have to go through and writing proper commit messages don’t belong to that category…yet. It’s more important to get you up to speed with contributing to a project than to do it without mistakes from the day 0.

Pushing changes to remote repo

With the changes on the branch committed, it’s time to show off on GitHub. Push the changes with the following command:

git push [remote-name] [branch-name]

Using command completion can save you a lot of typing here. A decent shell like oh-my-zsh is highly recommended (on Mac OS X at the very least).

remote-name is the nick of the remote repository, e.g. jacek while branch-name is the name of the branch you’re working on right now.

Creating pull request on GitHub

With the changes in the remote repository on GitHub, you should now be able to send a pull request to the original repo (usually called origin, but git lets you name it whatever you like).

GitHub shows the Pull Request button when you’re changes hit your repository that’s a fork of the project. Click the button and fill out the blanks. GitHub uses your commit message as the title that further easies the process.

Click Create and you’ve just contributed to the project! Open Source Contributor badge unlocked! Congratulations.

Squashing changes

There might be times when your work in progress generates a stream of changes to a branch. It’s assumed that the changes are already git added and the project maintainers have requested to squash the changes so they ultimately go (aka get merged) to the master as a single change/commit (since a branch is usually about a single feature that’s often reasonable to have the feature merged in as a single change – so it’s self-contained and makes code review a little easier).

Use git rebase -i [branch]:

git rebase -i origin/master

where origin/master is the name of the master branch of the project you forked and then branched for your changes from the remote origin repository.

Fix any merge issues while rebasing. When fixed, git add the files changed (because of the merge conflict) and git rebase --continue afterwards.

You can always go back to the previous state (before doing git rebase) with git rebase --abort.

Doing squashing is worth the time since merging the changes with master later on becomes a no-brainer for the project maintainers.

Once you’re done with modifying the history of the changes in your branch, do git push -f to push your changes forcefully. The reason for the -f option is that you make changes to the history of a public branch that others could’ve already featched and based their work on – a conflict may be coming. To prevent the conflict git makes sure that’s what you really want to do. You’ve been warned.

Useful links about git rebase:

• squashing commits with rebase
• About Git rebase
• How to Rebase a Pull Request

Deleting remote and local branches

When the work is over and all the changes are merged with the master, you can safely delete remote and local branches.

Once the work gets merged, GitHub asks you to delete the branch. Click the button under the pull request.

Delete the local branch with the command:

git branch -D [branch-name]

where branch-name is the name of the branch you want to delete.

You should change the branch to some other branch to be able to delete it.

Maintainers, use “Closes #XXX” to auto-close pull requests

It’s a feature of GitHub and mostly for project maintainers when they merging pull requests to master.

After you’re about to git push your local changes, git commit them and as the last line add Closes #XXX where XXX is the pull request id. It will auto-close the pull request.

Useful links about the feature:

• “Merge pull request” Considered Harmful

Say, you bought 589 CHF when it costed 3.4007 PLN and then 593 for 3.3704. How much would you profit when the price of selling CHF rose to 3.4107 PLN?

The question of how much profit you earned with a series of pairs (quantity, price) against a given CHF price can be calculated as follows:

def calculateProfit(series: Seq[(Int, Double)], currentPrice: Double): Double =
  series.foldLeft(0.0) {
    case (acc, (qty, price)) => acc + (currentPrice - price) * qty
  }

val qtyPriceSeries = Seq((589,3.4007),(593,3.3704))
val currPrice = 3.4107

scala> calculateProfit(qtyPriceSeries, currPrice)
res0: Double = 29.787899999999745

It gives you ca 30 PLN.

It’d be nice to have a series with the date when a given pair was made, and then compare it with other means of gaining profits. A web app developed in Play and deployed to Heroku or CloudBees might be of help, wouldn’t it?