I really like the idea that a project should include a compose.yml. New team members should be able to simply git clone ..., docker compose up and then fire up their preferred IDE and then they're off to the races!

Anyway, long story short if all the npm modules you're depending on cooperate you should be able to run npm as follows:

npm install --no-bin-links

And that should work for now. I read a thread that the Bash for Windows team is being consulted on this by the Docker for Windows Beta team - hopefully this won't be a problem for long!

Microsoft has created a Docker image for their Azure command-line client. Here's my setup:

$ docker create --name azure-profile -v /root/.azure microsoft/azure-cli

The Azure cli stores configuration and access tokens in ~/. Unless you want to authenticate each time to you fire up a new session I recommend creating a data volume container for your profile data. Then you can use it to make sure your profile data is persistent like this:

$ docker run -it --rm --volumes-from azure-profile microsoft/azure-cli

This will create an interactive tty session -it in a new container which will be deleted --rm as soon as you exit. But as long as you use --volumes-from your profile data will be persistent.

First let's look at the contents of the ~/.azure directory before we log in:

$ ls ~/.azure

config.json  telemetry.json

Next before we go through the login steps, let's make sure we've correctly setup our volume. Open a second terminal session (make sure to run eval "$(docker-machine env [name])"). Now run inspect to make sure we have the volume right:

$ docker inspect --format='{{json .Mounts}}' [container_id]

[{"Name":"668d3cf8050abe1a87c29f4c1bd2023ecc9c27bec6bbd26d17e0fd24729e8c35","Source":"/var/lib/docker/volumes/668d3cf8050abe1a87c29f4c1bd2023ecc9c27bec6bbd26d17e0fd24729e8c35/_data","Destination":"/root/.azure","Driver":"local","Mode":"","RW":true,"Propagation":""}]

You should see something like the above output. The important part is the Destination should be /root/.azure. If it is you did it right. Otherwise, go back and look for typos.

Back in our interactive container session: We need to login. You may want to review authentication options. As for me I have MFA enabled on my Microsoft account so I use the interactive login once I'm inside my interactive container session:

$ azure login

Follow the directions and you'll be logged in.

Let's inspect our profile directory again:

$ ls ~/.azure

accessTokens.json  azureProfile.json  config.json  telemetry.json

Now if your volumes are correct you can exit your container session (which will remove your container because of the --rm option we specified):

$ exit

Now fire up a new session again:

$ docker run -it --rm --volumes-from azure-profile microsoft/azure-cli

Look inside your ~/.azure directory and it should still have all 4 .json files. Now you're all set. Good luck!

Comment out or remove

fork in run := true

Configure IntelliJ

Create a remote debug configuration:

Run > Edit Configurations > Add New Configuration > Remote

Set Host to localhost and Port to 9999, name your configuration. I called mine "Debug Activator" then click "Apply", the "OK".

Run Activator

From the command line:

$ activator -jvm-debug 9999

Once that starts go back to IntelliJ and start your debug configuration. Then return to the command line prompt and execute:

$ run

You don't have to do the command line in two steps. That's just if you want to debug the application startup. Otherwise this will work just fine:

$ activator run -jvm-debug 9999

Then just set a breakpoint and make a request.

Well all hail open source! Because I found pinfinder on Github. A nifty little go program that will scan your unencrypted backups and pull your pin.

If you've encrypted all your backups, just run a new non-encrypted one and then delete it afterwards.

Just download the latest release for your platform, extract the archive and run it from the terminal:

$ pinfinder

That's it!

Below are all the points, what time to find them in the video and what I found were the important talking points. I left most stuff actually on the slide out.

1. One Actor Is No Actor

(about 3:17)

Semantically, a system with a single actor is not a concurrent system.

Use ActorRef over actor selection

2. Structure Your Actors

(about 6:13)

Prefer context.actorOf over system.actorOf. This will create a parent/child relationship which allows for actor supervision. system.actorOf creates all new actors on the same 'level' instead of within a hierarchy.

3. Name Your Actors

(about 7:22)

Allows actor names to have meaning within the domain:

• Default naming: bad - no context when logging/debugging
• [name]-[counter]: better - but still lacks important meaning. Example: fetch-worker-1, fetch-worker-2
• [name]-[contextual-id]: best - combines meaningful name with the execution context. Example for actor which fetches videos from web sites: fetch-yt-[youtube-identifier], fetch-vm-[vimeo-identifier]

Note: Don't use scala string interpolation for logging:

//BAD - the string is always built/evaluated
log.debug(s"Something heavy $generateId from $physicalAddress")

//GOOD - string built only when DEBUG level is ON
log.debug("Something heavy {} from {}", generateId, physicalAddress)

4. Matrix Of Mutability (Pain)

(about 11:49)

(Lower number is better):

1. Immutable Val - great, but you can't do anything with it
2. Immutable Var - most of the time, you can mutate it but if it gets "leaked" (sent to another actor) it can't be modified (thread-safe).
3. Mutable Val - shouldn't do this. Not thread-safe.
4. Mutable Var - never do this

5. Blocking Needs Careful Management

(about 14:25)

When you block - isolate it - use a non-defualt dispatcher on it's own thread pool. Aka "Bulkhead Pattern"

When configured in your config file like this:

// application.conf

my-blocking-dispatcher {
  type = Dispatcher
  executor = "thread-pool-executor"

  thread-pool-executor {
    //in akka previous to 2.4.2
    core-pool-size-min = 16
    core-pool-size-max = 16
    max-pool-size-min = 16
    max-pool-size-max = 16

    // or in Akka 2.4.2+
    fixed-pool-size = 16
  }

  throughput = 100
}

Then you can do this:

implicit val blockingDispatcher = system.dispatchers.lookup("my-blocking-dispatcher")

val routes: Route = post {
  complete {
    Future {
      Thread.sleep(5000)
      System.currentTimeMillis().toString
    }
  }
}

This is good because you're blocking only on a dedicated dispatcher instead of the default dispatcher.

6. Never Await, for/FlatMap instead

(about 19:20)

Use Monadic composition of futures (for block) or akka.pattern.after with Future.firstCompletedOf (best)

7. Avoid Java Serialization

(about 22:38)

Java serialization is the default because it made for easy "Hello World" default case for those learning. Problem is people don't change the defaults going into production and decide "Akka is slow".

Good suggestions are ProtoBuf or Kryo. Both have pros/cons. Kryo is easier to setup, but poor schema evolution. ProtoBuf requires more setup but better evolution. Other suggestions are SBE, Thrift, and JSON.

Even XML is smaller/faster than default Java serialization.

7.5. Trust no-one, benchmark everything

(about 29:54)

[all info is on the slide]

8. Let it Crash! Supervision, Failures, & Errors

(about 30:48)

Error - is expected and handled. Report to end user

Failure - an unexpected event. Report to supervisor

Ask who can do something about a problem?

9. Backoff Supervision

(about 35:00)

Exponential Backoff

Persistent actor -> Datbase

If fail/restart is immediate you're basically DDOS your database/system

Instead use Backoff Supervisor, something outside the hierarchy and is provided by Akka if you configure it.

10. Design using State Machines

(about 38:27)

Bad: Long list of case matches for 15 or more messages in an actor's receive looks like "pyramid of doom" found in async programming frameworks like javascript.

Good: You may end up using the FSM (Finite State Machine) DSL to help you achieve this goal.

11. Cluster Convergence and Joining

(about 40:25)

Cluster Gossip Convergence: Everybody knows something, then we make a decision on it.

allow-weakly-up-members: allow "up" (join) state before all nodes can see it. Allows nodes to join even when there is a network partition.

12. Cluster Partitions and "Down"

(about 43:04)

Opposite of last point.

When a node gets marked as unreachable it gets marked as "down". When it comes back everyone refuses to talk to it. If you don't want this behavior you can turn it off, but then make sure you don't use features that follow "single writer principle" (eg. Akka Persistance). Instead use CRDT based like Akka Distributed Data.

13. A fishing rod is a Tool. Akka is a Toolkit

(about 44:54)

Use the least powerful abstraction that gets the job for you. Sometimes a Future is enough but not distributed. Actors are distributed, but require more work - so it's a tradeoff.

Q&A

(about 47:20)

Distributed Akka Streams? Try Intel GearPump:

• https://www.lightbend.com/resources/case-studies-and-stories/intels-gearpump-real-time-streaming-engine-using-akka
• https://software.intel.com/en-us/blogs/2015/08/14/all-about-gearpump-the-real-time-big-data-streaming-engine-behind-intel-s-latest

Backpressure using Backoff Supervisor strategy? No

received unexpected HTTP status: 500 Internal Server Error

Enable Debug Mode

Docker logs wasn't telling me anything. But searching around I read you can enable debug mode -D for Docker. After locating in the ECS documentation how to enable debug mode I did the following on each EC2 instance:

sudo vi /etc/sysconfig/docker

And changed this line:

OPTIONS="--default-ulimit nofile=1024:4096"

To this:

OPTIONS="-D --default-ulimit nofile=1024:4096"

Then saved the file and restarted the Docker daemon:

sudo service docker restart

And restarted both the ECS Agent and my Docker registry container:

sudo docker start [ecs_container_id]
sudo docker start [registry_container_id]

Solution

Then after trying to push an image again I found this in our logs:'

time="2016-04-21T22:18:48Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AccessDenied: Access Denied\n\tstatus code: 403, request id: XXXXXXXXXXXXXXXX" err.message="unknown error" go.version=go1.6.1 http.request.host=[hostname] http.request.id=[UUID value] http.request.method=PATCH http.request.remoteaddr=[IP Address] http.request.uri="/v2/[image_name]/blobs/uploads/[digest]?_state=[data]" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/[hash] kernel/4.1.19-boot2docker os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(windows\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=137.964693ms http.response.status=500 http.response.written=117 instance.id=[UUID value] vars.name=[image_name] vars.uuid=[UUID value] version=v2.4.0

The most helpful bit of the message was:

err.detail="s3aws: AccessDenied: Access Denied\n\tstatus code: 403, request id: XXXXXXXXXXXXXXXX"

Telling me that the container was missing the correct S3 permissions.

If you have a container named my-app-container then you would run:

$ docker exec -it my-app-container bash

To open a bash shell in your container.

As an added bonus, if you're in development and you want to get into your Docker VM then the easiest way is to run:

$ docker-machine ssh [docker_vm_name]

For most of us using the default VM it would look like this:

$ docker-machine ssh default

Once you've opened ssh into your VM, then you can run the docker exec command at the top.

References:

• Docker exec command

$ docker cp [container_id]:[container_path] [localpath]

So to copy a file named output.log to the current working directory from a container named mycontainer you would run:

$ docker cp mycontainer:/my-app-logs/output.log output.log

The command works the other direction as well:

$ docker cp output.log mycontainer:/my-app-logs/output.log

Full details on the command can be found in the Docker cp documentation.

HowToGeek has a great detailed write-up on it including plenty of examples of using the cron pattern. What I wanted to share was how to use I/O Redirection to update the file instead of using the crontab editor (crontab -e) as described in the HowToGeek article.

I wanted to setup a really simple file backup to Amazon S3 on an Amazon EC2 instance. I also wanted to make sure I didn't have to set it up more than once. So I needed something that would work well as an EC2 User Data script:

First, copy crontab to a temp file named 'cron-temp' (arbitrary name):

$ crontab -l > cron-temp

Next, use the append output-redirector >> to write a string to the temp file named 'cron-temp'. If I wanted to overwrite the file completely I could use > instead:

$ echo '[minutes] [hours] [days] [months] [day-of-week] [bash command or script file]' >> cron-temp

In my case I wanted to run the script at the top of every hour so the script actually looked like this see the HowToGeek article for details on cron patterns:

$ echo '0 * * * * aws s3 cp [local file path] s3://[s3-bucket-path]' >> cron-temp

Now that we've updated the temp file replace crontab file with contents of the temp file:

$ crontab cron-temp

Finally, delete the temp file:

$ rm cron-temp

NOTE: Some posts recommend just changing the /lib/systemd/system/docker.service file. This works but the documentation advises against it:

The files located in /usr/lib/systemd/system or /lib/systemd/system contain the default options and should not be edited.

If you'd rather save a few steps you can skip the "create" section below and move ahead to the "configure" section. Other than the paths of the files you'll be editing everything else should be the same.

Create systemd Override Files

If the path /etc/systemd/system/docker.service.d/docker.conf already exists just open the docker.conf file and skip to the next section. If it doesn’t exist, create the directory /etc/systemd/system/docker.service.d directory and docker.conf file:

$ sudo mkdir /etc/systemd/system/docker.service.d
$ sudo touch /etc/systemd/system/docker.service.d/docker.conf
$ sudo vi /etc/systemd/system/docker.service.d/docker.conf

Then add the following contents to docker.conf:

$ sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd://

Configure insecure-registry

Append the --insecure-registry option to the end of the ExecStart options so it looks something like this: (multiple entries follow array convention. See docs)

ExecStart=/usr/bin/docker daemon -H fd:// --insecure-registry myregistry.mydomain.com

Just make sure you replace myregistry.mydoamin.com with the url (and optionally the port if you're not using port 80) for your registry.

Save the file, then flush changes and restart:

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Verify docker daemon is running

$ ps aux | grep docker | grep -v grep

Or Use /etc/default/docker

It may be because using systemd is new or it may be personal preference. But if you want to use the /etc/default/docker file to configure your docker daemon then you just need to change a couple things. First change your /etc/systemd/system/docker.service.d/docker.conf file to look like this:

ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS
EnvironmentFile=-/etc/default/docker

Now you can add the following to your /etc/default/docker file (create it if it doesn't exist) and replace myregistry.mydomain.com with the url (and optionally the port number if it isn't over port 80):

DOCKER_OPTS="--insecure-registry myregistry.mydomain.com"

Again save the file, flush changes and restart just like above.

I have a Docker Registry instance I just configured to run in AWS and is backed by S3 for storage. It's running as a farm of ECS containers behind an ELB. It is running in a private network so I have skipped setting up SSL. Which means I need to configure my docker daemon with the --insecure-registry option so I can pull images to my development machine. And as I mentioned above, I'm currently using the default Boot2Docker image so the configuration is very specific.

You should have a file in your Docker VM: /var/lib/boot2docker/profile. If you open it, it should look something like this:

$ cat /var/lib/boot2docker/profile

EXTRA_ARGS='
--label provider=virtualbox
'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem

To tell Docker to allow you to access an insecure registry you need to add the option to EXTRA_ARGS like so --insecure-registry=[url] like this:

$ sudo vi /var/lib/boot2docker/profile

EXTRA_ARGS='
--label provider=virtualbox
--insecure-registry=myregistry.mydomain.com
'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem

Then just restart the docker daemon:

$ sudo /etc/init.d/docker restart

You should now be able to push and pull using your registry.

sbt (Scala interactive built tool) has a plugin called sbt-native-packager which allows you to configure your project to be built and then packaged as a Docker container. Which means all I have to do is this:

$ sbt docker:publish

And once my build finishes it will create the image and push it to whichever repository I've configured in my build.sbt file.

Except that most everyone on my team uses Windows. Which means that Java and sbt need to be installed on the Docker VM. But every time you restart your VM it gets wiped clean!

Because of this, instead of just using the small, default boot2docker vm that comes packaged so nicely for you with the Docker install everyone goes through the process of installing a full Ubuntu VM and then configuring and installing Docker, Java and sbt.

I'm against the whole give the new guy a Word document with setup instructions. It's part of why I like Docker so much. And this whole setup smells like we're going right back down that path. So today I found a much better way.

Siblings Instead of Nesting

I can't claim credit for this, other than reading the documentation on the Docker-in-Docker (dind) page on Docker Hub. Here's links to the two relevant posts:

• Using the --privileged flag - Docker Run reference
• Using Docker-in-Docker for your CI or testing environment? Think twice - jpetazzo

After reading those two posts I decided there was a much better way, and after learning how to make my VirtualBox shared folders permanent mounts I went looking for a Docker image which already had Java and sbt installed for me.

UPDATE (3/17/16): It was been pointed out that using shared folders is a security concern. This is correct. So let me add a disclaimer: the shared folders aspect of this post was included because I have been helping developers who thought they needed to install/configure all their build tools inside their Docker host in addition to their host OS so they could build their docker images. I'm not advocating shared folders in any other context.

Sharing Docker Binaries

I settled on the excellent 1science/sbt image which is a mere 176 mb (not bad for an image with the Java SDK installed). There were three things I had to figure out to get all this to work together:

First, how to share the Docker binaries with the 1science/sbt container:

$ docker run -it --rm -v "$HOME/.ivy2":/root/.ivy2 -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/bin/docker 1science/sbt sbt sbt-version

Let's break this down a little:

• -it allows us to run the container like tty to an interactive shell. If you left off sbt-version you'd be in the sbt interactive console.

• --rm removes the container as soon as the command is over. This is helpful since we just want to run the command in the foreground and then exit.

• -v "$HOME/.ivy2":/root/.ivy2 maps the Ivy cache to a host volume so the cache will persist between runs and allow our builds to run faster after the first time.

• -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/bin/docker creates two more host mounted volumes that share the Docker binaries with the container. This is the most important piece since it's what allows sbt to build our container instead of just creating a Dockerfile for us to build ourselves.

• $(which docker) is an evaluated statement. In Linux which shows the full path of shell commands - so the output is the path to the docker command. We could look it up, but this makes the run command shorter.

The next step took me the longest to figure out because the README for the 1science/sbt image indicated I needed to something different. So either I was doing it wrong or the docs are wrong. Feel free to correct me we're in this journey together.

$ docker run -it --rm -v "$HOME/.ivy2":/root/.ivy2 -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/bin/docker -v $(pwd):/app 1science/sbt ls

This command doesn't do much more than the last (notice we're only calling ls). But we've added a volume for the root of our project: -v $(pwd):/app. If it worked correctly you should see your project root files listed in the output. This of course only works if your current working directory for the run command is the root of your source project. If not, replace $(pwd) with the path to your project root. It should be the path as seen by Docker, which means it's from the context of your Docker VM.

The reason we needed to map the volume here is because the 1science/sbt image sets /app as the WORKINGDIR for the image. Any commands will be executed from the context of /app which is empty, so we just map our project root to /app and sbt will find everything it needs (almost).

Most of you should be able to run the following now from your project root:

$ docker run -it --rm -v "$HOME/.ivy2":/root/.ivy2 -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/bin/docker -v $(pwd):/app 1science/sbt sbt docker:publish

Or if you're not setup with a central registry yet use publishLocal:

$ docker run -it --rm -v "$HOME/.ivy2":/root/.ivy2 -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/bin/docker -v $(pwd):/app 1science/sbt sbt docker:publishLocal

If your build.sbt file is configured correctly your project will build as a Docker image and publish to your configured registry (Docker Hub is the default). However, if your project is a Play application created using activator then you probably are getting the following error:

sbt.ResolveException: unresolved dependency: com.typesafe.sbtrc#client-2-11;0.3.1: not found unresolved dependency: com.typesafe.sbtrc#actor-client-2-11;0.3.1: not found

This is an open issue due to some funkyness with activator that doesn't work out of the box with sbt. But it's easily fixed. Just add the following to your build.sbt file:

resolvers += Resolver.url("Typesafe Ivy releases", url("https://repo.typesafe.com/typesafe/ivy-releases"))(Resolver.ivyStylePatterns)

And your build will be fixed. Repeat the above run command and this time it should work.

This post is certainly laking in any actual sbt details, but I didn't want it to be too long. I'll work through a very simple example using Scala and post it soon.

• The path used to start with C:\Documents and Settings\[USER_NAME]\
• msbuild blows up when your path is longer than 256 characters
• There's no convenient ~/ command-line shortcut when typing the path on the command-line.

However, that's the only default path you get when you create a boot2docker image for VirtualBox.

In my last post I showed how to mount a VirtualBox shared folder but as soon as you restart (that never happens on a Windows box) you've lost it and have to create the mount all over again.

Mount a shared folder

Just to review, so you don't have to read the last post:

1. Create a shared folder in VirtualBox:

   

2. ssh into your Guest OS (Docker VM):

   $ docker-machine ssh defualt

3. Create a new folder for your mount:

   $ sudo mkdir -p /mnt/src

   The -p is just in case /mnt doesn't already exist.

4. Add your new mount:

   $ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` src /mnt/src

   id -u docker and id -g docker will use the user id and group id respectively. You could just enter 1000 and 50 respectively and you'd be fine. I think the longer version is more clear and safe.

   You should now be able to view the contents of your directory within your Docker VM:

   $ ls /mnt/src

Make It Permanent

The problem is everything gets erased whenever you restart your VM. What we need is a setup script that won't be erased and will run on startup. It turns out it's pretty easy. Create a file named bootlocal.sh at the following location:

$ sudo touch /mnt/sda1/var/lib/boot2docker/bootlocal.sh

Now add the following to the file (no sudo needed):

mkdir -p /mnt/src
mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` src /mnt/src

Make sure you save and close the file. That's it! If you exit your ssh session, then restart your VM:

$ docker-machine restart default

Once it's finished, log back in and verify it's there:

$ docker-machine ssh default
$ ls /mnt/src

You should see the contents of your shared folder.

I thought as I started working with Docker I would need to setup either a separate Linux box and remote into it or install a full-size VM which would steal RAM and disk space and reserve CPU cores. Since I've always worked mostly on a laptop I was concerned about how I would have enough hardware to run everything I needed without always having to free up disk space and memory.

Now that I've been using it I realize how unfounded my fears were. Even though changing to a bare metal Linux install would mean I could get rid of VirtualBox and docker-machine I find once I get started they disappear into the background. I still have to use VirtualBox but the default Docker VM is only using 1GB of memory and a single CPU. Because of this change of perspective I thought it would be nice to share my workflow. It may help someone else and I may get some tips to improve my own.

I am assuming at this point you have already installed Docker by following their quickstart.

The Tools

We're going to use only two command-line utilities here in this walk-through:

• docker: This is the docker client. This is your main tool and what you will use for all things that relate to using and creating Docker images and containers.
• docker-machine: This is for managing the host machines where the Docker daemon is running.

Just running either utility from the command line with no arguments will give you a list of commands and idea what each tool is for. I suggest you take a minute and just read through the output for each of the two and then come back here - I'll wait.

Loading the Docker Environment

When you fire up a fresh bash shell on Windows or OSX before you can start using commands to the docker interface you need to connect it do the docker daemon. Otherwise you'll get the following error:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

As long as you've installed everything this means you forgot to initialize your session. You can check by running:

$ docker-machine ls
NAME      ACTIVE   URL          STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.0

If you notice the - under ACTIVE that means your terminal/console environment isn't configured to connect to the docker daemon. If you're running in terminal on OSX or Git Bash you can fix this by running:

$ eval "$(docker-machine env default)"

Assuming the default installation then you probably have a machine named default (the value under NAME when you run docker-machine ls). Otherwise for the instructions that follow replace default with the name of whatever machine you may have setup in VirtualBox.

If you're using Windows I recommend using Git Bash which is installed along with Docker. Not only is it useful for things like SSH in Windows but as you follow Docker tutorials there will be fewer differences than using Powershell or Command Prompt. For example, I find the default color scheme unreadable when I need to list the contents of a directory in my Docker host (more on that later). I could change it but since I prefer Git Bash I haven't bothered. Also, eval won't work. You'll need to run the following command from either command line:

> docker-machine env default 

And the result will tell you the command you need to run. Here's what that ends up looking like

Powershell

> docker-machine env default
$Env:DOCKER_TLS_VERIFY = "1"
$Env:DOCKER_HOST = "tcp://192.168.99.100:2376"
$Env:DOCKER_CERT_PATH = "C:\Users\[USER_NAME]\.docker\machine\machines\default"
$Env:DOCKER_MACHINE_NAME = "default"
# Run this command to configure your shell:
# & "C:\Program Files\Docker Toolbox\docker-machine.exe" env default | Invoke-Expression

Based on that output you would run:

> docker-machine env default | Invoke-Expression

Windows Command Prompt

> docker-machine env default
SET DOCKER_TLS_VERIFY=1
SET DOCKER_HOST=tcp://192.168.99.100:2376
SET DOCKER_CERT_PATH=C:\Users\[USER_NAME]\.docker\machine\machines\default
SET DOCKER_MACHINE_NAME=default
REM Run this command to configure your shell:
REM     FOR /f "tokens=*" %i IN ('docker-machine env default') DO %i

And here you would run:

FOR /f "tokens=*" %i IN ('docker-machine env default') DO %i

Once you've done that you can run docker-machine ls again and you should now see the following:

NAME      ACTIVE   URL          STATE     URL                         SWARM   DOCKER    ERRORS
default   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.0

Or you can run docker-machine active and it will simply print either the name of the active machine or No active host found.

Notice that you should now see * under ACTIVE. Now you're connected and should be able to run commands against the docker daemon. The good news is that all docker clients should use the same commands with a couple caveats that I'll point out as we go along.

Which Context?

I'm not sure if "context" is the right term here, but I'm hoping it conveys my meaning.

When your native environment is Windows or OSX you have a different context than if it's Linux. Where it bit me first was my first experience with volumes. When you are trying to map files between your host environment and a container's environment the tutorials I've seen always seem to be assuming how it will work in your production environment - which is running in a native Linux environment. But this trips up the uninitiated.

In Windows the C:\Users directory for your host file system is mounted into the Docker VM at /c/Users. In OSX the /Users path is mounted at /Users making the translation easy. You can confirm this in the VirtualBox Shared Folder settings for your virtual machine. In each case it requires you to be aware of something different.

• Windows: you have to translate paths from C:\Users\ to '/c/Users/' (lowercase c).
• OXS: No translation of paths, but if you want to share files outside of /Users/ you need to share them using VirtualBox, ensure they're mounted and use the path as the Docker VM sees it.

When you're working with your application files this is just something you need to understand and work with. You need to just remember that Docker is running inside a Linux VM, so it can't see your Windows or OSX file system. This can trip you up sometimes, but what find helps a lot is to work in the context of the Docker host (the VM). Here's how you do it:

    $ docker-machine ssh default
                        ##         .
                      ## ## ##        ==
                   ## ## ## ## ##    ===
               /"""""""""""""""""\___/ ===
          ~ { ~~ ~ ~~ ~ ~ /  ===- ~~~
               \______ o           __/
                 \    \         __/
                  \____\_______/
     _                 _   ____     _            _
    | |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
    | '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
    | |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
    |_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
    Boot2Docker version 1.10.0, build master : b09ed60 - Thu Feb  4 20:16:08 UTC 2016
    Docker version 1.10.0, build 590d5108

Now except for the files hosted in your native environment it's really not different than if you were running natively on Linux.

Running Containers

This is pretty well documented, but this wouldn't be a proper walk-through unless we started up a container and pointed out a few things.

First, lets just fire up a container:

$ docker run -d -P nginx

The -d runs the container in the background - which is what you want most of the time. If you leave it off the output of the docker logs command will tail to your console. That's not what we want at the moment so we'll leave it at that.

Once it's done the output to your console will look something like this:

Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
5c9a73b9c4ad: Pull complete 
152440d1ba68: Pull complete 
cd0794b5fd94: Pull complete 
18de280c0e54: Pull complete 
bf58831fd41c: Pull complete 
ae8e1e9c54b3: Pull complete 
Digest: sha256:83be29bb91fc47f3a7453fc51db66816c9c637131f302a83dfd136e4b1901fbe
Status: Downloaded newer image for nginx:latest
468a6404cfcc1240750866608b51cd41a8465315767605634b272f0eddca1b2d

That last long string of numbers is the container's hash, or id. But don't worry, you don't ever need to type out the whole thing - just enough of the numbers to uniquely identify it. Which means if there's only one container running, or none of the other containers start with the same character, then you can identify it by a single character like this:

$ docker logs 4

Where 4 is the first character of the container id.

But there are other, more friendly ways to reference a container. For example:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                           NAMES
468a6404cfcc        nginx               "nginx -g 'daemon off"   48 seconds ago      Up 47 seconds       0.0.0.0:32769->80/tcp, 0.0.0.0:32768->443/tcp   berserk_kalam

See that shortened CONTAINER ID? You can look it up anytime you need to like we just did. But there's some other interesting information here as well. Under name can you see berserk_kalam? Yours will most likely be something else. That's the random name assigned to every container unless you specify the --name option. So you can follow along it will be easier if we can use the exact same commands. So you can run this:

$ docker run -d -P --name www nginx
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                           NAMES
7ecb60689494        nginx               "nginx -g 'daemon off"   4 seconds ago       Up 3 seconds        0.0.0.0:32771->80/tcp, 0.0.0.0:32770->443/tcp   www
468a6404cfcc        nginx               "nginx -g 'daemon off"   20 minutes ago      Up 20 minutes       0.0.0.0:32769->80/tcp, 0.0.0.0:32768->443/tcp   berserk_kalam

You'll see a second container and under the name you'll see www.

You may have also noticed under PORTS that a couple of high ports are pointing to ports 80 and 443. Well those high ports are the ports that your Docker host is listening on, then those ports are mapped to 80 and 443 which are exposed by the container. So if we want to actually talk to our container we need to use the high ports. But at which IP? If you open your web browser right now and put in http://localhost:32771 you'll see this:

What we need is the IP of the Docker VM running in VirtualBox. So we want to run this:

$ docker-machine ip default
192.168.99.100

Which returns the IP we want. Now if we put that IP into our browser like before at http://192.168.99.100:32771 we should see this:

Those port mappings we're using were assigned randomly by Docker because we used the -P option. If we had omitted it, we couldn't access our nginx at all. But what if we wanted to use a specific port? We could use 80, or any other port. But since this is the only web service running on our container we'll go ahead and use 80. There is a way to change the port of a running container, but it's easier to just create a new one:

$ docker stop www
$ docker rm -v www
$ docker run -d -p 80:80 -p 443:443 --name www nginx

You should now be able to access nginx from your web browser using the default port 80:

Shared Folders

Update 3/05/16: I've written an addition post that reviews shared folders and shows you how to make them permanent.

I mentioned before that by default the OSX /Users directory and the Windows C:\Users directory are mapped by default so your default means of synchronizing files between your Host OS and your VM is to use some subdirectory under the appropriate path. However, not everybody uses those path hierarchies for their development files. As a long-time Windows developer I always needed to conserve every possible path character to stay below the dreaded 256 character limit for paths accessed by msbuild. So I developed a habit of putting all my source code under C:\dev\src instead of C:\Documents and Settings\[USER_NAME] or even C:\Users\[USER_NAME] was too much. Now that my primary system is OSX it matters less but here's how to map shared folders:

1. Open your 'Shared Folders' settings in VirtualBox for your Docker VM:

2. Select the path that you would like to share with your Docker VM and accept the default name or change it as you see I've done above.

3. Next you'll want to ssh into your Docker VM like I mentioned earlier:

       $ docker-machine ssh default    

4. Now you want to create a mount point for your shared folder:

   $ sudo mkdir /mnt/src
   

   I named mine src just like I did in the VirtualBox Shared Folder settings, but the names don't have to match here, just in the next step. I placed it in the mnt directory which is a convention to follow but you could put it at the root of the file system / if you like - just don't clash with any other directory names.

5. Now we mount the folder using the name from the Shared Folder settings and then the mount point we just created with the mkdir command:

   $ mount -t vboxsf -o uid=1000,gid=50 src /mnt/src

   The first src is the name I used in the VirtualBox Shared Folder settings and the /mnt/src is the directory I created with mkdir for the mount point.

6. You should now be able to verify that your share is mounted by viewing the contents of the mount like this:

   $ ls /mnt/src
   dvMENTALmadness/ scratchpad/
   

   Which shows the two folders I have underneath my shared folder. When you edit files in either the Host OS or the Docker VM the file will stay in sync - just be aware that Windows and Linux use different line endings so if you edit Windows files inside your Docker VM you may get some issues with line endings.

   You'll also want to make sure that the permissions on your Host OS allow the VirtualBox VM to access the directory. For example, in OSX if you want to create a directory at your file system root called my-dev you'd have to do this:

   $ sudo mkdir /my-dev

   In order to grant write access from within your Docker VM you'd have to run the following two commands from your OSX terminal:

   $ sudo chown [your-user-name] /my-dev
   $ sudo chown :staff /my-def

   Which would make the directory permissions match those in your /Users/[your-user-name]

Package Manager

I love package managers. They make setting up software clean and repeatable. So when I started using Docker I started trying to install tools in my Docker VM and then opening up a bash shell into my containers as well and trying to install tools there as well. But it was supremely frustrating because even though I can easily use a package manager (usually apt-get) inside any Dockerfile to install tools for my container it isn't available from inside my running host or container! Instead the only package manager is tce-load and I found I couldn't just translate my apt-get knowledge directly to use it - plus not all the packages are even there!

I wondered if I could use a different ISO image for VirtualBox? (Yes, you can). But before I got around to it I came to a realization: You don't want to use the package manager inside containers. You either need a new image with the tools already installed, or you need a separate, linked container with the tools installed instead of muddying your container. Docker is about microservices - Don't create monolithic containers

So I don't use the package manager. Docker provides tools like --volumes-from and --link (still available, but deprecated) and network settings to allow containers to interact with each other in defined ways - including adhoc administration and troubleshooting.

For example, if your container doesn't have a text editer like vi installed, don't install it in your container! You should have a separate image with vi installed, when you need to edit files in your container, run the image with vi and use the --volumes-from option to link the files in the two containers. Then you can use docker exec to access the container with vi installed and edit the files. NOTE: I'm not really advocating the direct editing of files in production (I would NEVER do that! :P ), this is just a simple and easy to understand example.

I'll write a post later demonstrating these principles. This one is already long enough, but I had to make the point.

Getting Inside a Container

Even taking into account the previous section, you often need to get into a container to see what's going on in there. When I first started everything I googled seemed to point to NSENTER. So I started by using that and completely missed docker exec, which is the now recommended way to access a container.

If I wanted to open up a bash shell directly inside a running container I would do it like this:

$ docker exec -it [container_name_or_id] bash

Where container_name_or_id would be either the name of the container (usually the --name option in the run command) or the unique portion of the container's hash id.

So if I were running a container for my postgres database that was created from the following command:

docker run --name pg-scratchpad -e POSTGRES_PASSWORD=$DB_PG_PWD -p 5432:5432 -v /var/lib/postgresql/data -d postgres

NOTE: For an explaination of the use of $DB_PG_PWD see: There's a password in my repository.

Then I could open a command prompt inside the container like this:

$ docker exec -it pg-scratchpad bash

Or to briefly demonstrate using a linked container, you could open up the psql command-line client like this:

docker run -it --link pg-scratchpad:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'

Which will create a second container, and open a shell prompt for you which is already connected to your postgres database in the pg-scratchpad container!

Persistent Data

If I don't tell you, you may find one day that after you restart your container (or your whole computer) that your data is gone! Well that's because by default data doesn't survive restarting the container. I'll try to be quick here and save the details for later, but there are a couple solutions:

Between restarts

Volumes provide the solution to persistence. At least that's part of what they do. You just saw an example of it when we created our postgres container above.

docker run --name pg-scratchpad -e POSTGRES_PASSWORD=$DB_PG_PWD -p 5432:5432 -v /var/lib/postgresql/data -d postgres

See that -v /var/lib/postgresql/data? That tells Docker to create a persistent volume for that container. If you create data in your database, then restart your container, that data will still be there. Without -v your data will disappear!

Between re-deploys

Except that once that container is removed nothing else can reference that data volume and it's gone! But there's a solution: Data Volume Containers. They work under the principle that as long as a container is referencing a volume it can still be accessed. Another interesting tidbit is that a container doesn't have to be running for another container to use it's volumes!

So we can create a data volume container like this:

$ docker create -v /var/lib/postgresql/data --name pg-data postgres

Then we can use it like this:

$ docker run -d --volumes-from pg-data --name pg-db -e POSTGRES_PASSWORD=$DB_PG_PWD -p 5432:5432 postgres

If we connect like this:

$ docker run -it --link pg-db:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'

Then we can create some data:

# CREATE TABLE my_table (my_column VARCHAR(64) NOT NULL);
# INSERT INTO my_table (my_column) VALUES ('Docker');

Make sure it's there:

# SELECT * FROM my_table;
 my_column 
-----------
 Docker
(1 row)

Then destroy everything but the data volume container:

# \q
$ docker stop pg-db
$ docker rm -v pg-db

And then create a new container - we'll call it pg-db-2 just to be explicit this is a new container:

$ docker run -d --volumes-from pg-data --name pg-db-2 -e POSTGRES_PASSWORD=$DB_PG_PWD -p 5432:5432 postgres

And then verify the data is there:

$ docker run -it --link pg-db-2:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'

# SELECT * FROM my_table;
 my_column 
-----------
 Docker
(1 row)

Conclusion

I'm sure you never thought we'd get here. There's much more to Docker - we didn't even create our own custom container. But that should be enough to get you started on solid footing if you understand everything we just covered. For a broader coverage of Docker checkout this Docker cheat sheet on Github.

Environment Variables

For this post you'll need the following environment variables setup:

DB_PG_URL="jdbc:postgresql://192.168.99.100:5432"
DB_PG_USER=postgres
DB_PG_PWD=12345

I've explained why here. And here's more information about Docker and environment variables.

The above IP address 192.168.99.100 will depend on your setup. If you're running an installation of postgres (localhost or remote) you'll use the IP or hostname you use to reach that instance. If you're following along, using Docker you'll need the IP address returned from docker-machine (assuming either OSX or Windows):

$ docker-machine ls

NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
default   *        virtualbox   Running   tcp://192.168.99.100:2376   

Whatever the IP returned for URL is what you'll use in the environment var DB_PG_URL.

The remaining Docker commands assume you have a VM named default. If not you'll need to substitute default with the name of your VM - or create a VM named default.

Also, to avoid the need to include separate instructions based on your VM host OS the remaining commands assume you are running a terminal session inside your Docker host, like this:

$ docker-machine ssh default

Once you run that command your terminal/console window context will be inside your Docker host.

Postgres setup

If you don't have postgres installed already, then go do that. Or if you have Docker installed, then you can just execute the following Docker run command and you're already done! Here's the official page for the Docker postgres image.

$ docker run --name pg-scratchpad -e POSTGRES_PASSWORD=$DB_PG_PWD -p 5432:5432 -v /var/lib/postgresql/data -d postgres

If you're still new to Docker the short explanation is that the command will start an installation of postgres in a new container (run) over port 5432 (-p) with a persistent volume (-v) running in the background (-d).

That -e POSTGRES_PASSWORD=$DB_PG_PWD is a password I've stored as an environment variable.

Creating some data

You'll need to connect to postgres somehow. If you're using a local install of postgres then you'll want to run psql from the command line. For those of us running Docker try this:

docker run -it --link pg-scratchpad:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'

We're creating a second container here and linking it to our pg-scratchpad container where our database instance is running. -it sets up an interactive shell that is using the link between containers and running the psql command. This will give us a client where we can setup our database. The above command came from the documentation for the Docker postgres image.

Like any ORM, Slick let's you define your model in code or config and it will create it for you. But for my purposes I wanted to work under the assumption I had an existing database. So we'll create a simple schema, put some data in it and use Slick to query the table.

So run the following commands in your psql session (# is just the command prompt - don't type it, but do make sure to terminate each command with a semi-colon (;)):

# CREATE TABLE records (
  id INT NOT NULL,
  value VARCHAR(64) NOT NULL
);

# INSERT INTO records (id, value) VALUES (1, 'booyah!');

If you run a query you should get some data back. As long as you do you're set:

# SELECT * FROM records;

Your results should look like this:

 id |     value      
----+----------------
  1 | booyah!
(1 row)

sbt setup

Your build.sbt file should include the following dependencies:

libraryDependencies ++= List(
// ... your other dependencies
  "com.typesafe.slick" %% "slick" % "3.0.0",
  "org.postgresql" % "postgresql" % "9.4-1201-jdbc41",
  "com.zaxxer" % "HikariCP" % "2.4.1"
)

NOTE: HikariCP is a "non-optional" dependency for Slick.

I'm not sure what HikariCP is for but I get a runtime error if it isn't there. As for slick and postgresql, the former is a Functional ORM which supports streaming and back-pressure and the latter is the postgresql client library and drivers Slick needs to connect to your database instance.

Configuration

Create an application.conf file under src/main/resources in your source tree. Here's how you'll configure it:

pg-postgres = {
  url = ${DB_PG_URL}/postgres
  user = ${DB_PG_USER}
  password = ${DB_PG_PWD}
  driver = org.postgresql.Driver
}

Case class mapping

Add a new scala class to your project and call it Tables.scala. The name is arbitrary although it does follow the convention used by the Slick Code Generator.

import slick.driver.PostgresDriver.api._

case class record(id: Int, value: String)

class Records(tag: Tag) extends Table[record](tag, "records") {
  def id = column[Int]("id")
  def value = column[String]("value")
  def * = (id, value) <> (record.tupled, record.unapply)
}

Here we've mapped a case class and class structure to match our database table. Most of this is pretty self-explanatory except for the def * = .... That function allows Slick to map back and forth between our case class record and the database.

Running a query

Now add a new scala class to your project and call it MyApp.scala. Enter the following object definition:

import slick.driver.PostgresDriver.api._
import scala.concurrent.Await
import scala.concurrent.duration.Duration

object MyApp extends App {
  val query = TableQuery[Records]
  val db = Database.forConfig("pg-postgres")
  try{
    Await.result(db.run(DBIO.seq(
      query.result.map(println)
    )), Duration.Inf)
  } finally db.close
}

Now run MyApp and you should see something like the following:

Vector(record(1,booyah!))

This doesn't cover querying or anything else of the sort, but now that you're started you've finished the hard part ;).

Slick Code Generator

As an bonus, it could be valuable to see what gets generated by different table structures. Add the following dependency to your build.sbt:

  "com.typesafe.slick" %% "slick-codegen" % "3.0.2" % "provided"

Now from the command line execute sbt console to drop into the scala REPL. Then enter the following:

First create a variable for each of your environment variables:

scala> val url = System.getenv("DB_PG_URL") + "/postgres"
url: String = jdbc:postgresql://192.168.99.100:5432/postgres

scala> val user = System.getenv("DB_PG_USER")
user: String = postgres

scala> val pwd = System.getenv("DB_PG_PWD")
pwd: String = 1qaz2wsx

Then call the Slick Code Generator like this:

scala> slick.codegen.SourceCodeGenerator.main(Array("slick.driver.PostgresDriver", "org.postgresql.Driver", url, "src/main/scala", "com.dvMENTALmadness", user, pwd))

The first two arguments are the slick and postgres drivers respectively. Next, pass in your url variable to point to your database instance. Then src/main/scala is the relative path from my sources root where I want my Tables.scala file created and com.dvMENTALmadness is my package name. Lastly pass in the user and pwd variables so the code generator can authenticate with postgres.

Once the script runs you should find (assuming you're using the above paramters) a new file: src/main/scala/com/dvMENTALmadness/Tables.scala created and the content should look something like this:

package com.dvMENTALmadness
// AUTO-GENERATED Slick data model
/** Stand-alone Slick data model for immediate use */
object Tables extends {
  val profile = slick.driver.PostgresDriver
} with Tables

/** Slick data model trait for extension, choice of backend or usage in the cake pattern. (Make sure to initialize this late.) */
trait Tables {
  val profile: slick.driver.JdbcProfile
  import profile.api._
  import slick.model.ForeignKeyAction
  // NOTE: GetResult mappers for plain SQL are only generated for tables where Slick knows how to map the types of all columns.
  import slick.jdbc.{GetResult => GR}

  /** DDL for all tables. Call .create to execute. */
  lazy val schema = Record.schema
  @deprecated("Use .schema instead of .ddl", "3.0")
  def ddl = schema

  /** Entity class storing rows of table Record
   *  @param id Database column id SqlType(int4)
   *  @param value Database column value SqlType(varchar), Length(64,true) */
  case class RecordRow(id: Int, value: String)
  /** GetResult implicit for fetching RecordRow objects using plain SQL queries */
  implicit def GetResultRecordRow(implicit e0: GR[Int], e1: GR[String]): GR[RecordRow] = GR{
    prs => import prs._
    RecordRow.tupled((<<[Int], <<[String]))
  }
  /** Table description of table record. Objects of this class serve as prototypes for rows in queries. */
  class Record(_tableTag: Tag) extends Table[RecordRow](_tableTag, "record") {
    def * = (id, value) <> (RecordRow.tupled, RecordRow.unapply)
    /** Maps whole row to an option. Useful for outer joins. */
    def ? = (Rep.Some(id), Rep.Some(value)).shaped.<>({r=>import r._; _1.map(_=> RecordRow.tupled((_1.get, _2.get)))}, (_:Any) =>  throw new Exception("Inserting into ? projection not supported."))

    /** Database column id SqlType(int4) */
    val id: Rep[Int] = column[Int]("id")
    /** Database column value SqlType(varchar), Length(64,true) */
    val value: Rep[String] = column[String]("value", O.Length(64,varying=true))
  }
  /** Collection-like TableQuery object for table Record */
  lazy val Record = new TableQuery(tag => new Record(tag))
}

Play around with it, create multiple tables, add joins then re-run the generator and see what you get - it should be instructive. Your query will change slightly to look like this instead:

Tables.Record.result.map(println)

But your result should be the same. Enjoy!