jasani.org

Docker now supports adding host mappings

In my previous blog entry, I mentioned that Docker didn't provide any way to add host-to-ip mappings to /etc/hosts. Well, the most recent version of Docker (1.3.1) now addresses that issue with a new add hosts option. Let's take a quick look at what it does.

Given a standard default /etc/hosts.

% docker run -it ubuntu cat /etc/hosts
172.17.0.8	b14031841b2b
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters

We can add a mapping for server 'foo' at '10.0.0.3'.

% docker run -it --add-host foo:10.0.0.3 ubuntu cat /etc/hosts
172.17.0.9	189a650112db
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
10.0.0.3	foo

We can also add mappings for multiple servers, in this case 'foo' and 'bar'.

% docker run -it --add-host foo:10.0.0.3 --add-host bar:10.7.3.21 ubuntu cat /etc/hosts
172.17.0.13	3f8543ecc372
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
10.7.3.21	bar
10.0.0.3	foo

So far so good. But there are a couple more scenarios to consider. The first is how to handle the case where a single server is hosting multiple services. These services could be in a single Docker container, separate containers or not in a container at all. What's important is that they are both on some remote server.

So let's consider the case where our services are named svca and svcb. We could try to add both of them as host mappings. Let's see how that works out.

% docker run -it --add-host svca:10.0.0.9 --add-host svcb:10.0.0.9 ubuntu cat /etc/hosts
172.17.0.16	5d7eaa5dcd16
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
10.0.0.9	svcb
10.0.0.9	svca

Both services are pointing to the same IP address, but I'm not sure this is valid for /etc/hosts. The format is supposed to be IP hostname [hostalias]*.

Lucky for us, there is another option. When invoking add-host, we can include both services inside of double quotes. It's not documented, but it does work as you can see below.

% docker run -it --add-host "svca svcb":10.0.0.9 ubuntu cat /etc/hosts
172.17.0.17	380df28b4612
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
10.0.0.9	svca svcb

There we go, both services pointing to the same IP address and /etc/hosts written in a valid format.

Another issue is how to provide hostaliases of localhost. As far as I can tell, Docker still doesn't have anything to address this issue. For the time being this means that I'll still be using Mungehosts to do that work.

In summary, add-host is a valuable new tool in the Docker toolbox for pointing to remote services. It's worked well for all the cases I needed for it and it's easy to use. It's good to see Docker adding more config options to make our lives easier.

Modifying /etc/hosts in Docker images for fun and profit

Update: The new Docker version 1.3.1 addresses some issues I describe in this post. Read about what problems Docker has solved.

Did you know that Docker doesn't let you (easily) modify /etc/hosts? You can't programmatically add aliases for localhost or new hostname-to-ip mappings. In short, Docker treats /etc/hosts like a static resource.

No problem. This will just take a quick Unix script to solve.

Attempt 1, the slate is wiped clean

No big deal, right? We simply write ourselves a little sed script to alias localhost and add a couple additional hostname mappings. We can run the sed script from our Dockerfile.

Well if you tried this at home, you'd quickly find out that it doesn't work. Each RUN command in a Dockerfile does another commit to the image. Ordinarily, that's just a space problem but Docker treats /etc/hosts special. It will overwrite it whenever it wants without any regard for your modifications. In a nutshell I wasn't able to persist any changes made during the Dockerfile.

Attempt 2, mount up

Okay, so we simply run our sed script when we run our container. Docker run will execute our startup script, running our sed script, updating /etc/hosts and finally running our server.

sed: cannot rename /etc/sedl8ySxL: Device or resource busy

Unfortunately, that doesn't work either. The way sed -i works is to read from the original file, write to a temp file and then move the temp file over the original file. This doesn't work in this case because /etc/hosts is a mount point.

root@07734c969aa6:/usr/local/src# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           19G  3.9G   14G  23% /
none             19G  3.9G   14G  23% /
tmpfs          1005M     0 1005M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/sda1        19G  3.9G   14G  23% /etc/hosts      <----
none            1.1T  159G  881G  16% /code
tmpfs          1005M     0 1005M   0% /proc/kcore

Alternatives

How are other people solving this problem today? Some people seem to be using dnsmasq as a way of bypassing the problem of /etc/hosts. Running another dns server process as a workaround seems excessive, but I'm not sure I blame them. Doing a search of issues in the Docker Github repo found 125 issues which mentioned /etc/hosts.

On Stackoverflow there is the realization that /etc/hosts is now writable, but doesn't persist changes in running container.

Even so, I hadn't seen anyone provide a programmatic way to address the issue.

Attempt 3, Mungehosts is born

I took a step back and decided to write my own utility, Mungehosts. It solves my major pain points:

Programmatic interface
Can add aliases for localhost (ipv4 and ipv6)
Can add hostname-to-ip mappings
Works with Docker
Compiled executable (Linux binary in releases)
No dependencies except C runtime
Small size (167 KB)

Check it out and provide feedback. I know that the smart Docker folks will eventually get all of the features into the main application. When that happens Mungehosts will retire to pasture, but it will be a good workaround until then.

Unable to remove docker image

I've been playing around with building Docker images and noticed I had a few that had <none> as both the repository and tag info.

% docker images

REPOSITORY                 TAG                 IMAGE ID	...

<none>                     <none>              c1e0e48a03bb  
<none>                     <none>              c8bf15029bc0

I tried to delete these untagged images, but couldn't.

% docker rmi c1e0e48a03bb
Error response from daemon: Conflict, cannot delete c1e0e48a03bb because the container 5f266c452a8b is using it, use -f to force

Ok, so we'll try to force deletion.

% docker rmi -f c1e0e48a03bb
Error response from daemon: No such id: b5dc67806c5f1e4abfa079a1c99f6d782e5696a25a3c9beb16043fe2e7164d19
2014/10/18 10:04:27 Error: failed to remove one or more images

Mmm ... the original message said the image couldn't be deleted because it was in use by a container. I didn't have any running containers as I verified by doing docker ps. But the issue is that docker holds on to images even from exited containers. Once you realize that, then this becomes much easier to resolve.

To delete all your exited containers.

% docker ps -a -q --filter "status=exited" | xargs docker rm

And to then delete all your untagged images.

% docker rmi `docker images -q --filter "dangling=true"`

Apple touch inconsistency

Apple is really known for their user interface design. I'd say they generally have a lot more hits than misses, but one miss recently came as a surprise for me.

Take an iPhone, go to the Settings screen. Scroll the screen with your finger. Notice that the text moves in the direction of your fingers and the scrollbar moves in the opposite direction. The paradigm is that your finger is controlling the text, directly moving it. The scrollbar is just illustrative to let you know how far you are in the document.

Now take a Macbook or Macbook Pro with a multitouch trackpad. Use two fingers to scroll a webpage or other window that has a scrollbar. Notice that this time the text moves in the opposite direction of your stroke, while the scrollbar moves in the same direction as your fingers. The paradigm here is that your fingers control the scrollbar, which then inversely moves the text. One might say this method is indirect control.

So that's kind of strange. Two Apple devices using touch in entirely different and opposite ways. One going for direct control and the other indirect. This inconsistency could be Apple experimenting with touch user interface concepts or just be a result of legacy user interface design. But whatever the reason, they will someday have to make a choice.

As we head towards a future in which iPhone apps and OSX apps merge and play together on a singular device - possibly the fabled touch tablet - Apple will have to decide which of the touch paradigms will live on. Which one will it be?

Error installing Haskell X11 libs

I recently bought a new Macbook Pro and with the recent announcment of GHC 6.10.1, I thought it was time to install the latest GHC and libs. Previously I had some trouble installing the Haskell X11 libs on Leopard, but I thought something might have been borked on my old laptop. With this new one and starting from scratch, I hoped things would go smoothly. Twas not to be.

The X11 libraries are installed under /usr/X11/lib.

bash-3.2$ locate libX11
/Applications/CrossOver Games.app/Contents/SharedSupport/X11/lib/libX11.6.2.dylib
/Applications/CrossOver Games.app/Contents/SharedSupport/X11/lib/libX11.6.dylib
/Applications/CrossOver Games.app/Contents/SharedSupport/X11/lib/libX11.dylib
/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.6.2.dylib
/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.6.dylib
/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.a
/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.dylib
/Developer/SDKs/MacOSX10.5.sdk/usr/X11/lib/libX11.6.2.0.dylib
/Developer/SDKs/MacOSX10.5.sdk/usr/X11/lib/libX11.6.dylib
/Developer/SDKs/MacOSX10.5.sdk/usr/X11/lib/libX11.dylib
/opt/local/var/macports/sources/rsync.macports.org/release/ports/x11/xorg-libX11
/opt/local/var/macports/sources/rsync.macports.org/release/ports/x11/xorg-libX11/Portfile
/usr/X11/lib/libX11.6.2.0.dylib
/usr/X11/lib/libX11.6.dylib
/usr/X11/lib/libX11.dylib
/usr/X11/lib/libX11.la

I've fallen down and I can't get up

Here are some things I tried that didn't work.

Try adding an entry for extra-lib-dirs to X11.cabal

A FAQ entry for XMonad offers up some helpful tips to try.

extra-lib-dirs:     /usr/X11/lib

and try to configure ...

bash-3.2$ runhaskell Setup.hs configure

checking for X... no
configure: error: X11 libraries not found, so X11 package cannot be built
See `config.log' for more details.

Try passing option to configure

bash-3.2$ runhaskell Setup.hs configure --extra-lib-dirs=/usr/X11/lib

checking for X... no
configure: error: X11 libraries not found, so X11 package cannot be built
See `config.log' for more details.

Try setting LD_LIBRARY_PATH

bash-3.2$ export LD_LIBRARY_PATH=/usr/X11/lib; runhaskell Setup.hs configure

checking for X... no
configure: error: X11 libraries not found, so X11 package cannot be built
See `config.log' for more details.

Try setting DYLD_LIBRARY_PATH

bash-3.2$ export DYLD_LIBRARY_PATH=/usr/X11/lib; runhaskell Setup.hs configure

checking for X... no
configure: error: X11 libraries not found, so X11 package cannot be built
See `config.log' for more details.

Try setting DYLD_FALLBACK_LIBRARY_PATH

bash-3.2$ export DYLD_LIBRARY_PATH=/usr/X11/lib
bash-3.2$ runhaskell Setup.hs configure

Setup.hs:3:0:
    Warning: In the use of `defaultUserHooks'
             (imported from Distribution.Simple):
             Deprecated: "Use simpleUserHooks or autoconfUserHooks, unless you need Cabal-1.2
             compatibility in which case you must stick with defaultUserHooks"
Setup.hs: &lt;command line&gt;: can't load .so/.DLL for: dl (dlopen(libdl.dylib, 9): image not found)

It's a bit disappointing how different OSX is from the Linux distros. Yeah it doesn't suck like Windows, but even so I wish it weren't so different from Linux.

So what did work?

Just like in my previous article, using the LDFLAGS variable solves the problem.

bash-3.2$ export LDFLAGS=-L/usr/X11/lib; runhaskell Setup.hs configure

Setup.hs:3:0:
    Warning: In the use of `defaultUserHooks'
             (imported from Distribution.Simple):
             Deprecated: "Use simpleUserHooks or autoconfUserHooks, unless you need Cabal-1.2
             compatibility in which case you must stick with defaultUserHooks"
Warning: defaultUserHooks in Setup script is deprecated.
Configuring X11-1.4.4...
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking how to run the C preprocessor... gcc -E
checking for X... libraries , headers 
checking for gethostbyname... yes
checking for connect... yes
checking for remove... yes
checking for shmat... yes
checking for IceConnectionNumber in -lICE... yes
checking whether to build Xinerama... yes
checking for egrep... grep -E
checking for ANSI C header files... rm: conftest.dSYM: is a directory
rm: conftest.dSYM: is a directory
yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking X11/extensions/Xinerama.h usability... yes
checking X11/extensions/Xinerama.h presence... yes
checking for X11/extensions/Xinerama.h... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating X11.buildinfo
config.status: creating include/HsX11Config.h
config.status: creating include/X11_extras_config.h

Error installing Haskell SDL-image

I really like OSX, except when I have to jump through extra hoops to install things on it versus Linux. The latest challenge was installing the Haskell SDL-image bindings on Leopard.

Configure would bomb right away, saying it couldn't find the libs even though I had them installed via MacPorts.

bash-3.2$ runhaskell Setup.lhs configure
Configuring SDL-image-0.5.2...
checking for sdl-config... /opt/local/bin/sdl-config
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for IMG_Load in -lSDL_image... no
configure: error: *** SDL_image lib not found! Get SDL_image from
http://www.libsdl.org/projects/SDL_image/index.html

bash-3.2$ locate SDL_image
/opt/local/include/SDL/SDL_image.h
/opt/local/lib/libSDL_image-1.2.0.1.5.dylib
/opt/local/lib/libSDL_image-1.2.0.dylib
/opt/local/lib/libSDL_image.a
/opt/local/lib/libSDL_image.dylib
/opt/local/lib/libSDL_image.la

So if you try using the same trick I did with installing Haskell X11 libs ...

bash-3.2$ export LDFLAGS=-L/opt/local/lib; runhaskell Setup.lhs configure
Configuring SDL-image-0.5.2...
checking for sdl-config... /opt/local/bin/sdl-config
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for IMG_Load in -lSDL_image... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating SDL-image.buildinfo
config.status: creating includes/HsSDLConfig.h
config.status: includes/HsSDLConfig.h is unchanged
bash-3.2$ runhaskell Setup.lhs build
Preprocessing library SDL-image-0.5.2...
Undefined symbols:
"_ZCMain_main_closure", referenced from:
	_ZCMain_main_closure$non_lazy_ptr in libHSrts.a(Main.o)
"___stginit_ZCMain", referenced from:
	___stginit_ZCMain$non_lazy_ptr in libHSrts.a(Main.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
linking dist/build/Graphics/UI/SDL/Image/Version_hsc_make.o failed
command was: /opt/local/bin/ghc -optl-L/opt/local/lib -optl-lSDL_image dist/build/Graphics/UI/SDL/Image/Version_hsc_make.o -o dist/build/Graphics/UI/SDL/Image/Version_hsc_make

It gets further, but still fails. Now's the time to read the nice MACOSX file included with the distro. It tells you to write your own hsc2hs script to help it find certain dependencies on your system. For the most part the instructions are good, but for me it was still having trouble finding HsFFI.h. I had to add a cflag directive to make it work. So here is my myhsc2hs.

#!/bin/sh
# added include dir of /opt/local/lib/ghc-6.8.2/include
echo "/opt/local/bin/hsc2hs -v --cc=gcc --ld=gcc --cflag="-I/opt/local/lib/ghc-6.8.2/include" --lflag="-L/opt/local/lib -lSDLmain -framework AppKit -framework SDL" $@"
/opt/local/bin/hsc2hs -v --cc=gcc --ld=gcc --cflag="-I/opt/local/lib/ghc-6.8.2/include" --lflag="-L/opt/local/lib" --lflag="-lSDLmain" --lflag="-framework" --lflag="AppKit" --lflag="-framework" --lflag="SDL" "$@"

Error installing Haskell X11 package

Recently I was having trouble installing the Haskell X11 package on OSX Leopard. Running configure generated an error.

bash-3.2$ runhaskell Setup.hs configure
Warning: defaultUserHooks in Setup script is deprecated.
Configuring X11-1.4.3...
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for X... no
configure: error: X11 libraries not found, so X11 package cannot be built
See `config.log' for more details.

Of course I had X11 libraries. In fact, I had them from both Leopard and MacPorts. I had them all over the place, but cabal wasn't finding them.

bash-3.2$ locate libX11.dylib
/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.dylib
/Developer/SDKs/MacOSX10.5.sdk/usr/X11/lib/libX11.dylib
/Previous Systems.localized/2008-07-15_1549/Developer/SDKs/MacOSX10.3.9.sdk/usr/X11R6/lib/libX11.dylib
/Previous Systems.localized/2008-07-15_1549/Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6/lib/libX11.dylib
/Previous Systems.localized/2008-07-15_1549/usr/X11R6/lib/libX11.dylib
/opt/local/lib/libX11.dylib
/opt/local/var/macports/software/xorg-libX11/1.1.3_0/opt/local/lib/libX11.dylib
/usr/X11/lib/libX11.dylib

I tried adding extra-lib-dirs: /usr/X11/lib to X11.cabal, but it didn't fix the error. I upgraded Cabal to version 1.4.0.2, but that didn't work either. What did finally work was setting LDFLAGS.

bash-3.2$ export LDFLAGS=-L/usr/X11/lib
bash-3.2$ runhaskell Setup.hs configure
Warning: defaultUserHooks in Setup script is deprecated.
Configuring X11-1.4.3...
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for X... libraries , headers
checking for gethostbyname... yes
...

Emacs Erlang snippets

Continuing with my previous yasnippet for Ruby, here are some snippets for Erlang mode.

Installation

Create an erlang-mode directory under text-mode

Save the following code in a file named mod.

#name : -module().
#contributor : hitesh &lt;hitesh.jasani gmail.com&gt;
# --
-module(${1:$(file-name-nondirectory 
               (file-name-sans-extension (buffer-file-name)))}).
$0

Save the following code in a file named exp.

#name : -export([]).
#contributor : hitesh &lt;hitesh.jasani gmail.com&gt;
# --
-export([${1:start/0}]).
$0

Save the following code in a file named imp.

#name : -import([]).
#contributor : hitesh &lt;hitesh.jasani gmail.com&gt;
# --
-import(${1:lists}, [${2:map/2, sum/1}]).
$0

Add a hook in your .emacs to enable yas mode with erlang.
```
(add-hook 'erlang-mode-hook 'yas/minor-mode-on)
```

textmate ruby snippet to yasnippet

Dr. Nic recently put together a screencast of adding a Ruby snippet to TextMate. This particular snippet makes a class definition using the filename as the default classname. I always enjoy listening to Dr. Nic, so definitely check out the 'cast.

I used to love TextMate, but I can't use it anymore since I'm working on multiple platforms. Emacs is my editor of choice now and I can't say that I'm missing anything.

Dr. Nic's example was interesting enough that I decided to replicate it using yasnippet and elisp. As you'll see in the video, TextMate supports shelling out to bash or Ruby to do complex processing of snippets. Yasnippet supports inline elisp to handle complex snippets and allows shelling out to other processes when necessary.

Usage

If you're editing a file named foo.rb and use the snippet, it will create a class definition for Foo. If you're editing a file named currency_converter.rb, it will create a class named CurrencyConverter.

Installation

Put the following code into a file named kls in the ruby-mode directory. Then use it like you would any other yasnippet.

#name : class ... end
#contributor : hitesh <hitesh.jasani gmail.com>
#contributor : Roberto H.
# --
class ${1:`(let ((fn (capitalize (file-name-nondirectory
				(file-name-sans-extension
		(or (buffer-file-name)
			(buffer-name (current-buffer))))))))
	(replace-regexp-in-string "_" "" fn))`}
$0
end

Update (2010-01-31): Thanks to some good input this routine has been improved to handle multiple underscores in the filename.

ELisp doesn't support closures

I'm hacking some ELisp functions recently and I realize that ELisp doesn't have a filter function or support closures. That's disappointing. One sort of takes it for granted nowadays that a decent language will have both. AFAIK ELisp was written before there was a Common Lisp standard. But even then, I'm surprised it didn't evolve with the times.

I don't mind writing my own filter function since I'm only going to have to do it once. But not having closures is a little bigger impact.

Instead of being able to write something like this ...

(defun string-match-c (s)
  (lambda (x) (string-match s x)))

I'll have to write this instead.

(defun string-match-c (str)
  (lexical-let ((s str))
    (lambda (x) (string-match s x))))

Usage is the same for both.

(filter (string-match-c "ba") '("foo" "bar" "baz"))
;; => ("bar" "baz")

But they really can't compare to Haskell, which doesn't need a string-match-c function defined since it supports partial application.

filter (isPrefixOf "ba") ["foo", "bar", "baz"]

Emacs is still my favorite editor, but I sure wish it was using Common Lisp, Scheme or Haskell.

Resources

Previous Comments

maybe wlcompose.el help to solve this problem. I think Elisp support closure indirectly. In Elisp, some functions are nothing more than a list whose car is "lambda", we can construct such list(or function) as we want.
Wang Chunye http://www.emacswiki.org/emacs/wcy-compose.el