27 April 2016

Overview

This is another atypical introduction to docker on OS X a follow up to my previous posts. It’s been 3 years since I blogged, I’ve been busy, burnt out and a lot has changed in my life. However I thought this would be a good thing to explore.

It aims to learn more about the internals of xhyve and the new docker native beta on OS X by experimenting and analyzing what docker is doing behind the scenes. Like my previous posts the inspiration for this post comes from my favourite article from sysadvent - Down the ls Rabbit Hole.

Requirements

Pre-requisites - access to the Docker beta.

What do we have

So lets see what we start off with

$ docker info
Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 19:36:04 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   a5315b8
 Built:        Mon Apr 18 19:19:21 2016
 OS/Arch:      linux/amd64

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.4.6
Operating System: Alpine Linux v3.3
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.956 GiB
Name: docker
ID: SGTB:RMFN:XSDC:P5SK:CTPV:IBBO:QFOE:M6FL:OG77:CXZL:ZBVS:HRDM
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 15
 Goroutines: 33
 System Time: 2016-04-27T17:28:52.124543023Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/

OK so it looks as if we are talking to a server running Alpine Linux on OS X.

Lets just see what is running on our OS X host using our old friend ps.

pnasrat         32157   0.2  0.1 573469540  11092   ??  Ss   10:05am   0:13.15 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -db /var/tmp/com.docker.db.socket -osxfs-volume /var/tmp/com.docker.osxfs.volume.socket -slirp /var/tmp/com.docker.slirp.socket -vmnet /var/tmp/com.docker.vmnetd.socket -port /var/tmp/com.docker.port.socket -vsock /var/tmp/com.docker.vsock -docker /var/tmp/docker.sock -addr fd:3 -debug
pnasrat         32144   0.1  0.0 573414380   4068   ??  S    10:04am   0:08.44 /Applications/Docker.app/Contents/MacOS/com.docker.osx.xhyve.linux -watchdog fd:0
pnasrat         32158   0.1  0.1 573412816   5012   ??  S    10:05am   0:08.44 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -db /var/tmp/com.docker.db.socket -osxfs-volume /var/tmp/com.docker.osxfs.volume.socket -slirp /var/tmp/com.docker.slirp.socket -vmnet /var/tmp/com.docker.vmnetd.socket -port /var/tmp/com.docker.port.socket -vsock /var/tmp/com.docker.vsock -docker /var/tmp/docker.sock -addr fd:3 -debug
root            32161   0.0  0.0  2469532    648   ??  S    10:05am   0:00.07 /Library/PrivilegedHelperTools/com.docker.vmnetd
pnasrat         32160   0.0  0.1 573422672   4992   ??  S    10:05am   0:00.02 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -xhyve /Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/xhyve.args
pnasrat         32159   0.0  0.1 575529704   5364   ??  S    10:05am   0:00.05 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -xhyve /Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/xhyve.args
pnasrat         32156   0.0  0.0  2469204   2132   ??  Ss   10:05am   0:00.03 com.docker.slirp --db /var/tmp/com.docker.db.socket --socket fd:3 --port-control fd:4
pnasrat         32155   0.0  0.0  2467860   3604   ??  Ss   10:05am   0:00.02 com.docker.osxfs --address fd:3 --connect /var/tmp/com.docker.vsock/connect --volume-control fd:4 --path /
pnasrat         32150   0.0  0.0 573405164   4008   ??  S    10:04am   0:00.02 /Applications/Docker.app/Contents/MacOS/com.docker.osx.xhyve.linux
pnasrat         32148   0.0  0.1 573426064   5532   ??  Ss   10:04am   0:01.62 com.docker.backend
pnasrat         32145   0.0  0.1  2562624  10816   ??  Ss   10:04am   0:02.37 com.docker.db --url=file:///var/tmp/com.docker.db.socket --git /Users/pnasrat/Library/Containers/com.docker.docker/Data/database
pnasrat         32143   0.0  0.1 573453428   5628   ??  S    10:04am   0:00.10 /Applications/Docker.app/Contents/MacOS/com.docker.osx.xhyve.linux -watchdog fd:0
root            32140   0.0  0.0  2436148   2140   ??  Ss   10:04am   0:00.02 /Library/PrivilegedHelperTools/com.docker.vmnetd

We can inspect the logs using syslog -k Sender Docker

Apr 27 13:07:17 enki Docker[44922] <Notice>: filesystem: osxfs
Apr 27 13:07:17 enki Docker[44922] <Notice>: Hypervisor: native; BootProtocol: direct; UefiBootDisk: /Users/pnasrat/UefiBoot.qcow2
Apr 27 13:07:17 enki Docker[44922] <Notice>: Docker is not responding: waiting 0.5s
Apr 27 13:07:17 enki Docker[44925] <Notice>: exec: /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux []string{"-A", "-m", "2G", "-c", "2", "-u", "-s", "0:0,hostbridge", "-s", "31,lpc", "-s", "2:0,virtio-ipc,uuid=e34e7389-c6ba-4f70-9f12-3a63edc453e6,path=/var/tmp/com.docker.vmnetd.socket,macfile=/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0", "-s", "3:0,virtio-ipc,uuid=2954abe4-b876-45d0-948c-1b1c51e80bdb,path=/var/tmp/com.docker.slirp.socket,macfile=/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.1", "-s", "4,virtio-blk,file:///Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2", "-s", "5,virtio-9p,path=/var/tmp/com.docker.db.socket,tag=db", "-s", "6,virtio-rnd", "-s", "7,virtio-9p,path=/var/tmp/com.docker.port.socket,tag=port", "-s", "8,virtio-sock,guest_cid=3,path=/var/tmp/com.docker.vsock,guest_forwards=2376;1525", "-l", "com1,pty=/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring", "-f", "kexec,/Applications/Docker.app/Contents/Resources/moby/vmlinuz64,/Applications/Docker.app/Contents/Resources/moby/initrd.img,earlyprintk=serial console=ttyS0 com.docker.driverDir=\"/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux\", com.docker.database=\"com.docker.driver.amd64-linux\""}
Apr 27 13:07:17 enki Docker[44927] <Notice>: Client reports version 12, commit 3c1bfeb0e86a9403f82302edfea4c4987cc2cb32
Apr 27 13:07:17 enki Docker[44927] <Notice>: com.docker.vmnetd reports MAC address: 22:b9:2e:1e:cf:32
Apr 27 13:07:17 enki Docker[44927] <Notice>: rx batching enabled (len=64)
Apr 27 13:07:20 enki Docker[44922] <Notice>: Docker is not responding: waiting 0.5s
--- last message repeated 8 times ---
Apr 27 13:07:26 enki Docker[44922] <Notice>: Docker is responding

Components of Docker for OS X

  • pinata - manage the Docker.app configuration database

Lets see what this returns - the default output to terminal is fancy but this does the right thing pinata list | pbcopy

These are advanced configuration settings to customise Docker.app on MacOSX.
You can set them via pinata set <key> <value> <options>.

*  hostname = docker
   Hostname of the virtual machine endpoint, where container ports will be
   exposed if using nat networking. Access it via 'docker.local'.

*  hypervisor = native (memory=2, ncpu=2)
   The Docker.app includes embedded hypervisors that run the virtual machines
   that power the containers. This setting allows you to control which the
   default one used for Linux is.

 >  native: a version of the xhyve hypervisor that uses the MacOSX
            Hypervisor.framework to run container VMs. Parameters: memory (VM
            memory in gigabytes), ncpu (vCPUs)


*  network = nat (external-bind=false)
   Controls how local containers can access the external network via the
   MacOS X host. This includes outbound traffic as well as publishing ports
   for external access to the local containers.

 >     nat: a mode that uses the MacOS X vmnet.framework to route container
            traffic to the host network via a NAT. Parameters:
            external-bind (bind ports to external network interface)
 > hostnet: a mode that helps if you are using a VPN that restricts
            connectivity. Activating this mode will proxy container network
            packets via the Docker.app process as host socket traffic.
            Parameters: docker-ipv4 (docker node), host-ipv4 (host node)

*  filesystem = osxfs 
   Controls the mode by which files from the MacOS X host and the container
   filesystem are shared with each other.

 >   osxfs: a FUSE-based filesystem that bidirectionally forwards OSX
            filesystem events into the container. 


*  daemon = run 'pinata get daemon' or 'pinata set daemon [@file|-]>
   JSON configuration of the local Docker daemon. Configure any custom
   options you need as documented in:
   https://docs.docker.com/engine/reference/commandline/daemon/. Set it
   directly, or a @file or - for stdin.

From the process listing we can see this uses /var/tmp/com.docker.db.socket

Running pinata go get the help shows us we need to look in --db-path which is defaulting to /Users/pnasrat/Library/Containers/com.docker.docker/Data/database/

$ ls -al
total 0
drwxr-xr-x   4 pnasrat  staff  136 27 Apr 09:02 .
drwxr-xr-x   8 pnasrat  staff  272 27 Apr 13:07 ..
drwxr-xr-x   8 pnasrat  staff  272 27 Apr 13:21 .git
drwxr-xr-x  17 pnasrat  staff  578 27 Apr 10:04 com.docker.driver.amd64-linux
$ git show
commit 83b0394b2e88f919861e7676ad7cb8a38019f698
Author: irmin9p <irmin@openmirage.org>
Date:   Wed Apr 27 17:07:17 2016 +0000

    (no commit message)

This shows us we are using openmirage for some of the internals

Let us look what is here

$ tree
.
└── com.docker.driver.amd64-linux
    ├── etc
    │   ├── docker
    │   │   └── daemon.json
    │   ├── hostname
    │   └── sysctl.conf
    ├── expose-docker-socket
    ├── filesystem
    ├── hypervisor
    ├── insecure-registry
    ├── memory
    ├── native
    │   ├── boot-protocol
    │   ├── port-forwarding
    │   └── uefi-boot-disk
    ├── ncpu
    ├── network
    ├── on-sleep
    ├── proxy-verbose
    ├── schema-version
    ├── slirp
    │   ├── docker
    │   └── host
    ├── version
    └── vmnet-simulate-failure

5 directories, 20 files

xhyve

So we see from our process listing xhyve - this is what is providing the containerisation/virutalization analgous to lxc. This builds on a new feature in OS X from 10.10 Hypervisor framework.

For now we can

cat /Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/xhyve.args
-A -m 2G -c 2 -u -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-ipc,uuid="e34e7389-c6ba-4f70-9f12-3a63edc453e6",path="/var/tmp/com.docker.vmnetd.socket",macfile="/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0" -s 3:0,virtio-ipc,uuid="2954abe4-b876-45d0-948c-1b1c51e80bdb",path="/var/tmp/com.docker.slirp.socket",macfile="/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.1" -s 4,virtio-blk,'file:///Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2' -s 5,virtio-9p,path="/var/tmp/com.docker.db.socket",tag=db -s 6,virtio-rnd -s 7,virtio-9p,path=/var/tmp/com.docker.port.socket,tag=port -s 8,virtio-sock,guest_cid=3,path='/var/tmp/com.docker.vsock',guest_forwards=2376;1525 -l com1,pty="/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty",log="/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring" -f 'kexec,/Applications/Docker.app/Contents/Resources/moby/vmlinuz64,/Applications/Docker.app/Contents/Resources/moby/initrd.img,earlyprintk=serial console=ttyS0 com.docker.driverDir="/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux", com.docker.database="com.docker.driver.amd64-linux"'

I’ll investigate this further in a future post but lots of useful information - eg

file /Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
/Users/pnasrat/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2: Qemu Image, Format: Qcow

Networking

I’ll investigate this in a future post

Filesystem

I created a simple Dockerfile to explore image and container storage

$ echo hi > hello
$ cat Dockerfile
FROM scratch
ADD hello /
CMD ["/hello"]
$ docker build .

I’ll investigate this in a future post.

Sources

As part of the beta there is a link to the GPL software used. A list of OSS licenses is in /Applications/Docker.app/Contents/Resources/OSS-LICENSES sources available.