Title

class: center, middle

# Docker for Cluster Computing in Python

Olivier Grisel - EuroSciPy 2016

.affiliations[
  ![Inria](images/inria-logo.png)
  ![scikit-learn](images/scikit-learn-logo.png)
]

---
# Outline

.middlebelowheader[

### Distributed ML with joblib and dask/distributed

### A quick intro to Docker

### Cluster orchestration with Docker and Kubernetes

]

---
class: center, middle, bgheader
background-image: url(images/racks.jpeg)
background-size: cover

# Distributed ML with joblib and dask/distributed

---
class: singleimg
# A cluster for sklearn

.middlebelowheader[

![Cluster diagram](images/gridsearch-user-case.png)

]

---
# Demo

.middlebelowheader[

https://github.com/ogrisel/docker-distributed

Have a look a the notebooks in the `examples` folder.

.center[

<iframe width="420" height="315"
  src="https://www.youtube.com/embed/6mKNSEQ0FIQ"
  frameborder="0" allowfullscreen></iframe>

https://youtu.be/6mKNSEQ0FIQ

]
]

---
class: center, middle, bgheader
background-image: url(images/container_terminal.jpeg)
background-size: cover

# What is Docker?

---
# Docker is:

.middlebelowheader[

### Not a virtual machine system

### Linux container + Layered file-system + Abstract network

### It can run in a VM (or not)

]

---
class: singleimg
# VM vs Container

.middlebelowheader[

![VM-based hosting vs Container-based hosting](images/container-vs-vm.png)

]

---
# Why is docker nicer than VMs

.middlebelowheader[

### Less overhead: isolation vs virtualization

### Layered FS: incremental diffs + common base images

### Nice command line based UX

### Simple reproducible builds and deployment

]

---
class: singleimg
# Docker architecture

.middlebelowheader[

![Architecture of Docker](images/docker-architecture.svg)

]

---
# Some use cases

.middlebelowheader[

### Host applications / services efficiently

### Fast developer environment setup

### Flexible build / Continuous Integration

### Quickly provision a compute cluster

### Reproducible science

]

---
# Getting started with Docker

.middlebelowheader[

https://docs.docker.com/engine/installation/

### Windows: daemon in Linux with Hyper-V or Virtualbox

### OSX: daemon in Linux with xhyve or Virtualbox

### Linux: daemon on host without VM

### Cross-platform docker client binary

]

---
class: medium
# Running in a container

```
$ docker run python:3.5 python -c "print(40 + 2)"
```

---
class: medium
# Running in a container

```
$ docker run `python:3.5` python -c "print(40 + 2)"
```

---
class: medium
# Running in a container

```
$ docker run python:3.5 `python -c "print(40 + 2)"`
```

---
class: medium
# Running in a container

```
$ docker run `--rm` python:3.5 python -c "print(40 + 2)"
```

---
class: medium
# Running in a container

```
$ docker run --rm python:3.5 python -c "print(40 + 2)"
Unable to find image 'python:3.5' locally
latest: Pulling from library/python

357ea8c3d80b: Already exists
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Downloading [=========>    ] 90.27 MB/129.7 MB
4211bb537697: Download complete
71f9074c0739: Download complete
3e5349707036: Download complete
```

---
class: medium
# Running in a container

```
$ docker run --rm python:3.5 python -c "print(40 + 2)"
Unable to find image 'python:3.5' locally
latest: Pulling from library/python

357ea8c3d80b: Already exists
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Pull complete
4211bb537697: Pull complete
71f9074c0739: Pull complete
3e5349707036: Pull complete
Digest: sha256:a755ad5a30b2[...]
Status: Downloaded newer image for python:3.5
42
```
---
class: medium
# Running in a container

```
$ docker run --rm python:3.5 python -c "print(40 + 2)"
Unable to find image 'python:3.5' locally
latest: Pulling from library/python

357ea8c3d80b: Already exists
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Pull complete
4211bb537697: Pull complete
71f9074c0739: Pull complete
3e5349707036: Pull complete
Digest: sha256:a755ad5a30b2[...]
Status: Downloaded newer image for python:3.5
42
$ docker run --rm python:3.5 python -c "print(40 + 3)"
43
```
---
class: medium
# Interactive sessions

```
$ docker run --rm -ti python:3.5 bash
root@10d2dfedb935:/#
```
---
class: medium
# Interactive sessions

```
$ docker run --rm -ti python:3.5 bash
root@10d2dfedb935:/# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
    8 ?        00:00:00 ps
root@10d2dfedb935:/#
```
---
class: medium
# Interactive sessions

```
$ docker run --rm -ti python:3.5 bash
root@10d2dfedb935:/# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
    8 ?        00:00:00 ps
root@10d2dfedb935:/# python
Python 3.5.2 (default, Aug  9 2016, 20:58:38)
[GCC 4.9.2] on linux
>>>
```
---
class: medium
# Interactive sessions

```
$ docker run --rm -ti python:3.5 bash
root@10d2dfedb935:/# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
    8 ?        00:00:00 ps
root@10d2dfedb935:/# python
Python 3.5.2 (default, Aug  9 2016, 20:58:38)
[GCC 4.9.2] on linux
>>> 40 + 2
42
```
---
class: medium
# Interactive sessions

```
$ docker run --rm -ti python:3.5 bash
root@10d2dfedb935:/# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
    8 ?        00:00:00 ps
root@10d2dfedb935:/# python
Python 3.5.2 (default, Aug  9 2016, 20:58:38)
[GCC 4.9.2] on linux
>>> 40 + 2
42
>>> ^D
root@10d2dfedb935:/# exit
```

---
class: medium
# A real-life example

```
$ cd bokeh/bokeyjs
$ npm install -g gulp
$ gulp build
[Bunch or errors caused by a broken nodejs install]
```

---
class: medium
# A real-life example

```
$ cd bokeh
$ docker run --rm --volume $PWD:/io node \
    bash -c "cd /io/bokehjs && npm install -g gulp && gulp build"
```

---
class: medium
# A real-life example

```
$ cd bokeh
$ docker run --rm `--volume $PWD:/io` node \
    bash -c "`cd /io/bokehjs` && npm install -g gulp && gulp build"
```

---
class: medium
# A real-life example

```
$ cd bokeh
$ docker run --rm --volume $PWD:/io node \
    bash -c "cd /io/bokehjs && npm install -g gulp && gulp build"
npm info it worked if it ends with ok
npm info using npm@3.10.3
npm info using node@v6.4.0
npm info attempt registry request try #1 at 12:45:51 PM
npm http request GET https://registry.npmjs.org/gulp
[...] # Download the Internet
[13:03:42] Finished 'build' after 10 s
*$ ls bokehjs/node_modules | wc -l
*604
```

---
class: center, middle, bgheader
background-image: url(images/lego-painter.jpeg)
background-size: cover

# Building an image

---
class: middle, medium

```
$ git clone https://github.com/ogrisel/docker-distributed
$ ls docker-distributed
docker-compose.yml
*Dockerfile
examples
kubernetes
README.md
*requirements.txt
```

---
class: middle, medium

```
$ git clone https://github.com/ogrisel/docker-distributed
$ ls docker-distributed
docker-compose.yml
Dockerfile
*examples
kubernetes
README.md
requirements.txt
```

---
class: middle, medium

```
$ git clone https://github.com/ogrisel/docker-distributed
$ ls docker-distributed
*docker-compose.yml
Dockerfile
examples
*kubernetes
README.md
requirements.txt
```

---
class: middle

```dockerfile
FROM debian:jessie
MAINTAINER Olivier Grisel <olivier.grisel@ensta.org>

RUN apt-get update -yqq && apt-get install -yqq wget bzip2 git \
  && rm -rf /var/lib/apt/lists/*

# Configure environment
ENV LC_ALL=C.UTF-8 LANG=C.UTF-8

RUN mkdir /work
WORKDIR /work

# Install Python 3 from miniconda
RUN wget -O miniconda.sh \
  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
  && bash miniconda.sh -b -p /work/miniconda \
  && rm miniconda.sh

ENV PATH="/work/bin:/work/miniconda/bin:$PATH"

RUN conda install -y \
  pip \
  notebook \
  pandas \
  scikit-learn \
  && conda clean -tipsy

# Install the master branch of distributed and dask
COPY requirements.txt .
RUN pip install -r requirements.txt && rm -rf ~/.cache/pip/

# Add the example notebooks
COPY examples .
```

---
class: medium
# docker build

```
$ docker build -t ogrisel/distributed .
Sending build context to Docker daemon 178.2 kB
Step 1 : FROM debian:jessie
 ---> 1b01529cc499
Step 2 : MAINTAINER Olivier Grisel <olivier.grisel@ensta.org>
 ---> Using cache
 ---> 37887ee139f1
Step 3 : RUN apt-get update -yqq  && apt-get install -yqq wget [...]
 ---> Using cache
 ---> 3c2b8caccb80
[...]
```

---
class: medium
# docker build & docker push

```
$ docker build -t ogrisel/distributed .
Sending build context to Docker daemon 178.2 kB
Step 1 : FROM debian:jessie
 ---> 1b01529cc499
Step 2 : MAINTAINER Olivier Grisel <olivier.grisel@ensta.org>
 ---> Using cache
 ---> 37887ee139f1
Step 3 : RUN apt-get update -yqq  && apt-get install -yqq wget [...]
 ---> Using cache
 ---> 3c2b8caccb80
[...]
$ docker run --rm ogrisel/distributed \
    python -c "import sklearn; print(sklearn.__version__)"
0.17.1
$ docker push ogrisel/distributed
```

---
class: center, middle, bgheader
background-image: url(images/orchestra.jpeg)
background-size: cover

# Orchestration

---
# Orchestration tools

.middlebelowheader[
<div style="display: flex">

<div style="width: 50%; padding: 1em">
<p><img height="50px" src="images/docker-logo.png" /></p>
<p>
<strong>Docker Swarm</strong> and Docker Compose
</p>
</div>

<div style="width: 50%; padding: 1em">
<p><img height="50px" src="images/kubernetes-logo.png" /></p>
<p>
<strong>Kubernetes</strong>
</p>
</div>

</div>

<div style="margin: 1em auto; width: 400px">
<p><img height="50px" src="images/dcos-logo.png" /></p>
<p>
<strong>DC/OS</strong> and Mesos
</p>
</div>
]

---
class: medium
# Kubernetes and GKE

```
$ gcloud config set compute/zone europe-west1-d
$ gcloud container clusters create cluster-1 \
    --num-nodes 3 \
    --machine-type n1-highcpu-32 \
    --scopes bigquery,storage-rw \
    --wait
Creating cluster cluster-1...done.
Created [https://container.googleapis.com/v1/.../cluster-1].
kubeconfig entry generated for cluster-1.
NAME       ZONE            ... MACHINE_TYPE   NUM_NODES  STATUS
cluster-1  europe-west1-d  ... n1-highcpu-32  3          RUNNING
```
--
```
$ gcloud container clusters get-credentials cluster-1
Fetching cluster endpoint and auth data.
kubeconfig entry generated for cluster-1.
```

---
class: medium
# Kubernetes and GKE

```
$ git clone https://github.com/ogrisel/docker-distributed
$ cd docker-distributed
$ kubectl create -f kubernetes/
service "dscheduler" created
service "dscheduler-status" created
replicationcontroller "dscheduler" created
replicationcontroller "dworker" created
service "jupyter-notebook" created
replicationcontroller "jupyter-notebook" created
```
--
```
$ kubectl get services
NAME                CLUSTER-IP       EXTERNAL-IP      PORT(S)
dscheduler          10.115.249.189   <none>           8786/TCP,9786/TCP
dscheduler-status   10.115.244.201   130.211.50.206   8787/TCP
jupyter-notebook    10.115.254.255   146.148.114.90   80/TCP
kubernetes          10.115.240.1     <none>           443/TCP
```

---
class: medium
# distributed-worker.yml

```yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: dworker-controller
spec:
  replicas: 3
  selector:
    name: dworker
  template:
    metadata:
      labels:
        name: dworker
    spec:
      containers:
        - name: dworker
          image: ogrisel/distributed:latest
          args: ["dask-worker", "dscheduler:8786"]
```

---
# Conclusion

.middlebelowheader[

### Docker is a useful developer tool

### Containers make cluster setup reproducible and cloud agnostic

### Docker Swarm vs Kubernetes vs Mesos: learn any one and understand others

]

---
# Thank you

.middlebelowheader[
<div style="display: flex">

<div style="width: 50%; padding: 1em">
<p><img height="35px" src="images/carina-logo.svg" /></p>
<p>
<strong>Rackspace</strong> for free cloud resources to support SciPy
projects and for helping the Python community in general.
</p>
</div>

<div style="width: 50%; padding: 1em">
<p><img height="35px" src="images/gcp-logo.png" /></p>
<p>
<strong>Google</strong> for GCP credits to test distributed Python with
Kubernetes.
</p>
</div>

</div>

<div style="margin: 1em auto; width: 400px">
<p><img height="50px" src="images/inria-logo.png" /></p>
<p>
<strong>Inria</strong> for supporting my work on scikit-learn and related
projects.
</p>
</div>
]

---
class: middle

# Thank you!

## Sample configuration:

- [github.com/ogrisel/docker-distributed](http://github.com/ogrisel/docker-distributed)

## Those slides:

- [ogrisel.github.io/decks/2016_euroscipy_docker](
    http://ogrisel.github.io/decks/2016_euroscipy_docker)

---
class: small

# Image credits

- "Supermicro Storage" by Robert: https://flic.kr/p/e17qr3

- "smiiile!" by mooglet: https://flic.kr/p/am9zQe

- "Docker Architecture" https://docs.docker.com/engine/understanding-docker/

- "Container terminal" by Martin Abegglen: https://flic.kr/p/b7rw5D

- "Symphonic Orchestra and Choir of the WYD" by HazteOir.org:
  https://flic.kr/p/c5B6Cd