Lab A12 - Containerization and Docker

Introduction

This lab is extra credit. It can be used to meet the n-2 lab requirement to pass this DeCal.

Docker

This exercise is designed to give you some hands-on experience with Docker! By the end of this assignment, you should be able to:

Just a forewarning: this lab holds your hand until the last section. Not that the last part is super hard, but it’ll have a lot less instruction than previous portions which are designed to gently introduce you to Docker.

Installing Docker

Unfortunately, Docker isn’t in the official Debian apt repository. This means there are two methods we can use to install it:

  1. Add the official Docker repository to /etc/apt/sources.list. This allows us to update the version of Docker we have on our machines with a simple apt update and apt upgrade
  2. Download the .deb manually and just install it. If we want to upgrade Docker after we do this, we’ll have to do it manually as well.

You’re welcome to use either method to install Docker. For the first method, refer to the Docker website for instructions. For the second method, here is a link to a recent .deb you can use: https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.5~3-0~ubuntu-bionic_amd64.deb. At this point in the course, I trust you can figure out how to install it just given that link 🙂 hint: use wget anddpkg

After installing, I recommend running the command sudo usermod -aG docker $USER so you can use Docker as a non-root user. This means you won’t have to type sudo docker all the time. This is optional but for the rest of this exercise I’m going to assume that you did this. If you see some output like

sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting

This is normal, it’s just adding a user to a group.

Creating your first Docker container

To verify that you installed things correctly, try running

docker run hello-world

You should see some friendly output like so (hashes are probably different, don’t worry about it):

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b04784fba78d: Already exists
Digest: sha256:f3b3b28a45160805bb16542c9531888519430e9e6d6ffc09d72261b0d26ff74f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Some quick definitions from Docker’s website:

An image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files. Images are useful primarily for their speed, but images can also be used as a base to be built on top of in future images, as you’ll see later with Dockerfiles. In the last example hello-world was the image used to test our docker installation.

A container is a runtime instance of an image—what the image becomes in memory when actually executed. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so. A container gets created upon executing docker run on an image.

This is similar to the distinction between objects and classes in Object Oriented Programming. Images would be classes, and containers would be objects.

Be sure to read through the output from running the hello-world image to get an understanding of what the Docker daemon was doing.

Running an interactive container

We’re now going to walk you through running a container interactively. This is useful if you ever need to play around and install stuff on a bare system without messing up your current system. Try running the following command:

docker run -it ubuntu:xenial /bin/bash

The -i flag tells docker to keep STDIN open to your container, and the -t flag allocates a pseudo-TTY for you. Basically you need both for you to have a way to enter text and have this display properly. At the end of the command, /bin/bash is just the command you want to run once the container starts up. Try installing some packages from apt or just play around. It should look like a bare Linux system.

You can exit the container with CTRL+D.

Notice how even though your VM is running Debian, you were able to run the Xenial version of Ubuntu in a container. If you are curious about other variants of Linux, you can run a lot of them inside containers as well! This all works because Linux distributions all share the Linux kernel. For that same reason, you won’t be able to run MacOS or Windows in a container. You can try running Fedora (*tips hat* M’Linux), another long-running Linux distribution:

docker run -it fedora:latest /bin/bash

Questions (answer on Gradescope)

  1. What user are you logged in as by default?
  2. If you start and then exit an interactive container, and then use the docker run -it ubuntu:xenial /bin/bash command again, is it the same container? How can you tell?

Basic Management

The Docker CLI (Command Line Interface) has some basic commands for you to monitor running and stopped containers, downloaded images, and other information. We’ll go over the basic commands you’ll probably use, but be sure to check out the full reference if you’re interested.

Firstly, you might want to see the running containers on a system. Use the following command:

docker ps

Since you (likely) have no containers running, you probably won’t see anything interesting. However, if you pass in the -a flag, you’ll also be able to see containers that have stopped (make your terminal wider or it’ll display weird):

baisang@rapture ~/d/labs> docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                            PORTS               NAMES
35c048c03588        fedora:latest       "/bin/bash"              7 minutes ago       Exited (130) About a minute ago                       mystifying_edison
dd8f7cc2e0cd        fedora:latest       "/bin/bash"              10 minutes ago      Exited (1) 8 minutes ago                              romantic_mahavira

This lets you see a lot of useful information about the container. Observe that each container has a unique container id and a unique human-readable name. To get more information about a container, you can use the docker logs command to fetch the logs of a container (whether it’s still running or exited):

docker logs <container_id_or_name>

This basically just gives you stdout and stderr for process(es) running in the container.

At some point, you may want to cleanup containers that have exited and you don’t plan on using anymore:

docker rm <container_id_or_name>

will remove the container.

You may have noticed when you were running the Ubuntu or Fedora containers the first time that Docker downloaded a good chunk of data before running the image. This is the image of the container. You can view all of the images you’ve downloaded with the docker images command:

baisang@rapture ~/d/labs> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
fedora              latest              9110ae7f579f        2 weeks ago         235MB
ubuntu              xenial              f975c5035748        3 weeks ago         112MB

Images can take up quite a bit of space on your machine, so you may want to clean up images that you don’t plan on using. This is especially relevant if you get errors about not having enough disk space on your machine:

docker rmi <image_id>

The image files, as well as various filesystems of containers, are stored in /var/lib/docker

We’ll go over more commands later on in the lab.

Dockerfiles

A more powerful way to interface with Docker is by using a Dockerfile. A Dockerfile allows you to define an image by specifying all of the commands you would type manually to create an image. Docker can then build images from a specified Dockerfile. These Dockerfiles can be put into version control and the images distributed as a binary to keep track of both how the image is constructed and also to keep pre-built images around.

Dockerfiles are very powerful and have many different commands and features. We’ll go over a basic example, but you should check out the reference page if you are trying to do anything more complex.

Here is an example Dockerfile that will build an image that has python3.6 installed. It will also run python3.6 directly, so you’ll be at a python prompt instead of a bash prompt when you run it.

FROM ubuntu:bionic

RUN apt-get update && apt-get install -y python3.6 --no-install-recommends

CMD ["/usr/bin/python3.6", "-i"]

Note: there are some “best practices” for writing Dockerfiles that the above example doesn’t use, because it’s a basic example. For instance, we probably would want to delete /var/lib/apt/lists/*, where apt stores the package list information from apt update, after we are done installing packages. We may also choose to use Linux variants that are smaller and lighter, e.g. Alpine Linux. The general philosophy is containers should be kept as small and “light” as possible. If you’re interested in this stuff, check out this article.

What is this doing? We specify a base image ubuntu:bionic (release 18.04 of ubuntu, soon to be the latest LTS as of writing). We then specify that we should run (RUN) the command apt-get update and then apt-get install python3.6 so we can install python3.6 (note that this package probably isn’t in apt on your own VMs, which are running an older Debian stable!) Then we set the default command (CMD) of the container to run the python3.6 interpreter in interactive mode.

Copy the contents of the Dockerfile above into a file named Dockerfile. Then use Docker to build it with the following command:

docker build -t mypython:latest .

This tells Docker to look in the current directory for a Dockerfile to build, and build it. The -t flag tells it to tag this build with the name mypython:latest. Docker will look for a Dockerfile in the current directory since you specified .

Remember, you can see all of the images you’ve built on your machine with the docker images command.

Questions

  1. Run the image you just built. Since we specified the default CMD, you can just do docker run -it mypython:latest. What do you observe?
  2. Write and build a Dockerfile that installs the packages fortune and fortunes-min and runs the fortune executable (located in /usr/games/fortune after you install it). Note that you won’t need to use the -it flags when you run the container as fortune doesn’t need STDIN. Submit your Dockerfile with this lab. Hint: if you’re having trouble writing your Dockerfile, try booting an interactive container and installing both packages. Translate what you did into a Dockerfile. How can you translate what you did interactively to a Dockerfile?
  3. Paste the output of your docker images command after questions 1 and 2.

Detached Containers and Ports

You might not always want containers to be running interactively. For instance, if you are running a web server, you’ll likely want the container to continue keep running until you explicitly want to end it. Docker supports this use case with the -d flag, which starts containers in detached mode.

We’ll explore a bit about detached containers by running a standalone Apache container. The image has already been built for you; you can find it on Docker Hub as httpd.

Docker creates a separate virtual network for containers, so you will need to do forward your host port to your container’s port (this is called port forwarding, or port mapping). The container is listening on port 80, so let’s try to forward our host machine’s port 5050 to the container’s port 80 when we run the container:

docker run -d -p=5050:80 httpd

The -p flag takes in a colon separated pair of HOST_PORT:CONTAINER_PORT (it can actually accept a ton of more options, but don’t worry about that for now). You should be able to view visit <url_of_host_machine>:5050, assuming you don’t have anything else running on that port (if you’re not on campus, you can just curl <url_of_host_machine>:5050 from your VM or another machine on campus, e.g. ssh.ocf.berkeley.edu), and see the words “It works!”. You may need to allow the port 5050 on your firewall, simply run the command ufw allow 5050

You can actually “attach” to running containers and run more commands in them, similar to how docker run works. Use the docker exec command:

docker exec <container_id_or_name> <command>

To stop this container, use docker stop <container_id_or_name>.

You can restart the container using docker restart <container_id_or_name>.

Questions

  1. With httpd running in a detached container, run /bin/bash on the same container and paste the output of ps aux. Observe that there’s very few processes running as compared to running ps aux on your VM. Why is this the case?

Dungeons and docker-compose

Congratulations! You’ve just been hired by some trash SF Bay Area tech bubble startup as their systems administrator. Unfortunately, both the CEO and CTO are busy handling the business side, which leaves it up to you to get their web application deployed using Docker and docker-compose.

Don’t worry though – while you may not have health insurance or a nice salary, you do have some of the CTO’s notes and equity to help you with your task. You get off BART at 12pm and enter your cramped SOMA coworking space, sitting down at the desk you share with the CTO while cracking open a cold LaCroix. Checking your email, you find the following notes from the CTO:

About docker-compose

docker-compose lets you define applications that require more than one container to function. For example, on a web application you may want your actual web application running inside of a single container, and your database running in a different container.

Typically you define applications in terms of services. Again, going with the web application example, there are two distinct services: the app itself, and the database backing it. docker-compose lets you define different services within a YAML file and run the services accordingly.

One of the nice things about docker-compose is that it automatically sets up a network for your containers in which:

This means it should be pretty simple to get our web app to connect to the database.

Installing

docker-compose is really just a python package, so ideally we should just install it with virtualenv and pip (as with most things, the version packaged in Debian stable is too old, so we can’t just apt install it). Feel free to do it this way if you want, but since we’re a hotshot tech startup we’re gonna do this the cowboy way (yeehaw):

sudo curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

The web application

The web application is written only the most badass rockstar tech, Node.js. For the database, it uses the most webscale, reliable, and persistent database available on the market today, MongoDB.

everything he just said was wrong

The web application can be found on GitHub. Note that the web app listens on port 8080, so you’ll need to forward or expose that port. Don’t forget to allow it on your firewall, ufw allow 8080. Instructions for setting it up are located in the README.md of the repository: https://github.com/0xcf/decal-sp18-a10

For MongoDB, you can just use the image on DockerHub (a website where people can upload built Docker images). It’s just called mongo. For example, if you wanted to run MongoDB within a container in detached mode:

docker run -d mongo:latest

Putting it all together

Your task is to use docker-compose to deploy the Node.js app with the MongoDB database. The CTO has roughly mapped out a suggested order of tasks:

  1. Write a Dockerfile that will allow you to run the Node.js web application
  2. Write a docker-compose.yml file that will glue the Node.js app container with a MongoDB container

Here is a basic skeleton for docker-compose.yml. You will need to fill out the web and database service entries:

version: '3'
services:
  web:
  database:

By default, the Node.js web application is designed to look for a MongoDB instance at hostname database, so be sure that your MongoDB service is defined under that name. docker-compose will make sure that the hostname database maps to the container for that service.

One huge caveat: your hotshot CTO unfortunately wrote the Node.js app in a crap way where if it’s not able to connect to the MongoDB database once it starts, it’ll fail and exit without retrying. Since Javascript is poison, you’ll need to find a way to make sure that the Node.js app only starts after the MongoDB container is ready to accept incoming connections without modifying server.js. I included a wrapper script wait-for in the repo for the Node.js app that will allow you to wait for the MongoDB service to be ready before launching the Node.js app. But, in order to use the script, you will need to have netcat installed in your container, so be sure to include that in your Dockerfile. See the repo for the script for instructions on how to use the script. Feel free to come up with other ways to solve this issue though!

You will likely find the Compose File Reference useful. Additionally, the Getting Started guide will help as well, although it’s a python example instead of the superior Node.js /s I suggest poking around at other docs on that site also. I expect you to run into errors and difficulties – this is intended as part of the lab. Please try googling error messages and coming up with your own fixes before asking for help, but of course feel free to ask if you get really stuck.

Hints (if you want them):

Once you’ve set things up properly, just running docker-compose up in the same directory as the docker-compose.yml file will bring up your web application!

Questions

  1. Paste your Dockerfile for the Node.js web application
  2. Paste your docker-compose.yml file

Submission

Don't forget to submit to Gradescope!