Bits of Freedom
Bits of Freedom

Tinkerer and thinker on everything free and open. Exploring possibilities and engaging with new opportunities to instigate change.

Jonas Öberg
Author

Jonas is a dad, husband, tinkerer, thinker and traveler. He's passionate about the future and bringing people together, from all fields of free and open.

Share


My Newsletters


If you're interested in what Commons Machinery, Elog.io, or myself are up to, I'd love for you to be part of my notification lists. It's pretty low volume, a few messages per month, depending on which notifications you sign up for. Thanks for taking an interest!

Read more & subscribe
Bits of Freedom

Containerising the FSFE, aiming for Kubernetes & 2020

Jonas ÖbergJonas Öberg

When the FSFE started using virtualisation technologies on our servers, about 15 years ago, we decided to use Linux-Vservers, a budding project which Jacques Gélinas had started just a year or two prior. To those of us administering the FSFE machines at the time, it was quite like magic, being able to run several virtual containers on one machine, and being able to delegate control of them to different teams without fearing for the overall security or stability.

A number of years later, we started migrating our Linux-Vservers to Proxmox, and as of last week, we've also added Docker as an experiment for our deployments of new services. I will try to give a bit of an insight into what this looks like right this moment, with the caveat that this is a work in progress. I should also say up front that while we're currently using GitLab for some of our repositories, we also have a self-hosted installation of Gitea which isn't publicly released yet and part of the work we're doing is building towards actually being able to make better use of that Gitea, rather than GitLab.

Let's start at the bottom of the stack: the individual service. I'll be using our oidcp (OpenID Connect Provider) as an example in this.

Service level

The oidcp is the authentication provider of our new account management system. The master branch of this is on GitLab (https://gitlab.com/lukewm/oidcp/) and from the perspective of deployments, there are two critical pieces in that code, the Dockerfile and GitLab-CI File. The latter file is used by the tools for continuous integration and defines three separate stages: testing, building and deployment.

Changes to the master branch are typically made as merge requests. I maintain my own version of the branch for my own development, and when it's reached a certain point of maturity, I file a merge request against the master branch.

For any commit that is made, to a fork or branch of the main repository, the CI tools run through the three stages defined in the setup. This consists of quality checking the code, running our test suites and testing to build the code. Once all of those tasks have been done and succeeded, we get a nice green check mark showing that the commit has passed and -- if it's a merge request -- is probably fine to merge, from a quality and test perspective at least.

The Dockerfile is fairly simple, but if you haven't used Docker before, it may seem like magic. Think of Docker as a bunch of helper tools around a chroot. It's not entirely fair towards Docker, but it feels as if that's where it all started. Each Docker container has everything it needs to be a self-contained unit and doesn't (typically) share anything with any other container.

The Dockerfile for oidcp starts off by telling Docker that we would like our oidcp container to be built using Ubuntu zesty as the base container. This is how Docker typically work: you have a number of base containers, which you add something to, and then use those as new base containers for other work, and so on.

When constructing a container for oidcp, Docker starts by pulling in a complete Ubuntu zesty, copies over our oidcp code to that running system, sets up a few symlinks, and installs some Python packages needed. This creates an image which we can run on our host machine.

One final note on that Docker image, the Dockerfile also contain instructions about exposing port 8000 to the outside world. When a Docker image runs, it gets connected to a private network and doesn't typically expose any open ports to the outside world. In order to enable the outside world to reach our oidcp on port 8000, we need to explicitly expose it.

As we'll see later though, the "outside world" here is just the host machine. Exposing a port still doesn't mean that anyone outside of the host machine where the service run can access it. For this we need a bit more.

Running containers

Once we have an image built from a Dockerfile, we need to run it somewhere. In Düsseldorf, the FSFE has a cluster of three physical machines running Proxmox. On this cluster, I've created the virtual machine Lund, named after.. well, I'm honestly not sure right now. The naming scheme for the FSFE is that we name our virtual servers after chemists (Cavendish, Ekeberg, Auer, ...), and the physical machines after chemical elements (Argon, Zinc, Gallium, ...).

When I named Lund, I am absolutely sure I found a chemist with the last name "Lund", but when I now try to find said chemist, I come up empty. So even if there is one, he or she is probably not the most famous. But there you have it. One virtual machine. Lund.

Lund is installed with a basic Debian Jessie, and pretty much nothing else. We have an ansible playbook which cares for installing SSH keys, and I then added it to our backup cycle and installed Docker. Otherwise, there's nothing very specific set up on this machine.

Over on our (unofficial) FSFE Gitea installation, I have a repository with ansible playbooks for Lund. In this repository there's a playbook for each service we run on Lund, including one for the oidcp.

To install the oidcp on Lund, I run the corresponding playbook:

$ ansible-playbook -i hosts --ask-vault-pass oidcp.deploy.yml

This runs through the tasks in the oidcp playbook which in turn does the following:

  1. Build the image locally from a given Git repository. In principle, the image is already built on Gitlab, so this step should eventually be changed.
  2. Create a local network, which we'll need to allow selected running containers a way to communicate between each other. It's important to keep in mind that an ansible task is often described as the state one wishes the target object to be in. For the network, we say state: present, giving an indication to ansible we wish the network to be present. If it's not present, ansible will create it. If it's present, with the correct parameters, ansible won't change anything.
  3. We run a standard redis container. This one isn't actually built by us. We just make use of the distribution standard redis which is registered in the Docker registry with the name redis. So all we need to do is tell ansible we would like to run an instance of the redis image, connected to the network we've just created.
  4. We then run also our oidcp container, give it a bunch of environment variables to control its operation, and connect it to the same network, also indicating the alias "redis" should point to the running redis container. If we were to enter the oidcp container and run ping redis, this would work as expected.
  5. We similarly start a celery process using the same container and connected to the same network.

When we've made changes to our Git repository which I'd like to deploy, I just run the relevant ansible playbook and let ansible care for getting it on the machine.

But we're still missing one part: our oidcp provider exposed port 8000, but so far we only have a local network between some of our oidcp containers. Technically, we also have a local network which connects the host machine to its containers, so I can login to Lund and ping the IP numbers of the oidcp containers. But that network is also local to the host machine, so it's not reachable from the outside.

Connecting the network

We don't typically run our services on non-standard ports. We like to keep publicly accessible web services on port 80 (http), and even more on 443 (https). So we need something on Lund to answer on those ports, and we need to tie that to the relevant containers somehow.

I used to run this on the host machine itself, until I realised it's equally possible to put this in a Docker container too, and put the relevant configuration and deployment scripts into Git.

The way I've done this to allow for FSFE specific modifications is I've created a Git repository for fsfe-rp which contains the simplest Dockerfile one could ever imagine almost. It's entirety is:

FROM jwilder/nginx-proxy:alpine  
COPY sites-enabled/ /etc/nginx/conf.d/  

What this does is it creates a Docker image based on the already existing jwilder/nginx-proxy container (which in itself is based on the nginx container), of the alpine version. It then copies any FSFE local configuration files to the nginx configuration directory in the image we build.

Alpine Linux is a popular choice to base Docker containers on. It's a small and light weight distribution using musl libc (instead of GNU libc) and busybox (instead of GNU Bash etc). This makes the entire distribution tiny, around 5MB in size, which is perfect for Docker images where you don't need a fully fledged distribution. You just need the bare minimum to run a service.

What's particular about Jason Wilder's nginx-proxy container is that when it starts, it connects to the Docker daemon running on the host machine. It monitors the running Docker images and whenever a Docker image is started or removed, it springs into action.

It looks for all Docker images which are run with an environment variable VIRTUAL_HOST. For example, our oidcp installation has the following environment:

env:  
  VIRTUAL_HOST: id.fsfe.org

For any image running with such a variable, it automatically adds a configuration to the running nginx. For our oidcp, this configuration looks like this:

upstream id.fsfe.org {  
            ## Can be connect with "bridge" network
            # oidcp
            server 172.17.0.8:8000;
}
server {  
    server_name id.fsfe.org;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://id.fsfe.org;
    }
}

As you can see, it has automatically detected the running container for oidcp, added its domain name as the server name to listen for (on port 80), and added an upstream. The upstream is then connected to the local IP number of that container, with port 8000 (the one exposed by the Docker container).

Any request which comes in on port 80 with the domain name id.fsfe.org is automatically proxied to the oidcp container. This nginx container in itself has port 80 (and 443) exposed to the wider Internet, so requsts to Lund will first pass through the nginx container, and then down to whichever local container is relevant for the request.

Adding SSL

Of course it would be nice if we could also break free from only having HTTP and have HTTPS as the default. This is accomplished by Yves Blusseau's LetsEncrypt companion to nginx-proxy. This is a Docker container which, similarly to nginx-proxy, listens to Docker events and springs into action when a container is added or removed (as well as regularly every hour).

What it does is it looks at the running containers and their environment. It looks for an environment like this:

env:  
  VIRTUAL_HOST: id.fsfe.org
  LETSENCRYPT_HOST: id.fsfe.org
  LETSENCRYPT_EMAIL: jonas@fsfe.org

When it finds the LETSENCRYPT_HOST variable, it gets in touch with LetsEncrypt and requests an SSL certificate for that domain name. When it gets one, it installs it in a folder shared with nginx-proxy, and the configuration of nginx is updated so it also listens to, and responds to HTTPS requests, for the given host. It also updates the HTTP section, so any HTTP requests are automatically redirected to the HTTPS version.

As mention, it also runs every hour. It does this in order to also renew any certificates which may need it. Here's the log output from when it detected id.fsfe.org and generated a certificate for it, and the regular checks for renewals:

2017/05/12 12:44:44 Running '/app/update_certs'  
Creating/renewal id.fsfe.org certificates... (id.fsfe.org)  
2017-05-12 12:44:46,328:INFO:simp_le:1211: Generating new account key  
2017-05-12 12:44:47,178:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org  
2017-05-12 12:44:49,576:INFO:simp_le:1305: id.fsfe.org was successfully self-verified  
2017-05-12 12:44:49,825:INFO:simp_le:1313: Generating new certificate private key  
2017-05-12 12:44:52,338:INFO:simp_le:391: Saving key.pem  
2017-05-12 12:44:52,338:INFO:simp_le:391: Saving chain.pem  
2017-05-12 12:44:52,339:INFO:simp_le:391: Saving fullchain.pem  
2017-05-12 12:44:52,339:INFO:simp_le:391: Saving cert.pem  
...
Creating/renewal id.fsfe.org certificates... (id.fsfe.org)  
2017-05-18 19:36:12,356:INFO:simp_le:1383: Certificates already exist and renewal is not necessary, exiting with status code 1.  
Sleep for 3600s  
...

The end game & where to now?

So we're now at the point where any service added with the right environment will automatically have an SSL certificate generated for it, with a reverse proxy setup to accept requests on its behalf, forwarded to the local container where it runs.

There are a few parts which is missing from this setup. First of all, the deployment which is currently done with ansible should be automated as part of our continuous integration chain. When a Git commit is merged into our master branch, this should trigger the testing, building and deployment of the new version.

Ideally, we should also truly have a separate staging environment, so new versions first gets deployed in a staging environment where we can test it out before deploying it in production. There are also a lot of tools to make container orchestration easier and more manageable, including Kubernetes, which I could see us using in the future.

We still have some ways to go, and if you want to help us containerise the FSFE, we'd love your help!

Jonas Öberg
Author

Jonas Öberg

Jonas is a dad, husband, tinkerer, thinker and traveler. He's passionate about the future and bringing people together, from all fields of free and open.