Background

Prior to using our own project-specific GitLab Runner, our pipeline was reliant on GitLab Shared Runners for executing our CI / CD jobs. Unfortunately, we did not set a job timeout, and a rogue job ended up consuming all our free minutes for the month. Without any minutes left for the month, none of our automated jobs were being executed, thereby blocking any changes from going live. To unblock the pipeline, we needed to start our own runners.

Kubernetes

As we were already using Kubernetes, we decided that the simplest option would be to launch the GitLab Runner on it using GitLab’s integration, and Helm. We had already enabled Autoscaling on our Google Cloud Kubernetes Engine (GKE) node pool so we were not concerned about the bare metal requirements.

After integrating our Kubernetes cluster in GitLab, and setting up Helm, we installed the GitLab Runner, and begin running jobs. When tailing the logs however, very little progress was being made across all our concurrent jobs. When we checked the workload on our Kubernetes cluster we discovered that several nodes had become unhealthy, and some of our services were impacted by the jobs. Our node pool however, did not autoscale immediately to account for the load. We thought that this might have been the reason behind several of our jobs timing out, and taking down the underlying node with them. But, after a manual scaling of the node pool, our jobs continued to take very long to progress, even despite being on nodes without any workload.

We concluded that it was likely that our nodes (f1-micro, why we went with this among other decisions will be covered another time) were not big enough to support the requirements of our concurrent jobs, and that we should have designated a separate Kubernetes cluster for running non-customer facing workload. Alternatively, we could have perhaps used node affinity to dedicate specific nodes in a cluster for these non-customer facing workload. Unfortunately, as we used GitLab’s Kubernetes integration to automatically install the GitLab Runner, we were not too familiar with configuring the runner. Our numerous attempts at configuring it through Kubernetes were unsuccessful. Realising that we had already spent a lot of time, we opted for trying to run GitLab Runner manually via Docker.

Hetzner Cloud with Docker

Prior to this incident, I had already created an account with Hetzner to launch a Minecraft server for my friends, and colleagues. My experience with them thus far has been positive. Initially, the network connectivity was occasionally poor between certain regions, but it has since improved.

We created, and purchased a CX21 instance for $7, and within 5 minutes it was up, and running. We could have used other cloud providers, but for the specifications that Hetzner offered, we believed that it was the best value for money option currently available in the market. We received an instance with 2 CPU, 4 GB RAM, and 40 GB SSD.

After SSH’ing into our instance, we installed Docker, and begin trying to follow the instructions on “Run GitLab Runner in a container”. In hindsight, the process can be simplified tremendously.

Running GitLab Runner using Docker

These are simplified instructions for running GitLab Runner on any machine with Docker installed. It assumes that Docker is installed, and ready-to-use.

1. Register Runner

   docker run --rm -t -i \
       -v /srv/gitlab-runner/config:/etc/gitlab-runner \
       --name gitlab-runner-register \
       gitlab/gitlab-runner register

The command above will launch gitlab-runner calling the register command. It will create some basic configuration which you can find in /srv/gitlab-runner/config on your host machine (copied from Docker as we mounted the volume).

We used docker:stable as the base image, but use whatever you prefer. You can find the runner URL, and token in your project’s CI / CD settings (look for Specific Runners) → https://gitlab.com/<user or org>/<project>/settings/ci_cd.

2. Configure

Modify /srv/gitlab-runner/config/config.toml if necessary. For example, we adjusted the concurrent to 2 so that we can run 2 jobs in parallel, and we enabled privileged to run Docker inside Docker (aka dind).

3. Launch the Runner

docker run -d --name gitlab-runner --restart always --privileged \
    -v /srv/gitlab-runner/config:/etc/gitlab-runner \
    -v /var/run/docker.sock:/var/run/docker.sock \
    gitlab/gitlab-runner:latest

See https://docs.gitlab.com/runner/install/docker.html#update for instructions on upgrading.

4. Done!

Your dedicated GitLab Runner should now be picking up jobs.

Closing

I hope this post will have helped / inspired you to setup GitLab Runner with ease on your own Docker-enabled machine!

You do not have to wait to do this if you still have minutes left on your shared runners. For example, we noticed improvements in our pipeline speed as a result of using our own dedicated runner, and we are likely to continue using our dedicated runner once our minutes are restored.

We could have launched the runner in a separate, and dedicated Kubernetes cluster, we felt that it would be overkill considering our CI / CD workload. Most of our instances would be idle more than half the time. Furthermore, we were not too comfortable running the workload on the same cluster as our production services due to the latency in autoscaling among other isolation, and configuration issues.

In the future, we hope to create an instance template (or “snapshot”) from which we can use to provision GitLab Runners on-demand via a script.