So You Want to Build Quarkus Native Image Inside Container, Think Again!

Table of Contents

• Overview
• Building Jenkins JNPL Slave Container for GraalVM and Podman
  • First Error Encountered – Podman Unable to Write Events
  • Second Error Encountered – Podman Networking Issue
• Building Jenkins JNPL Slave Container for GraalVM and Docker
  • Docker in Docker (DND)
  • Let’s Use Docker Daemon on The Host
• Use Physical Node or VM Node for Jenkins Slave
• Summary

Overview

I blogged about creating GraalVM container image in my earlier post so that we can use the container as part of our CI process for our Jenkins environment.

As you may have already know, we need to have Podman or Docker runtime available in the same GraalVM build environment in order to build Quarkus native image for Podman or Docker.

I had spent the whole week trying to build the container for GraalVM with Podman, and GraalVM with Docker runtime, both failed miserably. So think again if you are on this journey, at least not for now at the time of this writing, until they are ready for container.

Let us walk through the miserable journey here…

Building Jenkins JNPL Slave Container for GraalVM and Podman

There are number of workaround tweaks that I need to perform in order to make Podman runs with fewer errors in container. Let’s look at each of them.

First Error Encountered – Podman Unable to Write Events

The first error I encountered is something looks like the following. The default Podman is designed for non-container environment, thus it is expecting to write events into the Linux journal.

unable to write pod event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory"

So what I did is to use the following command option (–events-backend=none) in my Jenkins pipeline to instruct the Podman not to write any events. I was thinking after all the events for this short-live container is not important since we are just using it to build Quarkus native image.

sh "podman --events-backend=none build -f docker/Dockerfile -t ${artifactId}:${version} ."

Later I found out that I can hack the configuration for Podman during the container creation process. I patched the events_logger to use file instead of journald. The catch for this is that the container need to be run as privileged container. I thought let’s just proceed to make it work first and worry about that later.

RUN sed -i 's/# events_logger = "journald"/events_logger = "file"/g' $GOPATH/src/github.com/containers/libpod/libpod.conf \
&& cp /var/go/src/github.com/containers/libpod/libpod.conf /etc/containers/

Second Error Encountered – Podman Networking Issue

Next came the big road block that I decided to abandon this container idea. It turns out Podman current networking is not designed for container. The following shows the error that I encountered. You can refer to the reported issue here. Thankfully it is being worked on for the fix now. Probably I will come back to this when it fixed.

time="2020-03-24T09:43:03Z" level=error msg="Error preparing container 12b1b20f183bad5c4215bd53a8dd160e604c62b8c16f566b7f3e99a92ef9619f: error creating network namespace for container 12b1b20f183bad5c4215bd53a8dd160e604c62b8c16f566b7f3e99a92ef9619f: mount --make-rshared /var/run/netns failed: "operation not permitted""

For you reference, the following is the Dockerfile that I was using.

FROM jenkinsci/jnlp-slave
ARG version=1.0.0

LABEL Author="CK Gan" Email="[email protected]" Description="Jenkins jnpl agent for Quarkus." Version="$version"

USER root

ENV GRAALVM_BASE /opt/graalvm
ENV GRAALVM_HOME $GRAALVM_BASE/graalvm-ce-java11-20.0.0
ENV JAVA_HOME $GRAALVM_HOME
ENV GOPATH /var/go
ENV PATH $GOPATH/bin:$JAVA_HOME/bin:$PATH

RUN apt-get -y update && apt-get -y install --no-install-recommends gcc build-essential \
make cmake git btrfs-progs golang-go go-md2man iptables libassuan-dev libc6-dev libdevmapper-dev \
libglib2.0-dev libgpgme-dev libgpg-error-dev libostree-dev libprotobuf-dev libprotobuf-c-dev \
libseccomp-dev libselinux1-dev libsystemd-dev pkg-config runc uidmap libapparmor-dev

RUN mkdir -p $GOPATH \
&& mkdir -p /run/systemd/journal/socket \
&& git clone https://go.googlesource.com/go $GOPATH \
&& cd $GOPATH && cd src && ./all.bash \
&& cd /tmp \
&& git clone -b v2.0.14 https://github.com/containers/conmon \
&& cd conmon \
&& make \
&& make podman \
&& cp /usr/local/libexec/podman/conmon /usr/local/bin/ \
&& git clone https://github.com/containernetworking/plugins.git $GOPATH/src/github.com/containernetworking/plugins \
&& cd $GOPATH/src/github.com/containernetworking/plugins \
&& ./build_linux.sh \
&& mkdir -p /usr/libexec/cni \
&& cp bin/* /usr/libexec/cni \
&& mkdir -p /etc/cni/net.d \
&& curl -qsSL https://raw.githubusercontent.com/containers/libpod/master/cni/87-podman-bridge.conflist | tee /etc/cni/net.d/99-loopback.conf \
&& mkdir -p /etc/containers \
&& curl https://raw.githubusercontent.com/projectatomic/registries/master/registries.fedora -o /etc/containers/registries.conf \
&& curl https://raw.githubusercontent.com/containers/skopeo/master/default-policy.json -o /etc/containers/policy.json \
&& git clone https://github.com/containers/libpod/ $GOPATH/src/github.com/containers/libpod \
&& cd $GOPATH/src/github.com/containers/libpod \
&& make \
&& make install

$GOPATH/src/github.com/containers/libpod/libpod.conf

RUN sed -i 's/# events_logger = "journald"/events_logger = "file"/g' $GOPATH/src/github.com/containers/libpod/libpod.conf \
&& cp /var/go/src/github.com/containers/libpod/libpod.conf /etc/containers/

RUN mkdir $GRAALVM_BASE && mkdir $GRAALVM_HOME
COPY ./graalvm-ce-java11-20.0.0 $GRAALVM_HOME/

RUN chmod -R g+rw $GRAALVM_HOME && gu install native-image

RUN apt-get clean && rm -rf /tmp/*

Building Jenkins JNPL Slave Container for GraalVM and Docker

Docker in Docker (DND)

At first attempt, I was trying to use Jenkins JNPL container image to build with Docker runtime but it seems it is massive amount of effort to do Docker in Docker (DND). One of the challenge there is no way to have Docker daemon to start up in the container. After all it is not recommended to do so for many reasons. Let’s forget about this.

I also tried to the idea of using Docker base image to install GraalVM and Docker inside. I curious how Docker make the daemon runs in the container.

Apparently Docker base image is using Alpine Linux and I was thinking this is not going to be good since we are facing so many problem to get some of the things works in the popular Linux based image.

Abandoned this again…

Let’s Use Docker Daemon on The Host

Let’s try running Docker command against the Docker daemon on the host instead DND. So I proceed to build the container with Docker runtime inside. Configured Jenkins to run the container with the following parameter.

-v /var/run/docker.sock:/var/run/docker.sock

The Jenkins build started and ended with the following error. Remember this error that I mentioned in my previous post. It is caused by out of memory issue. The container was given 2048Mb of memory but it still failed. So I bumped it to 4096Mb. The build still failed with the same error.

Error: Image build request failed with exit status 137 Failed to execute goal io.quarkus:quarkus-maven-plugin:1.2.1.Final:build (default) on project account-service: Failed to build a runnable JAR: Failed to augment application classes: Build failure: Build failed due to errors

Abandoned this approach. Not worth the effort to try to make this works if I need massive amount of memory to have it running.

The following is the Dockerfile used.

FROM jenkinsci/jnlp-slave
ARG version=1.0.0

LABEL Author="CK Gan" Email="[email protected]" Description="Jenkins jnpl agent for Quarkus with Docker." Version="$version"

USER root

ENV GRAALVM_BASE /opt/graalvm
ENV GRAALVM_HOME $GRAALVM_BASE/graalvm-ce-java11-20.0.0
ENV JAVA_HOME $GRAALVM_HOME
ENV PATH $GRAALVM_HOME/bin:$JAVA_HOME/bin:$PATH

#RUN apt-get -y update && apt-get -y install --no-install-recommends build-essential && apt-get clean
RUN apt-get -y update && apt-get -y install --no-install-recommends build-essential \
&& apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common \
&& curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - \
&& add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" \
&& apt-get update \
&& apt-cache policy docker-ce \
&& apt-get install -y docker-ce \
&& apt-get clean

RUN mkdir $GRAALVM_BASE && mkdir $GRAALVM_HOME
COPY ./graalvm-ce-java11-20.0.0 $GRAALVM_HOME/

RUN chmod -R g+rw $GRAALVM_HOME && gu install native-image

Use Physical Node or VM Node for Jenkins Slave

So what works then?

At the end I settled with running the Jenkins slave on physical node. I installed and configured the necessary GraalVM, Podman and Docker in just within an hour. I managed to build Quarkus native image for Docker and Podman in a blink of click, and pushed them to their respective container registry in no time.

Summary

The idea of running Jenkins slave nodes as container is not new and many people are doing this now. The Microservices and Serverless works as they claim so far. However the supporting technology such as the CI/CD tools are becoming heavier and thick.

The current traditional CI/CD tools such as Jenkins are not designed for Microservices and Serverless environments. They are not able to scale with the rapid changes and massive scale of the Microservices and Serverless environments. They take out too much of infrastructure resources if they are to be deployed onto Kubernestes environment.

In aware of this, the open source community has nature the Tekton as the next CI/CD pipelines for Kubernestes environment, designed for Microservices and Serverless.

Stay tuned for more articles coming for this.

Note: Building these Jenkins slave containers resulted in massive container image size, which is roughly around 5GB, not less. For sure we can optimise the Dockerfiles but I believe it will not reduced much.