Table of Contents
- Overview
- What was Happening?
- Pre-Requisites Before Performing Disaster Recovery
- Steps for Kubernetes Disaster Recovery
- Summary
Overview
As you may have already known I am running Kubernetes on RPI4 cluster. At time, it could be challenging to maintain the RPI4s. This including performing disaster recovery for this tiny Kubernetes production environment.
This Kubernetes environment has been corrupted many times in the past, mainly due to the the ETCD database corruptions.
There are sometime applications failures in this Kubernetes cluster but this can be handled with ease as long as there are proper backup of the database.
The ETCD database corruption is the most headache situation even there are snapshots. This can happens very often due to the storage problem (low cost storage) or accidents caused by OS upgrades.
At other times, these RPI4 nodes work pretty well if I leave them running with proper air cooling.
Hate and love of these fragile RPI4 nodes.
The ETCD database can be easily backup and restored by using the etcd commands such as etcdutl or etcdctl. However there are no proper documented procedure on how to restore the control planes together with ETCD. There are many posts out there but none of them solve my problem.
In the past, I spent amount of time to recreate the entire Kubernetes cluster because I was not able to properly bring up the Kubernetes cluster due to unforeseen problems, even after restoring the ETCD database.
Not until now. This is the first time I managed to recover my Kubernetes cluster without rebuild it.
What was Happening?
I have a Kubernetes cluster with 3 control planes and 5 worker nodes. It is hosting my WordPress personal blog site and my personal file server Nextcloud. Occasionally, I also tests my application modernization demo on it.
Here is the list of servers I have running on RPI4.
NAME STATUS ROLES AGE VERSION
master1.internal Ready control-plane 144d v1.24.2
master2.internal Ready control-plane 144d v1.24.2
master3.internal Ready control-plane 144d v1.24.2
worker1.internal Ready <none> 144d v1.24.2
worker2.internal Ready <none> 144d v1.24.2
worker3.internal Ready <none> 144d v1.24.2
worker4.internal Ready <none> 144d v1.24.2
worker5.internal Ready <none> 144d v1.24.2One day, as usual, I decided to upgrade my Ubuntu OS to the lates patches, with overly confident. I did not realise that my containerd services are not configured to wait for storage to be ready, which I thought I had. All the control planes lost their mind after the OS rebooted. I had attempted to restore the containerd iSCSI storage and restart the services, which works previously, but not this time, unfortunately.
Luckily the Kubernetes worker nodes are running fine and continue to serve me at that moment.
So that’s start the adventure to attempt to restore the ETCD database again.
Pre-Requisites Before Performing Disaster Recovery
There are 2 sets of files that you need to backup before any disaster happens to your Kubernetes cluster. These are:
- ETCD database
- Kubernetes PKI certificates
ETCD Database Backup
You can backup the ETCD database using the etcdctl tool. This provide a simple approach for you to perform your ETCD database backup provided that you have the required technical skill.
You can also refer to my earlier post on Automate Kubernetes etcd Data Backup for how to do this better.
You may also choose to do this automatically by deploying the etcdbk container to do this for you. I am using this container to perform the backup daily.
You only need one copy of this snapshot from any of your control plane.
Note: I think we fully aware of commercial available products that can do this but why should we spend the money for a RPI4 environment. Plus where is the fun without doing it the hard way!
Kubernetes PKI Certs Backup
In order to seamlessly restore your corrupted ETCD database (and your Kubernetes control planes or the cluster), you need to have the latest copy of the PKI certs located in the default directory at /etc/kubernetes/pki
Please make sure you perform the backup at least monthly, as you will not know when your cluster will be having problem or when the certificate rotation will happens (according to Kubernetes document, this happens yearly).
Same as the ETCD backup, you only need one set of this PKI certificates from any of your control plane node.
Do I need to mention you need to keep multiple copies for these backups!
Steps for Kubernetes Disaster Recovery
In the past, I had been googling around for the proven steps to restore the Kubernetes cluster if the control planes are totally gone but no luck.
The following procedure works for me for the event where my control planes are gone while the worker nodes are still working fine. I have no doubt this works for a completely lost cluster (i.e. both control planes and worker nodes fail).
You do not need to perform these steps if you still have one of the control plane running. In this case, in short, you can just clean the problematic control planes nodes and join the control planes nodes back to the healthy cluster.
Cleaning All The Server Nodes
You should proceed run the following command to reset the control planes. Perform this steps at all your control planes nodes.
$ sudo kubeadm resetOptionally clean the control planes nodes with the following command:
sudo systemctl stop containerd && \
sudo systemctl stop kubelet && \
sudo rm -rf /var/lib/containerd/* && \
sudo systemctl start containerd && \
sudo systemctl start kubelet && \
sudo rm -rf /etc/kubernetes && \
sudo rm -rf /etc/cni/net.dSometime the Kubernetes related containers hung due to many attempts and cannot be cleaned by the kubeadm reset command. You may encounters weird errors if you do not stop the phantom containers and proceed to bootstrap your control plane.
You can do this by using the crictl stop command. The best is to restart the server before proceeding to the next step.
In this scenario I do not need to perform the Kubernetes pre-requisites installation because the kubeadm reset and the commands above should be able to clean your existing nodes. In the event of hardware lost without backup, you will need to do zero start of Kubenetes nodes preparation. You can refer to Installing Kubernetes on RPI for more detail on how to do this.
Note: I am using ContainerD for the container runtime. Refers to my earlier series of posts on the running Kubernetes on RPI4 for more details.
Pulling The Right Version of Kubernetes Container Images
I would recommend to manually pull the Kubernetes container images before performing the control planes bootstrapping. Explicitly specifying --kubernetes-version at the kubeadm init command will not attempt to pull the correct minor version but will result in pulling the latest minor version. For example, my current Kubernetes version is 1.24.2 but the kubeadm init will pull 1.24.8 at the time of this writing.
$ sudo kubeadm config images pull --kubernetes-version=1.24.2You need to make sure this is the exact Kubernetes version number that is currently running.
You should perform the manual container images pulling at all your control planes nodes.
Restoring Kubernetes PKI Certificates
Restore your PKI certs by copying the PKI certs from your backup to the default location at /etc/kubernetes.
sudo mkdir -p /etc/kubernetes/pki && \
sudo cp -r $HOME/backup/pki/* /etc/kubernetes/pki/You only need to perform this on the first node that you are going to bootstrap the control plane.
Restoring ETCD Database
You can now proceed to download the correct version of ETCD into the the first control plane node that we are working on now.
$ ETCD_VER=v3.5.3
# choose either URL
$ GOOGLE_URL=https://storage.googleapis.com/etcd
$ GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
$ DOWNLOAD_URL=${GOOGLE_URL}
$ mkdir -p /tmp/etcd-download-test
$ curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-arm64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-arm64.tar.gz
$ tar xzvf /tmp/etcd-${ETCD_VER}-linux-arm64.tar.gz -C /tmp/etcd-download-test --strip-components=1
$ rm -f /tmp/etcd-${ETCD_VER}-linux-arm64.tar.gz
$ /tmp/etcd-download-test/etcd --version
$ /tmp/etcd-download-test/etcdctl version
$ /tmp/etcd-download-test/etcdutl version
$ mkdir ~/etcd && mv /tmp/etcd-download-test/ ~/etcdAt the same node, proceed to restore the ETCD database from the snapshot. Copy your ETCD snapshot to the ~/etcd directory created in the previous step. Change your directory to ~/etcd.
sudo ETCDCTL_API=3 $HOME/etcd/etcdctl snapshot restore etcd-master2.internal-2022-11-22-08-00-1669075228.etcdbk \
--data-dir /var/lib/etcd/ \
--name master1.internal \
--initial-advertise-peer-urls https://10.168.0.5:2380 \
--initial-cluster master1.internal=https://10.168.0.5:2380 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.keyThe following shows the outcomes of the restore command. Please note the command output states that etcdctl is deprecated and it is advised to use etcdutl command instead. I choose to use etcdctl command is because I want to be able to specify the parameters for --cacert, --cert and --key.
Deprecated: Use `etcdutl snapshot restore` instead.
...
2022-11-23T14:07:25+08:00 info membership/store.go:141 Trimming membership information from the backend...
2022-11-23T14:07:25+08:00 info membership/cluster.go:421 added member {"cluster-id": "ce81131cb1791e0a", "local-member-id": "0", "added-peer-id": "6745abcbb01d9c53", "added-peer-peer-urls": ["https://10.168.0.5:2380"]}
2022-11-23T14:07:25+08:00 info snapshot/v3_snapshot.go:269 restored snapshot {"path": "etcd-master2.internal-2022-11-22-08-00-1669075228.etcdbk", "wal-dir": "/var/lib/etcd/member/wal", "data-dir": "/var/lib/etcd/", "snap-dir": "/var/lib/etcd/member/snap"}Take note that we only need to do this once at the first control plane node. Thus, we only enter one ectd node at the parameter --initial-cluster. Once this control plane is ready later, we can proceed to join the remaining nodes into the cluster.
Note: You can also try to restore this snapshot using standalone ETCD container, which is not covered in this post.
Bootstrapping The First Control Plane
Now we can proceed to bootstrap the control plane. Take note at the following command we need to ignore the preflight error for /var/lib/etcd since we have created this directory when we restore the etcd database. This is done via the parameter --ignore-preflight-errors=DirAvailable--var-lib-etcd.
sudo kubeadm init \
--control-plane-endpoint=lb.kube.internal \
--apiserver-advertise-address=10.168.0.5 \
--kubernetes-version=1.24.2 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=DirAvailable--var-lib-etcdPlease make sure the other configurations are the same as when you first created your Kubernetes cluster.
The following show part of the command output. Please make sure that the messages indicate that it is using the existing PKI certs. Otherwise, you need to find out why it is not.
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[WARNING DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig fileJoin The Other Control Planes
You can now sit back and wait for the control plane become ready. This may take a while especially for RPI4 node.
While waiting you may want to copy the admin.conf to your current user home directory. Skip this if the existing config is still there.
mkdir -p $HOME/.kube && \
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && \
sudo chown $(id -u):$(id -g) $HOME/.kube/configYou can perform kubectl get nodes now and you should see all the nodes appear again. The other control planes are in NotReady stage. We will resolve this in the next steps.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1.internal Ready control-plane 144d v1.24.2
master2.internal NotReady control-plane 144d v1.24.2
master3.internal NotReady control-plane 144d v1.24.2
worker1.internal Ready <none> 144d v1.24.2
worker2.internal Ready <none> 144d v1.24.2
worker3.internal Ready <none> 144d v1.24.2
worker4.internal Ready <none> 144d v1.24.2
worker5.internal Ready <none> 144d v1.24.2Once the current control plane is in Ready stage, we proceed to generate the join command at the control plane node (master1.internal).
$ sudo kubeadm init phase upload-certs --upload-certs
I1123 14:17:16.779607 4086 version.go:255] remote version is much newer: v1.25.4; falling back to: stable-1.24
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
606842b5740239a14b9eaf0ba20e899c6af43ea4b890464c8a73c9989b36c41bUsing the above generated certificate key, generate the join command as per the following.
$ sudo kubeadm token create --certificate-key 606842b5740239a14b9eaf0ba20e899c6af43ea4b890464c8a73c9989b36c41b --print-join-command
sudo kubeadm join lb.internal:6443 --token zp386n.9j5hh1m49mzg3ytc --discovery-token-ca-cert-hash sha256:a6a1e7dc1864081d634105e2247f0e255028b8ff07d2cc07469f217dbe618864 --control-plane --certificate-key 9d815cb4512cd792961d6c1acaddaae3b2c4f42e21c8a86fdf87ca55089f0fabYou can now use the generated join command at each control plane node.
Summary
We have gone through how we can use the ETCD database snapshot and PKI certificates to restore a completely failed control planes nodes. It should also works on a completely lost cluster (i.e. both control planes and worker nodes are not working).
This disaster recovery steps might not suit all possible scenarios that you may encountered but I hope these proven steps, which I had performed on my RPI4 may provide you a great insights on how this can be done if you have a complete control plane nodes failure.
Let me know if this helps you. Any feedback is most welcome.