kubernetes storage

K8s on RPi 4 Pt. 4 – Configuring Kubernetes Storage

Posted on by CK Gan

Table of Contents

Overview

In a perfect world, we would prefer to implement microservices to be stateless. This stateless approach eliminates a lot of problems towards maintaining the storage for the applications data. However, we are not in a perfect world. At most time, applications need to deal with data and keep them on storage for long term references. We need persistent storage solution so that the applications data can survive containers (or systems) restart or crash. Kubernetes provides this type of storage support in the concept of volumes. We are going to look at how can we configured storage for Kubernetes on Raspberry Pi 4.

Note: When application is mentioned in this post, it can be assumed it is also referring to microservices or container.

This post is part of the series to cover the details of my initiative to deploy production alike Kubernetes on Raspberry Pi 4. If you miss the previous posts, you can refers them at the following links.

Kubernetes Storage

The story of Kubernetes storage is really long. I will try to make it short here so that you will be able to follow the content covered in this post.

Generally, you can keep your data at local disk in the container. However your data will be lost when the container restarts or crashes. This type of storage is described as ephemeral. We need a better managed storage to keep the application data safe and sound.

Docker provides storage through the concept of volumes and it is somewhat less managed. Kubernetes volumes provides a better management compared to Docker volumes, in many ways.

Kubernetes volumes can be defined as ephemeral or persistent. Persistent volumes are maintained and managed beyond the Pod[1] lifetime. Kubernetes supports many type of volumes which the list can be found here. For this post, I am covering how to configure the NFS as the Kubernetes volume.

In order to define and use the respective storage for Kubernetes, the supported storage providers need to provide the definition and details of how to define, consume and manage the storage via Kubernetes resource definition called StorageClass.

Once the respective storage class is being defined and configured in Kubernetes, you can start defining additional Kubernetes resources (Persistent Volume and Persistent Volume Claim). These additional resources define how can a Pod be able to consume the volumes.

Persistent volume (PV) defines details such as the storage class, the volume size, the volume type, the volume path and etc. You need to manually create the necessary storage volumes at the storage server and configure the volume detail using the PV at the Kubernetes server. After that, the Pod can starts consuming this storage volume via the details defined in the Kubernetes resource persistent volume claim (PVC).

Up to now we are talking about storage administrators create the volumes manually and configure the volume definition using the PV. You can also use the dynamic volume provisioning to create the volumes and PVs automatically whenever Pods require them. For this approach, your applications deployment only need to define the PVC in order to consume a storage volume. The only requirement is the storage provider together with the storage class provided must support the dynamic volume provisioning.

In short, the above cover the quick overview of Kubernetes volumes concept. I urge you to proceed to the Kubernetes storage section to learn more.

[1] Pod is a term used to define the smallest manageable resource unit in Kubernetes. Technically pod and container are not the same although people tend to refer pod as container. In animal world, pod refers to a group of whale (Docker container logo relates to this). In Kubernetes world, pod refers to a group of containers. It implies that you can run multiple containers inside a pod. For more information on Pod, please refer here.

Enabling NFS Storage for Kubernetes on RPI

There are many considerations that’s need attentions when using NFS for Kubernetes production such as the file system performance. However this will not be the focus discussion of this post.

I will be using my Kubernetes for non-critical and predicted data volumes. Thus, I am very confident NFS storage works fine for my use cases.

In this section, I will continue to explain the steps that I have taken to configure the NFS storage for the Kubernetes running on my Raspberry Pi 4.

According to Kubernetes documentation, there are 2 open source NFS provisioners than we can use to create the NFS storage class definition.

I have the experience of using NFS subdir external provisioner in the past. So I will proceed to use this for my Kubernetes cluster.

Creating NFS Storage

You need to create a dedicated NFS volume for this exercise. I am assuming you have this taken care at your storage server. Make sure the volume is configured with all permission allowed by running the following command. Please replace the volume path with the correct value.

chmod -R 0777 /volume/mykubepv

Configuring NFS Storage Class

You can proceed by cloning the project onto your local disk.

git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git

If you decide to deploy into a namespace other than the “default”, please run the following command to change the namespace in the “rbac.yaml” and “deployment.yaml” files, or you can just open the files and manually change it.

# Set the current namespace into NS variable
NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')

# Set NAMESPACE to default if NS is empty.
NAMESPACE=${NS:-default}

# Search and replace the namespace in the files
sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml ./deploy/deployment.yaml

If your cluster has RBAC enabled, you must enable the RBAC for your provisioner. 

kubectl create -f deploy/rbac.yaml

Modify the “nfs-subdir-external-provisioner/deploy/deployment.yaml” file. I have placed some comments in the following yaml sample for self explanation. It should not be hard to follow.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed. We leave this as "default".
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      # matching the ServiceAccount from the rbac.yaml
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            # This is the name that need to match the "provisioner" parameter in the StorageClass definition in the class.yaml
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            # Configure your NFS server hostname or IP
            - name: NFS_SERVER
              value: xxx.xxx.xxx.xxx
            # Configure the NFS volume that we can configure at the earlier step.
            - name: NFS_PATH
              value: /volume/mykubepv
      volumes:
        - name: nfs-client-root
          nfs:
            # NFS server hostname or IP. Same as above.
            server: xxx.xxx.xxx.xxx
            # the NFS volume path. Same as above.
            path: /volume/mykubepv

You can now run the following command to configure the NFS provisioner. Before you do this, please make sure your Kubernetes nodes can connect to your NFS server. You can do this test by trying to mount the NFS volume at one of your Kubernetes node.

kubectl apply -f deploy/deployment.yaml

Open the “nfs-subdir-external-provisioner/deploy/class.yaml” file and change the content according to your environment.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # Name you StorageClass to something you can remember and understand
  name: managed-nfs-storage
# Use this default provisioner name or choose another, must match deployment's env 'PROVISIONER_NAME'
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  # Default is "false". Set to "true" if you would like to keep and archive the volume content when PVC is deleted.
  archiveOnDelete: "false"
  # the path pattern to use when the volume if created. This pattern works best for me.
  pathPattern: "${.PVC.namespace}-${.PVC.name}"

Explicitly change the current namespace to Kubernetes “default” namespace. 

kubectl config set-context --current --namespace=default

At the “nfs-subdir-external-provisioner” root directory, run the following command to apply the StorageClass definition. 

kubectl apply -f deploy/class.yaml

Test The NFS Volume

You can now test your NFS provisioner.

You need to modify the storageClassName in the “deploy/test-claim.yaml” to the new StorageClass (managed-nfs-storage) we have just configured.

storageClassName: managed-nfs-storage

Run the following command to create a PVC and Pod to test the NFS storage class.

kubectl create -f deploy/test-claim.yaml -f deploy/test-pod.yaml

Proceed to your NFS volume at the storage server to verify that a file named “SUCCESS” is created. Notice the directory structure is created according to the “pathPattern” defined in the StorageClass definition.

You can now proceed to delete the Pod and the PVC. Since we defined ‘archiveOnDelete: “false”‘ in the StorageClass definition, the directory and the content at the storage server are deleted once the PVC is deleted.

kubectl delete -f deploy/test-pod.yaml -f deploy/test-claim.yaml

Configure Default Storage Class

If you open the “test-claim.yaml” file, you will notice there is a definition of storageClassName in the PersistentVolumeClaim. This storageClassName is used to define which storage class that you would like to create and consume the storage volume from. In this example is the new storage class (managed-nfs-storage) that we have just created earlier. 

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: managed-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

There will be situations that you may need multiple StorageClass definitions for different type of backend storages that you have. For example, you may need an object storage for RWX (read-write many) access mode. This storageClassName definition provides the flexibility to allow you to use different storages for different Pods.

If you are certain there is always one storage available, or most of time the specific storage should be used as the default StorageClass for all your PODs(unless otherwise specified explicitly), you can configure this specific StorageClass as the default for the Kubernetes cluster.

You can configure a default StorageClass by running the following command:

kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

With the default storage class configured, when the storageClassName is not defined in the PVC, it is assumed the default storage class will be used.

Summary

Now we have configured the storage volume for Kubernetes we can proceed to deploy applications that need storage.

Bear in mind, although I configured the storage volume on Raspberry Pi 4 but the same approaches works for typical Kubernetes environment.

Next, I will be covering configuring Nginx ingress controller for Kubernetes. Stay tuned.

Related posts:

  1. K8s on RPi 4 Pt. 2 – Proven Steps to PXE Boot RPi 4 with Ubuntu
  2. K8s on RPi 4 Pt. 3 – Installing Kubernetes on RPi
  3. K8s on RPi 4 Pt. 5 – Exposing Applications to the World
  4. Monitor and Analyze Nginx Ingress Controller Logs on Kubernetes using ElasticSearch and Kibana

Leave a Reply

Your email address will not be published. Required fields are marked *