How to Provision Persistent Storage on GKE With Quobyte

Quobyte is a true software-defined storage system that can be deployed anywhere, on any server and the public clouds. It can also be installed on Kubernetes, and it runs just as another application on your k8s cluster. Its distributed file system provides shared read-write-many (RWX) persistent volumes with scalable high performance storage for any workload.

Quobyte runs on Google Kubernetes Engine (GKE), as well as other clouds and also your own hardware. This allows you to have the same storage system anywhere, making it easy to migrate workloads, meaning you get a true multi cloud and hybrid cloud system.

In addition, Quobyte offers you a scale-out shared file system with high performance for workloads like AI/ML, VFX, 3D rendering, image analysis or traditional HPC.

In this article, we will walk you through the installation of Quobyte and the Quobyte CSI plugin on a GKE cluster. And within just a few minutes, you will be able to provision persistent volumes to your k8s clusters with Quobyte.

Quobyte Installation on GKE Cluster

We will install Quobyte using a Helm chart, but first, you need to add the Quobyte repo to your environment.

helm repo add quobyte https://quobyte.github.io/quobyte-k8s-resources/helm-charts

Now, let’s proceed with the installation

helm install my-storage-cluster quobyte/quobyte-cluster -f myvalues.yaml — wait=true

Note that this starts the core Quobyte services in containers in your k8s cluster.

On bare metal servers, Quobyte uses the disks (HDD, NVMes) to store data, on VMs block storage (like persistent disks). In a bare metal k8s cluster, Quobyte services can use the disks in the servers. On VM based clusters — like GKE — we need block storage and Quobyte requests it through persistent volume requests.

Later, Quobyte provides persistent volumes itself — but these are scale-out share file system volumes vs. the block storage persistent volumes provided by the cloud platform.

Let’s take a look at the persistent volume claims that were created by the services:

kubectl get pvc -A

You will be able to see the persistent volume claims for the data, metadata, and registry services. Please note that you can adjust the type and size of storage using the values.yaml file which can be found in the quobyte-cluster folder.

Next, we will set up the Quobyte webconsole account. Let’s retrieve the webconsole address:

kubectl get svc/quobyte-webui

Copy the external IP address, paste it into a new browser window and add port 8080 to it. The Quobyte webconsole should load with a welcome message. After you click “START” you will need to agree to the Quobyte Terms and Conditions, and set up an admin account.

Please note that you will need to add your credentials base64 encoded to the 02_secret_quobyte.yaml file under quobyte-k8s-resources>usage-examples>01_getting_started.

You can also configure which hosts should be trusted to run Quobyte services. Let’s retrieve the IP address:

kubectl get pods -o wide

Copy the IP address and change the last number to zero. For the subnet you can choose 16, which will take half of the network.

You can skip the SMTP configuration, and the license step. However, if you have a license, you can add it here. If you would like to get a free license visit the Quobyte website.

You now have a Quobyte cluster up and running.

The next step to access the Quobyte cluster from Kubernetes is to deploy the quobyte-client and quobyte-csi. The Quobyte client runs on all worker nodes on your k8s cluster; it automatically mounts new volumes and makes them available to the host via a bi-directional volume mapping. On the other hand, the Quobyte CSI is responsible for fulfilling all the PVC that have a StorageClass configured to use Quobyte. For PVCs that have a size, the Quobyte CSI plugin also creates a Quobyte volume quota.

helm install my-storage-client quobyte/quobyte-client
helm install my-storage-csi quobyte/quobyte-csi

Provisioning Persistent Volumes With Quobyte

Next, we will do a quick example on how to provision a persistent volume, and mount it on a container deployed as a single pod.

For the next part, you will need to access quobyte-k8s-resources repo and cd into the usage_examples>01_getting_started directory:

cd quobyte-k8s-resources/usage-examples/01_getting_started

Add your credentials to the 01_secret_quobyte.yaml. Please remember that your credentials need to be base64 encoded.

Next, apply the examples declarations:

kubectl apply -f 01_namespace-quobyte.yaml -f 02_secret_quobyte.yaml -f 03_storageClass.yaml -f 04-testpvc.yaml -f 05_testpod.yaml

These declarations will create the following:

• Quobyte namespace
• The credentials to authenticate against Quobyte
• A Quobyte default storage class to enable access to the Quobyte cluster
• A persistent volume claim that provisions volumes
• A example pod deployment of a container that uses that volume

Now, we can check if the PVC was successfully created

kubectl get pvc quobyte-default-pvc

We can get more details of the PVC:

kubectl describe pvc quobyte-default-pvc

Next, we can check if the persistent volume was successfully mounted inside a container:

kubectl exec -it consumer-pod — mount | grep quobyte

Lastly, we can check if the Quobyte mount is readable and writable from within the container:

kubectl exec -it consumer-pod — /bin/bash

cd /usr/share/nginx/html/

Now, let’s write “Hello World!” to an html file:

echo “Hello World!” > index.html

And lastly, read the content of the index.html file:

cat index.html

As you can see you now provide persistent storage with Quobyte to any application on your GKE cluster.

Video: How to Provision Persistent Storage to a GKE Cluster with Quobyte

Originally posted on Quobyte’s blog on Feb 9, 2023.