Table of Contents
Kubernetes StatefulSets are used to deploy stateful packages inside of your cluster. Each and every Pod within the StatefulSet can get entry to native power volumes that persist with it even after it’s rescheduled. This permits Pods to take care of particular person state that’s become independent from their neighbors within the set.
Sadly those volumes include a large limitation: Kubernetes doesn’t supply a approach to resize them from the StatefulSet object. The spec.assets.requests.garage belongings of the StatefulSet’s volumeClaimTemplates box is immutable, fighting you from making use of any capability will increase you require. This text will display you the way to workaround the issue.
Making a StatefulSet
Reproduction this YAML and put it aside to ss.yaml:
apiVersion: v1
sort: Provider
metadata:
title: nginx
labels:
app: nginx
spec:
selector:
app: nginx
ports:
- title: nginx
port: 80
clusterIP: None
---
apiVersion: apps/v1
sort: StatefulSet
metadata:
title: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 3
serviceName: nginx
template:
metadata:
labels:
app: nginx
spec:
boxes:
- title: nginx
symbol: nginx:newest
ports:
- title: internet
containerPort: 80
volumeMounts:
- title: information
mountPath: /usr/proportion/nginx/html
volumeClaimTemplates:
- metadata:
title: information
spec:
accessModes: ["ReadWriteOnce"]
assets:
requests:
garage: 1Gi
Observe the YAML in your cluster with Kubectl:
$ kubectl practice -f ss.yaml carrier/nginx created statefulset.apps/nginx created
You’ll desire a garage elegance and provisioner to your cluster to run this case. It creates a StatefulSet that runs 3 replicas of an NGINX internet server.
Whilst this isn’t consultant of when StatefulSets must be used, it’s good enough as a demo of the amount issues you’ll face. A quantity declare with 1 Gi of garage is fixed to NGINX’s information listing. Your internet content material may outgrow this rather small allowance as your carrier scales. Then again seeking to adjust the volumeClaimTemplates.spec.assets.requests.garage box to 10Gi will document the next error whilst you run kubectl practice:
$ kubectl practice -f ss.yaml carrier/nginx unchanged The StatefulSet "nginx" is invalid: spec: Forbidden: updates to statefulset spec for fields rather than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
This happens as a result of virtually the entire fields of a StatefulSet’s manifest are immutable after advent.
Manually Resizing StatefulSet Volumes
You’ll bypass the restriction through manually resizing the power quantity declare (PVC). You’ll then want to recreate the StatefulSet to liberate and rebind the amount out of your Pods. This may occasionally cause the true quantity resize match.
First use Kubectl to search out the PVCs related along with your StatefulSet:
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES data-nginx-0 Sure pvc-ccb2c835-e2d3-4632-b8ba-4c8c142795e4 1Gi RWO data-nginx-1 Sure pvc-1b0b27fe-3874-4ed5-91be-d8e552e515f2 1Gi RWO data-nginx-2 Sure pvc-4b7790c2-3ae6-4e04-afee-a2e1bae4323b 1Gi RWO
There are 3 PVCs as a result of there are 3 replicas within the StatefulSet. Each and every Pod will get its personal particular person quantity.
Now use kubectl edit to regulate the capability of each and every quantity:
$ kubectl edit pvc data-nginx-0
The PVC’s YAML manifest will seem to your editor. In finding the spec.assets.requests.garage box and alter it in your new desired capability:
# ...
spec:
assets:
requests:
garage: 10Gi
# ...
Save and shut the document. Kubectl must document that the trade has been carried out in your cluster.
persistentvolumeclaim/data-nginx-0 edited
Now repeat those steps for the StatefulSet’s final PVCs. Record your cluster’s power volumes must then display the brand new dimension towards each and every one:
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO Delete Sure default/data-nginx-2 pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO Delete Sure default/data-nginx-0 pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO Delete Sure default/data-nginx-1
The claims will take care of the previous dimension for now:
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES data-nginx-0 Sure pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO data-nginx-1 Sure pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO data-nginx-2 Sure pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO
It’s because the amount can’t be resized whilst Pods are nonetheless the usage of it.
Recreating the StatefulSet
Whole the resize through liberating the amount declare from the StatefulSet that’s keeping it. Delete the StatefulSet however use the orphan cascading mechanism so its Pods stay to your cluster. This may occasionally lend a hand decrease downtime.
$ kubectl delete statefulset --cascade=orphan nginx statefulset.apps "nginx" deleted
Subsequent edit your authentic YAML document to incorporate the brand new quantity dimension within the spec.assets.requests.garage document. Then use kubectl practice to recreate the StatefulSet to your cluster:
$ kubectl practice -f ss.yaml carrier/nginx unchanged statefulset.apps/nginx created
The brand new StatefulSet will suppose possession of the up to now orphaned Pods as a result of they’ll already meet its necessities. The volumes would possibly get resized at this level however normally you’ll must manually begin a rollout that restarts your Pods:
$ kubectl rollout restart statefulset nginx
The rollout proceeds sequentially, concentrated on one Pod at a time. This guarantees your carrier stays available all over.
Now your PVCs must display the brand new dimension:
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES data-nginx-0 Sure pvc-33af452d-feff-429d-80cd-a45232e700c1 10Gi RWO data-nginx-1 Sure pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5 10Gi RWO data-nginx-2 Sure pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7 10Gi RWO
Check out connecting to considered one of your Pods to test the higher capability is visual from inside:
$ kubectl exec -it nginx-0 bash root@nginx-0:/# df -h /usr/proportion/nginx/html Filesystem Measurement Used Avail Use% Fastened on /dev/disk/by-id/scsi-0DO_Volume_pvc-33af452d-feff-429d-80cd-a45232e700c1 9.9G 4.5M 9.4G 1% /usr/proportion/nginx/html
The Pod’s reporting the predicted 10 Gi of garage.
Abstract
Kubernetes StatefulSets assist you to run stateful packages in Kubernetes with power garage volumes which are scoped to particular person Pods. Then again the versatility this allows ends when you want to resize considered one of your volumes. It is a lacking function which lately calls for a number of handbook steps to be finished in collection.
The Kubernetes maintainers are acutely aware of the problem. There’s an open function request to expand an answer which must in the end assist you to begin quantity resizes through modifying a StatefulSet’s manifest. This will likely be a lot faster and more secure than the present state of affairs.
One ultimate caveat is that quantity resizes are depending on a garage motive force that lets in dynamic growth. This selection handiest was typically to be had in Kubernetes v1.24 and now not all drivers, Kubernetes distributions, and cloud platforms will fortify it. You’ll test whether or not yours does through working kubectl get sc and on the lookout for true within the ALLOWVOLUMEXPANSION column of the garage motive force you’re the usage of along with your StatefulSets.
