How to create and debug a Kuber­netes State­ful­Set

Each pod in a Kuber­netes State­ful­Set has a unique and stable hostname as well as a complete DNS identity. This ID is retained across restarts and scaling.

What are Kuber­netes State­ful­Sets?

Kuber­netes State­ful­Sets are special entities to manage ap­plic­a­tions that have special re­quire­ments for per­sist­ent data and fixed iden­tit­ies. These sets ensure that pods with replicas are started in a specific order. They assign each Kuber­netes pod a unique ID and access to per­sist­ent storage. Such functions are useful for databases that rely on stable re­la­tion­ships between instances and that need to store per­sist­ent data. Therefore, State­ful­Sets represent an effective solution for the or­ches­tra­tion and operation of complex ap­plic­a­tions in Kuber­netes.

How to create a Kuber­netes State­ful­Set

We start by defining a YAML con­fig­ur­a­tion file in which we specify the desired prop­er­ties of the State­ful­Set and create Kuber­netes pods. After con­fig­ur­a­tion, the State­ful­Set con­tinu­ously monitors the cluster status and whether the pre­defined number of pods is always running and available. In the event of a pod failure or removal from the node, the State­ful­Set auto­mat­ic­ally re­cog­nises the situation. It initiates the de­ploy­ment of a new pod with the same unique ID. This new pod is connected to the existing per­sist­ent storage and receives the identical con­fig­ur­a­tion as the original pod. This includes resource requests and limits.

This precise handling of pods and their identity is critical so that clients pre­vi­ously served by the un­avail­able pod can be re­dir­ec­ted to the new pod without in­ter­rup­tion. Access to per­sist­ent storage ensures that op­er­a­tions continue to function smoothly.

Here’s a complete YAML file that il­lus­trates the steps for creating a Kuber­netes State­ful­Set for Nginx:

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: nginx-statefulset
spec:
    serviceName: nginx
    replicas: 3
    selector:
        matchLabels:
            app: nginx
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
            - name: nginx
                image: nginx:latest
                volumeMounts:
                - name: nginx-config
                    mountPath: /etc/nginx/conf.d
                - name: nginx-html
                    mountPath: /usr/share/nginx/html
    volumeClaimTemplates:
    - metadata:
            name: nginx-html
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi
    volumeClaimTemplates:
    - metadata:
            name: nginx-config
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi

This YAML document defines a Kuber­netes State­ful­Set for Nginx with three replicas. It uses a service object called nginx and labels to correctly identify the pods. The latter use the latest Nginx image and have two volumes for con­fig­ur­a­tion and HTML files. The Volume­ClaimTem­plates secure per­sist­ent storage for these volumes with a size of 1 gigabyte and allow Read­WriteOnce access.

Debugging State­ful­Sets

Debugging State­ful­Sets in Kuber­netes requires specific steps to verify that the pods are ini­tial­ised correctly and that you can identify and fix errors if necessary.

Step 1: List the pods

Before you start debugging State­ful­Sets, you should check the status of the pods.

Open the command line and use the following command to list all pods in the desired State­ful­Set:

kubectl get pods -l app=statefulset-label

Output:

NAME                         READY      STATUS    RESTARTS   AGE
nginx-statefulset-0           1/1      Running       0       2m
nginx-statefulset-1           1/1      Running       0       1m
nginx-statefulset-2           1/1      Running       0       1m

• NAME: Each pod is given a unique name based on the naming scheme of the State­ful­Set and a con­sec­ut­ive number.
• READY: Indicates how many of the desired con­tain­ers are ready in the pod. In the example, each pod has one container and 1/1 means that the container is ready for use.
• STATUS: This indicates the current status of the pod.
• RESTARTS: Shows how often the container has been restarted in the pod. A value of 0 means that there have been no restarts so far.
• AGE: Indicates how long the pod has been running.

These are the status messages that indicate errors:

• Failed: One or more con­tain­ers in the pod caused an error that resulted in the failure of the pod. This can have various reasons, such as missing de­pend­en­cies or con­fig­ur­a­tion problems.
• Unknown: The state of the pod could not be de­term­ined. It could indicate a problem with the com­mu­nic­a­tion between the Kuber­netes cluster and the pod. These include network problems, missing au­thor­isa­tions or other factors.

In both cases, it’s important to use accurate dia­gnost­ic tools such as checking pod logs or the kubectl describe pod command to get more details and determine the causes of the errors.

Step 2: Debug in­di­vidu­al pods

Adding an­nota­tions to a pod can be a useful debugging tool to influence the ini­tial­isa­tion process or trigger special actions.

First, you need to identify the name of the pod you want to debug.

kubectl get pods

Now enter the following command in the terminal to define the an­nota­tion for the selected pod:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="false" --overwrite

Replace [pod-name] with the actual name of the pod. This an­nota­tion sets the ini­tial­isa­tion status to false, which means that the pod is marked as un­ini­tial­ised.

Monitor the pod to see how the an­nota­tion affects its behaviour. In par­tic­u­lar, you can check the pod events and logs:

kubectl describe pod [pod-name]
kubectl logs [pod-name]

Look for events or log output that could cause problems during the ini­tial­isa­tion process. When debugging is complete and you want to restore the normal ini­tial­isa­tion process, set the an­nota­tion back to true.

Step 3: Step-by-step ini­tial­isa­tion

If debugging the pod using the tech­niques described above was not suc­cess­ful, this could indicate race con­di­tions during the bootstrap process of the State­ful­Set. To overcome this issue, you can specify initialized="false" in the manifest of the Kuber­netes State­ful­Set and then create it with this an­nota­tion in the cluster.

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: [statefulset-name]
spec:
    template:
        metadata:
            annotations:
                pod.alpha.kubernetes.io/initialized: "false"
        ...

Apply the updated manifest to your Kuber­netes cluster:

kubectl apply -f statefulset.yaml

Inspect the pods and identify any sources of error. Carry out the necessary debugging measures based on the observed events and logs. If necessary, delete pods with kubectl delete statefulsets or kubectl delete service.

After you have completed debugging, you can remove the ini­tial­ised an­nota­tion and update the Kuber­netes State­ful­Set in the cluster:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="true" --overwrite

The command sets the ini­tial­ised an­nota­tion of a pod to true and ensures that existing values are over­writ­ten. After you have checked the first pod, the Kuber­netes State­ful­Set will auto­mat­ic­ally ini­tial­ise the next pod. You can then repeat the debugging steps for each ad­di­tion­al pod in the State­ful­Set.

In the Kuber­netes tutorial, you’ll find detailed and practical in­form­a­tion on setting up a Kuber­netes cluster.