Using the “change-cause” Kubernetes annotation as a changelog

Suppose you have an application you are deploying to your kubernetes cluster. For most purposes, running kubectl rollout history deployments/your-app will give you a very simple revision history.

$ kubectl rollout history deployments/awesome-app
REVISION  CHANGE-CAUSE
1         <none>

However, what if you had multiple deployments by different people. How would you know what was the reason for the deployment? Especially when you have something like this?

REVISION  CHANGE-CAUSE
1         <none>
2
3
4
5
...
...
100       <none>
101       <none>
102       <none>

It is possible to set a value into the change-cause field via an annotation, but that field is quite volatile, it is also filled/replaced if someone uses the --record flag when doing an apply. However, it can be utilised to make it much more useful:

REVISION  CHANGE-CAUSE
11        Deploy new version of awesome-app to test environment
12        Deploy new version of awesome-app to staging environment
13        Deploy new version of awesome-app, Thu 21 Jun 07:01:03 BST 2018
14        Deploy new version of awesome-app with integration to gitlab v0.0.0 [test]

How is this done? Pretty simply, actually. here’s a snippet from the deploy script I use.

echo Deploy message?
read MESSAGE
if [ -z "$MESSAGE" ]; then
  MESSAGE="Deploy new version of awesome-app, $(date)"
  echo Blank message detected, defaulting to \"$MESSAGE\"
fi
echo Deploy updates...
cat deploy.yaml | sed s/'SUB_TIMESTAMP'/"$(date)"/g | kubectl replace -f -
kubectl annotate deployment awesome-app kubernetes.io/change-cause="$MESSAGE" --record=false --overwrite=true
kubectl rollout status deployments/awesome-app
kubectl rollout history deployment awesome-app

For lines 1 to 6, I read in a message from the terminal to populate the annotation, and if nothing is provided, a default is used.
On line 8, I replace the timestamp to trigger a change to the deployment (this can be anything, for example, changing the version tag of your docker image from awesome-app:release-1.0 to awesome-app:release-1.1)

Note that I used replace and not applyreplace will reset the deployment declaration, and since my deploy yaml does NOT contain a change-cause annotation, replace will remove the annotation.

On line 9, I annotate the deployment, making sure I don’t record it and overwrite the annotation in the event it’s there already (though those two switches might be redundant)

On line 10 I check the status of the rollout — this blocks until it is complete

On line 11, I then dump the deployment history.

This is an example of a script run:

$ ./deploy.sh
Deploy message?
[typed] Deploy new version of awesome-app with gitlab integration v0.0.0 [test]
Deploy updates...
deployment "awesome-app" replaced
deployment "awesome-app" annotated
Waiting for rollout to finish: 1 old replicas are pending termination...
deployment "awesome-app" successfully rolled out
deployments "awesome-app"
REVISION  CHANGE-CAUSE
11        Deploy new version of awesome-app, Thu 21 Jun 07:00:19 BST 2018
12        Deploy new version of awesome-app, Thu 21 Jun 07:00:52 BST 2018
13        Deploy new version of awesome-app, Thu 21 Jun 07:01:03 BST 2018
14        Deploy new version of awesome-app with integration to gitlab v0.0.0 [test]

kubectl Displaying Taints

One of the questions in my CKA exam was how to display taints with kubectl. While you can use kubectl describe, it creates a lot of other information too.

Then I found out about jsonpath and it’s similarity to jq

You can display the taints with something like

for a in $(kubectl get nodes --no-headers | awk '{print $1}')
do
echo $a -- $(kubectl get nodes/$a -o jsonpath='{.spec.taints[*].key}{":"}{.spec.taints[*].effect}')
done

Sample output

ip-10-10-10-147.eu-west-2.compute.internal -- node-role.kubernetes.io/master:NoSchedule
ip-10-10-10-159.eu-west-2.compute.internal -- :

So the first one has a taint (it’s the master node) and the second one doesn’t.
(maybe I need to hack this a bit more when I have multiple taints but I’ll do that when I have some multi-tainted nodes to play with)

EDIT: Another way as provided by tdodds81

$ kubectl get nodes -o=custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME TAINTS
ip-10-10-10-148.eu-west-2.compute.internal [map[effect:NoSchedule key:node-role.kubernetes.io/master]]
ip-10-10-10-218.eu-west-2.compute.internal
ip-10-10-10-239.eu-west-2.compute.internal
ip-10-10-10-249.eu-west-2.compute.internal
ip-10-10-10-51.eu-west-2.compute.internal

And with multi-taints, it looks like this (on a GKE cluster)

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME                                                  TAINTS
gke-test-cluster-k8s-n1-highmem-2-nod-589548dd-3z1v   [map[effect:NoSchedule key:key1 timeAdded:<nil> value:value1] map[value:value2 effect:NoSchedule key:key2 timeAdded:<nil>]]
gke-test-cluster-k8s-n1-highmem-2-nod-7a33f13b-9lwk   <none>
gke-test-cluster-k8s-n1-highmem-2-nod-7a33f13b-lnvl   <none>
gke-test-cluster-k8s-n1-highmem-2-nod-7a33f13b-nt49   <none>
gke-test-cluster-k8s-n1-highmem-2-nod-7a33f13b-xgbd   <none>

CKAD

While booking my CKA exam earlier, I spotted that there was a second Kubernetes certification — Certified Kubernetes Application Developer.

While the CKA is for the admin of the cluster and tests the knowledge of the infrastructure, CKAD is more for the developers.

Looks like that’ll be my next certification…

%d bloggers like this: