Kubernetes is an awesome piece of kit, you can set applications to run within the cluster, make it visible to only apps within the cluster and/or expose it to applications outside of the cluster.
As part of my tinkering, I wanted to setup a Docker Registry to store my own images without having to make them public via docker hub. Doing this proved a bit more complicated than expected since by default, it requires SSL which requires a certificate to be purchased and installed.
Enter Let’s Encrypt which allows you to get SSL certificates for free; and by using their API, you can set it to regularly renew. Kubernetes has the kube-lego project which allows this regular integration. So here, I’ll go through enabling an application (in this case, it’s a docker registry, but it can be anything).
First, lets ignore the lego project, and set up the application so that it is accessible normally. As mentioned above, this is the docker registry
I’m tying the registry storage to a pv claim, though you can modify this to tie to S3, instead etc.
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: registry
namespace: default
labels:
name: registry
spec:
replicas: 1
selector:
matchLabels:
name: registry
template:
metadata:
creationTimestamp:
labels:
name: registry
spec:
volumes:
- name: registry-data
persistentVolumeClaim:
claimName: registry-data
containers:
- name: registry
image: registry:2
resources: {}
volumeMounts:
- name: registry-data
mountPath: "/var/lib/registry"
terminationMessagePath: "/dev/termination-log"
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
strategy:
type: Recreate
---
kind: Service
apiVersion: v1
metadata:
name: registry
namespace: default
labels:
name: registry
spec:
ports:
- protocol: TCP
port: 9000
targetPort: 5000
selector:
name: registry
type: LoadBalancer
sessionAffinity: None
externalTrafficPolicy: Cluster
Once you’ve applied this, verify your config is correct by ensuring you have an external endpoint for the service (use kubectl describe service registry | grep "LoadBalancer Ingress"). On AWS, this will be an ELB, on other clouds, you might get an IP. If you get an ELB, CNAME a friendly name to it. If you get an IP, create an A record for it. I’m going to use registry.blenderfox.com for this test.
Verify by doing this. Bear in mind it can take a while before DNS records updates so be patient.
host $(SERVICE_DNS)
So if I had set the service to be registry.blenderfox.com, I would do
host registry.blenderfox.com
If done correctly, this should resolve to the ELB then resolve to the ELB IP addresses.
Next, try to tag a docker image of the format registry-host:port/imagename, so, for example, registry.blenderfox.com:9000/my-image.
Next try to push it.
docker push registry.blenderfox.com:9000/my-image
It will fail because it can’t talk over https
docker push registry.blenderfox.com:9000/my-image The push refers to repository [registry.blenderfox.com:9000/my-image] Get https://registry.blenderfox.com:9000/v2/: http: server gave HTTP response to HTTPS client
So let’s now fix that.
Now let’s start setting up kube-lego
Checkout the code
git clone git@github.com:jetstack/kube-lego.git
cd into the relevant folder
cd kube-lego/examples/nginx
Start applying the code base
kubectl apply -f lego/00-namespace.yaml kubectl apply -f nginx/00-namespace.yaml kubectl apply -f nginx/default-deployment.yaml kubectl apply -f nginx/default-service.yaml
Open up nginx/configmap.yaml and change the body-size: "64m" line to a bigger value. This is the maximum size you can upload through nginx. You’ll see why this is an important change later.
kubectl apply -f nginx/configmap.yaml kubectl apply -f nginx/service.yaml kubectl apply -f nginx/deployment.yaml
Now, look for the external endpoint for the nginx service
kubectl describe service nginx -n nginx-ingress | grep "LoadBalancer Ingress"
Look for the value next to LoadBalancer Ingress. On AWS, this will be the ELB address.
CNAME your domain for your service (e.g. registry.blenderfox.com in this example) to that ELB. If you’re not on AWS, this may be an IP, in which case, just create an A record instead.
Open up lego/configmap.yaml and change the email address in there to be the one you want to use to request the certs.
kubectl apply -f lego/configmap.yaml kubectl apply -f lego/deployment.yaml
Wait for the DNS to update before proceeding to the next step.
host registry.blenderfox.com
When the DNS is updated, finally create and add an ingress rule for your service:
---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: registry
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'
spec:
tls:
- hosts:
- registry.blenderfox.com
secretName: docker-tls
rules:
- host: registry.blenderfox.com
http:
paths:
- path: "/"
backend:
serviceName: registry
servicePort: 9000
status:
loadBalancer:
ingress:
- {}
Look add the logs in nginx-ingress/nginx and you’ll see the Let’s Encrypt server come in to validate:
100.124.0.0 - [100.124.0.0] - - [19/Jan/2018:09:50:19 +0000] "GET /.well-known/acme-challenge/[REDACTED] HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 277 0.044 100.96.0.3:8080 87 0.044 200
And look in the logs on the kube-lego/kube-lego pod and you’ll see the success and saving of the secret
time="2018-01-19T09:49:45Z" level=info msg="requesting certificate for registry.blenderfox.com" context="ingress_tls" name=registry namespace=default time="2018-01-19T09:50:21Z" level=info msg="authorization successful" context=acme domain=registry.blenderfox.com time="2018-01-19T09:50:47Z" level=info msg="successfully got certificate: domains=[registry.blenderfox.com] url=https://acme-v01.api.letsencrypt.org/acme/cert/[REDACTED]" context=acme time="2018-01-19T09:50:47Z" level=info msg="Attempting to create new secret" context=secret name=registry-tls namespace=default time="2018-01-19T09:50:47Z" level=info msg="Secret successfully stored" context=secret name=registry-tls namespace=default
Now let’s do a quick verify:
curl -ILv https://registry.blenderfox.com ... * Server certificate: * subject: CN=registry.blenderfox.com * start date: Jan 19 08:50:46 2018 GMT * expire date: Apr 19 08:50:46 2018 GMT * subjectAltName: host "registry.blenderfox.com" matched cert's "registry.blenderfox.com" * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. ...
That looks good.
Now let’s re-tag and try to push our image
docker tag registry.blenderfox.com:9000/my-image registry.blenderfox.com/my-image docker push registry.blenderfox.com/my-image
Note we are not using a port this time as there is now support for SSL.
BOOM! Success.
The tls section indicates the host to request the cert on, and the backend section indicates which backend to pass the request onto. The body-size config is at the nginx level so if you don’t change it, you can only upload a maximum of 64m even if the backend service (docker registry in this case) can support it. I have it set here at “1g” so I can upload 1gb (some docker images can be pretty large)
Blender Fox 
You must be logged in to post a comment.