Enabling and using Let’s Encrypt SSL Certificates on Kubernetes

Kubernetes is an awesome piece of kit, you can set applications to run within the cluster, make it visible to only apps within the cluster and/or expose it to applications outside of the cluster.

As part of my tinkering, I wanted to setup a Docker Registry to store my own images without having to make them public via docker hub.  Doing this proved a bit more complicated than expected since by default, it requires SSL which requires a certificate to be purchased and installed.

Enter Let’s Encrypt which allows you to get SSL certificates for free; and by using their API, you can set it to regularly renew. Kubernetes has the kube-lego project which allows this regular integration. So here, I’ll go through enabling an application (in this case, it’s a docker registry, but it can be anything).

First, lets ignore the lego project, and set up the application so that it is accessible normally. As mentioned above, this is the docker registry

I’m tying the registry storage to a pv claim, though you can modify this to tie to S3, instead etc.

kind: Deployment
apiVersion: extensions/v1beta1
  name: registry
  namespace: default
    name: registry
  replicas: 1
      name: registry
        name: registry
      - name: registry-data
          claimName: registry-data
      - name: registry
        image: registry:2
        resources: {}
        - name: registry-data
          mountPath: "/var/lib/registry"
        terminationMessagePath: "/dev/termination-log"
        terminationMessagePolicy: File
        imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
    type: Recreate
kind: Service
apiVersion: v1
  name: registry
  namespace: default
    name: registry
  - protocol: TCP
    port: 9000
    targetPort: 5000
    name: registry
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster

Once you’ve applied this, verify your config is correct by ensuring you have an external endpoint for the service (use kubectl describe service registry | grep "LoadBalancer Ingress"). On AWS, this will be an ELB, on other clouds, you might get an IP. If you get an ELB, CNAME a friendly name to it. If you get an IP, create an A record for it. I’m going to use registry.blenderfox.com for this test.

Verify by doing this. Bear in mind it can take a while before DNS records updates so be patient.


So if I had set the service to be registry.blenderfox.com, I would do

host registry.blenderfox.com

If done correctly, this should resolve to the ELB then resolve to the ELB IP addresses.

Next, try to tag a docker image of the format registry-host:port/imagename, so, for example, registry.blenderfox.com:9000/my-image.

Next try to push it.

docker push registry.blenderfox.com:9000/my-image

It will fail because it can’t talk over https

docker push registry.blenderfox.com:9000/my-image
The push refers to repository [registry.blenderfox.com:9000/my-image]
Get https://registry.blenderfox.com:9000/v2/: http: server gave HTTP response to HTTPS client

So let’s now fix that.

Now let’s start setting up kube-lego

Checkout the code
git clone git@github.com:jetstack/kube-lego.git

cd into the relevant folder
cd kube-lego/examples/nginx

Start applying the code base

kubectl apply -f lego/00-namespace.yaml
kubectl apply -f nginx/00-namespace.yaml
kubectl apply -f nginx/default-deployment.yaml
kubectl apply -f nginx/default-service.yaml

Open up nginx/configmap.yaml and change the body-size: "64m" line to a bigger value. This is the maximum size you can upload through nginx. You’ll see why this is an important change later.

kubectl apply -f nginx/configmap.yaml
kubectl apply -f nginx/service.yaml
kubectl apply -f nginx/deployment.yaml

Now, look for the external endpoint for the nginx service
kubectl describe service nginx -n nginx-ingress | grep "LoadBalancer Ingress"

Look for the value next to LoadBalancer Ingress. On AWS, this will be the ELB address.

CNAME your domain for your service (e.g. registry.blenderfox.com in this example) to that ELB. If you’re not on AWS, this may be an IP, in which case, just create an A record instead.

Open up lego/configmap.yaml and change the email address in there to be the one you want to use to request the certs.

kubectl apply -f lego/configmap.yaml
kubectl apply -f lego/deployment.yaml

Wait for the DNS to update before proceeding to the next step.

host registry.blenderfox.com

When the DNS is updated, finally create and add an ingress rule for your service:

kind: Ingress
apiVersion: extensions/v1beta1
  name: registry
  namespace: default
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
  - hosts:
    - registry.blenderfox.com
    secretName: docker-tls
  - host: registry.blenderfox.com
      - path: "/"
          serviceName: registry
          servicePort: 9000
    - {}

Look add the logs in nginx-ingress/nginx and you’ll see the Let’s Encrypt server come in to validate: - [] - - [19/Jan/2018:09:50:19 +0000] "GET /.well-known/acme-challenge/[REDACTED] HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 277 0.044 87 0.044 200

And look in the logs on the kube-lego/kube-lego pod and you’ll see the success and saving of the secret

time="2018-01-19T09:49:45Z" level=info msg="requesting certificate for registry.blenderfox.com" context="ingress_tls" name=registry namespace=default 
time="2018-01-19T09:50:21Z" level=info msg="authorization successful" context=acme domain=registry.blenderfox.com 
time="2018-01-19T09:50:47Z" level=info msg="successfully got certificate: domains=[registry.blenderfox.com] url=https://acme-v01.api.letsencrypt.org/acme/cert/[REDACTED]" context=acme 
time="2018-01-19T09:50:47Z" level=info msg="Attempting to create new secret" context=secret name=registry-tls namespace=default 
time="2018-01-19T09:50:47Z" level=info msg="Secret successfully stored" context=secret name=registry-tls namespace=default 

Now let’s do a quick verify:

curl -ILv https://registry.blenderfox.com
* Server certificate:
*  subject: CN=registry.blenderfox.com
*  start date: Jan 19 08:50:46 2018 GMT
*  expire date: Apr 19 08:50:46 2018 GMT
*  subjectAltName: host "registry.blenderfox.com" matched cert's "registry.blenderfox.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.

That looks good.

Now let’s re-tag and try to push our image

docker tag registry.blenderfox.com:9000/my-image registry.blenderfox.com/my-image
docker push registry.blenderfox.com/my-image

Note we are not using a port this time as there is now support for SSL.

BOOM! Success.

The tls section indicates the host to request the cert on, and the backend section indicates which backend to pass the request onto. The body-size config is at the nginx level so if you don’t change it, you can only upload a maximum of 64m even if the backend service (docker registry in this case) can support it. I have it set here at “1g” so I can upload 1gb (some docker images can be pretty large)

Kali Linux Pentesting Distribution — Now Runnable in Browser

Everyone loves hearing about pentesting and ethical hacking distros these days, and it looks like it is even becoming a trend among aspiring security professionals.

Therefore, today we have some good news for those who want to try one of the best penetration testing and security auditing operating systems based on the Linux kernel, Kali Linux, the successor of the popular BackTrack, and don’t have the resources to run the Live CD or install the OS on their computers.

Network security specialist Jerry Gamblin has created a project called KaliBrowser, which, if you haven’t already guessed, it allows you to run the famous Kali Linux operating system on a web browser, using the Kali Linux Docker image, Openbox window manager, and NoVNC HTML5-based VNC client.

Source: http://news.softpedia.com/news/you-can-now-run-backtrack-successor-kali-linux-pentest-os-in-your-web-browser-504809.shtml

Docker Builds – Update

As I relied on grive to do the sync between my local machine and Google Drive, where the builds were stored, I found out (at work, ironically, since we use some Google APIs), that Google shut off some of their APIs on 20th April, which killed some of our functionality and also, killed grive’s functionality with some really cryptic messages in the console window. Nonetheless, I found that an alternative, “drive” works, although a hell of a lot slower.

Docker.io Builds Page For 32-bit Architectures

I have started posting up my builds of Docker.io. They are unofficial, and unsupported by the community, pending official support and code release supporting 32-bit architectures.


I have setup my system to auto-build every week and post to this shared directory. There’s a readme in the shared folder.

Docker.io Build Script Update

Some changes to the Docker.io code has caused the build script to fail, this was down to the code now using btrfs to build a driver. It has taken me a while to figure out how to fix that error message, but the script now works. You have to add this chunk of code anywhere before the main docker build

git clone git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
mv btrfs-progs btrfs #Needed to include into Docker code
export PATH=$PATH:$(pwd)
cd btrfs
make || (echo "btrfs compile failed" && exit 1)
export C_INCLUDE_PATH=$C_INCLUDE_PATH:$(pwd) #Might not be needed
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:$(pwd) #Might not be needed
echo PATH: $PATH
cd ..

How Splitting A Computer Into Multiple Realities Can Protect You From Hackers

Virtualisation, Sandboxes, Containers. All terms and technologies used for various reasons. Security is not always the main reason, but considering the details in this article, it is a valid point. It is simple enough to setup a container in your machine. LXC/Linux Containers for example, don’t have as much overhead as a VirtualBox or VMWare virtual machine and can run almost, if not just as fast as a native installation (I’m using LXC for my Docker.io build script), but conceptually, if you use a container, and it is infected with malware, you can drop and rebuild the container, or roll back to a snapshot much more easily than reimaging your machine.

Right now I run three different containers — one is my main Ubuntu Studio, which is not a container, but my core OS. the second is my Docker.io build LXC, which I rebuild everytime I compile (and I now have that tied into Jenkins, so I might put up regular builds somehow), and the final one is a VirtualBox virtual machine that runs Windows 7 so I don’t have to dual boot.

How Splitting A Computer Into Multiple Realities Can Protect You From Hackers | WIRED.

Building Docker.io on Ubuntu 32-bit

Interestingly, after upgrading to Ubuntu Utopic Unicorn, the build script I made for Docker.io fails during the Go build. Something inside the Utopic minimal install is not being liked by the Go build script, so for now, you will have to force the LXC container to use Trusty instead.

lxc-create -n Ubuntu -t ubuntu -- --release trusty --auth-key /home/user/.ssh/id_rsa.pub



%d bloggers like this: