Guide to creating a Kubernetes Cluster in existing subnets & VPC on AWS with kops

This article is a guide on how to setup a Kubernetes cluster in AWS using kops and plugging it into your own subnets and VPC. We attempt to minimise the external IPs used in this method.

Export your AWS API keys into environment variables

export AWS_ACCESS_KEY_ID='YOUR_KEY'
export AWS_SECRET_ACCESS_KEY='YOUR_ACCESS_KEY'
export CLUSTER_NAME="my-cluster-name"
export VPC="vpc-xxxxxx"
export K8SSTATE="s3-k8sstate"</pre>

Create the cluster (you can change some of these switches to match your requirements. I would suggest only using one worker node and one master node to begin with and then increase them once you have confirmed the config is good. The more workers and master nodes you have, the longer it will take to run a rolling-update.

kops create cluster --cloud aws --name $CLUSTER_NAME --state s3://$K8SSTATE --node-count 1 --zones eu-west-1a,eu-west-1b,eu-west-1c --node-size t2.micro --master-size t2.micro --master-zoneseu-west-1a,eu-west-1b,eu-west-1c --ssh-public-key ~/.ssh/id_rsa.pub --topology=private --networking=weave --associate-public-ip=false --vpc $VPC

Important note: There must be an ODD number of master zones. If you tell kops to use an even number zones for master, it will complain.

If you want to use additional security groups, don’t add them yet — add them after you have confirmed the cluster is working.

Internal IPs: You must have a VPN connection into your VPC or you will not be able to ssh into the instances. The alternative is to use the bastion functionality using the --bastion flag with the create command. Then doing this:

ssh -i ~/.ssh/id_rsa -o ProxyCommand='ssh -W %h:%p admin@bastion.$CLUSTER_NAME' admin@INTERNAL_MASTER_IP

However, if you do this method, you MUST then use public IP addressing on the api load balancer, as you will not be able to do kops validate otherwise.

Edit the cluster

kops edit cluster $CLUSTER_NAME --state=s3://$K8SSTATE

Make the following changes:

If you have a VPN connection into the VPC, change spec.api.loadBalancer.type to “Internal“, otherwise, leave it as “Public
Change spec.subnets to match your private subnets. To use existing private subnets, they should also include the id of the subnet and match the CIDR range, e.g.:

subnets:
- cidr: 10.10.10.0/23
  id: subnet-xxxxxxx
  name: eu-west-1a
  type: Private
  zone: eu-west-1a</pre>

The utility subnet is where the Bastion hosts will be placed, and these should be in a public subnet, since they will be the inbound route into the cluster from the internet.

If you need to change or add specific IAM permissions, add them under spec.additionalPolicies like this to add additional policies to the node IAM policy (apologies about the formatting. WordPress is doing something weird to it.)

additionalPolicies:
  node: | 
    [ 
      {  
        "Effect": "Allow", 
        "Action": ["dynamodb:*"],
        "Resource": ["*"] 
      },  
      {  
        "Effect": "Allow",  
        "Action": ["es:*"],     
        "Resource": ["*"]     
      }    
    ]

Edit the bastion, nodes, and master configs (MASTER_REGION is the zone where you placed the master. If you are running a multi-region master config, you’ll have to do this for each region)

kops edit ig master-{MASTER_REGION} --name=$CLUSTER_NAME --state s3://$K8SSTATE

kops edit ig nodes --name=$CLUSTER_NAME --state s3://$K8SSTATE
kops edit ig bastions --name=$CLUSTER_NAME --state s3://$K8SSTATE

Check and make any updates.

If you want a mixture of instance types (e.g. t2.mediums and r3.larges), you’ll need to separate these using new instance groups ($SUBNETS is the subnets where you want the nodes to appear — for example, you can provide a list “eu-west-2a,eu-west-2b)

kops create ig anothernodegroup --state s3://$K8SSTATE --subnets $SUBNETS

You can later delete this with

kops delete ig anothernodegroup --state s3://$K8SSTATE

If you want to use spot prices, add this under the spec section (x.xx is the price you want to bid):

maxPrice: "x.xx"

Check the instance size and count if you want to change them (I would recommend not changing the node count just yet)

If you want to add tags to the instances (for example for billing), add something like this to the spec section:

cloudLabels:
  Billing: product-team</pre>

If you want to run some script(s) at node startup (cloud-init), add them to spec.additionalUserData:

spec:
  additionalUserData:
  - name: myscript.sh
    type: text/x-shellscript
    content: |
      #!/bin/sh
      echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt

Apply the update:

kops update cluster $CLUSTER_NAME --state s3://$K8SSTATE --yes

Wait for DNS to propagate and then validate

kops validate cluster --state s3://$K8SSTATE

Once the cluster returns ready, apply the Kubernetes dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/alternative/kubernetes-dashboard.yaml

Access the dashboard via

https://api.$CLUSTER_NAME/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

also try:

https://api.$CLUSTER_NAME/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

If the first doesn’t work

(ignore the cert error)

Username is “admin” and the password is found from your local ~/.kube/config

Add the External DNS update to allow you to give friendly names to your externally-exposed services rather than the horrible elb names.

See here: https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/aws.md

(You can apply the yaml directly onto the cluster via the dashboard. Make sure you change the filter to match your domain or subdomain. )

Note that if you use this, you’ll need to change the node IAM policy on the cluster config as the default IAM policy won’t allow the External DNS container to modify Route 53 entries, and also annotate (use kubectl annotate $service_name key:value) your service with text such as:

external-dns.alpha.kubernetes.io/hostname: $SERVICE_NAME.$CLUSTERNAME

And also you might need this annotation, to make the ELB internal rather than public – otherwise Kubernetes will complain “Error creating load balancer (will retry): Failed to ensure load balancer for service namespace/service: could not find any suitable subnets for creating the ELB”

service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

(Optional) Add the Cockpit pod to your cluster as described here

http://cockpit-project.org/guide/133/feature-kubernetes.html

It will allow you to visually see a topology of your cluster at a cluster and also provides some management features too. For example, here’s my cluster. It contains 5 nodes (1 master, 4 workers and is running 4 services (Kubernetes, external-dns, cockpit, and dashboard). Cockpit creates a replication controller so it knows about the changes.

chrome_2018-01-14_15-44-00

Add any additional security groups by adding this under the spec section of the node/master/bastions config, then do a rolling-update (you might need to use the --force switch), do this as soon as you can after creating and verifying the cluster updates work.

additionalSecurityGroups:
- sg-xxxxxxxx
- sg-xxxxxxxx

If the cluster breaks after this (i.e. the nodes haven’t shown up on the master), reboot the server (don’t terminate, use the reboot option from the AWS console), and see if that helps. If it still doesn’t show up, there’s something wrong with the security groups attached — i.e. they’re conflicting somehow with the Kubernetes access. Remove those groups and then do another rolling-update but use both the --force and --cloudonly switches to force a “dirty” respin.

If the cluster comes up good, then you can change the node counts on the configs and apply the update.

Note that if you change the node count and then apply the update, the cluster attempts to make the update without rolling-update. For example, if you change the node count from 1 to 3, the cluster attempts to bring up the 2 additional nodes.

Other things you can look at:

Kompose – which converts a docker-compose configuration into Kubernetes resources

Finally, have fun!

Uploading (and Resuming) videos to YouTube via GoogleCL and AWS – Updated 15th Dec

If you are like me, and have a slow and/or unreliable internet connection, trying to upload any reasonably-sized video to YouTube can be a nightmare, forcing you to have your computer on for hours on end, and then finding your upload failed because your connection dropped, and then having to start all over again.

Well, one way to have resume protection is to use a middle-point, which is Amazon Web Services, or a similar cloud-based provider, then using that to upload to YouTube. Since the connection between the cloud system and YouTube is likely to be more reliable (and faster) than your connection, the upload from the cloud system to YouTube will be faster.

The first step is to setup and start an instance on AWS. I am using the Ubuntu image.

SSH into the instance and install supporting packages via apt-get or aptitude. Make sure you change the IP (xx.xx.xx.xx) and the key (AWS_Ireland.pem) to match your files.

$ ssh -o IdentityFile=/home/user/.ssh/AWS_Ireland.pem ubuntu@xx.xx.xx.xx

$ sudo apt-get install python-gdata python-support rsync

Then download the latest googlecl deb file from https://code.google.com/p/googlecl/downloads/list

$ wget https://googlecl.googlecode.com/files/googlecl_0.9.14-2_all.deb

Now, install the deb file using dpkg

$ sudo dpkg -i googlecl_0.9.14-2_all.deb

We can now start using the Google services, but first we need to authenticate. This is normally done via a browser, but since we are in a terminal, we skip this.

$ google youtube list
Please specify user: [enter your email address here]

You will see a text-version of the login page. Don’t bother entering your values. Just press ‘q’ to quit and confirm exit. Then, you’ll see in the terminal window, a url along the lines of this:

Please log in and/or grant access via your browser at:
https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token={hidden}&hd=default

Go to that url and sign in. Then, come back to the console and press enter. If all goes well, you should see your video uploads in the console window.

Now, to upload a video to the AWS instance. You can use rsync for that, and the command to enter into your local terminal is as follows (change the key file to match yours and the IP address field to match your instance’s IP):

rsync -vhPz --compress-level=9 -e "ssh -o IdentityFile=/home/user/.ssh/AWS_Ireland.pem" source ubuntu@{EC2_IP}:.

This uploads the video called “source” onto your EC2 instance at the home folder of the default user (if you have another location in your instance, use that here). Rsync will allow you to resume uploads via the P switch. When the rsync command successfully completes, you can then SSH back onto the instance, and use the “google youtube post” command to upload your video onto YouTube.

NOTE: On some large files, rsync breaks on resuming with the error message “broken pipe”, if this happens to you, see this page (specifically, Q3).

Once your video is uploaded to your EC2 instance, you can then upload that video to YouTube by using this:

$ google youtube post path/to/video

Cloud Storage

Gah, I really hate corporate firewalls. They block so many sites it’s sometimes impossible to do any work. I have Cloud Storage accounts with various sites, and can’t access:

However, I can access

My main cloud storage is with DropBox because they have a linux-native application for Fedora and Ubuntu. Whilst JustCloud also have a linux application, it’s Ubuntu only, so even though I have an account with JustCloud, I can’t effectively use it yet on my laptop. But, what I do like is that JustCloud tracks your mobile device once you install the app on it. Similar to the Cerberus app. So you can keep tabs on where it is. Although there’s nowhere to hide the app or make it a device administrator, it at least gives you an idea of where to start looking for your device if you can’t find it.

Behind my draconian corporate firewall the only one I can effectively use is Google Drive since they haven’t blocked Google, and I doubt they will since it’s probably the de facto search engine for most of the Internet users (although I prefer a metasearch engine like IXQuick.

%d bloggers like this: