Kops: Kubernetes cluster with Autoscaling on AWS

November 13, 2019 - Last updated: July 6, 2020

The autoscaling groups created by Kops is only to keep a number of nodes up without any scaling policy; The idea of the following article is create a dynamic Kubernetes cluster, measure the load of the nodes and scale up or down. To do that we need to set up some tags to the worker nodes, give permission to the worker nodes to reach the AWS service autoscaling and deploy the tool Kubernetes autoscaler.

The cluster will be scale when:

  • There are pods that failed to run in the cluster due to insufficient resources.
  • There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

The environment:

The version of "Kubernetes autoscaler" need to be the same as your Kubernetes cluster version.

1. Edit the node instances group

First, you need to add proper tags to your worker nodes, edit via kops your instance group called nodes.

kops edit instancegroups nodes

Add the following cloud labels, and modify the <YOUR CLUSTER NAME> for the name of your cluster; Also set the maxSize and minSize, for example, my worker's nodes are going to scale-down up to 2 instances and scale-up up to 10 instances.

The values for the labels can be empty.

kind: InstanceGroup
...
spec:
  cloudLabels:
    service: k8s_node
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: ""
  maxSize: 10
  minSize: 2
...

2. Edit your cluster

The Autoscaler application running in the worker nodes need access to certain resources from AWS, such as the autoscaling service.

kops edit cluster

Add the following policies to the nodes.

kind: Cluster
...
spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:DescribeLaunchConfigurations",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:TerminateInstanceInAutoScalingGroup",
            "autoscaling:DescribeTags"
          ],
          "Resource": "*"
        }
      ]
...

3. Apply the changes

After doing the changes in your cluster via Kops you need to apply them.

kops update cluster
kops rolling-update cluster

To really apply the changes you need to add "--yes" at the end of each command.

4. Deploy the Kubernete Autoscaler

For this article we are going to use the Autoscaler with the Auto-Discovery.

Download the yaml descriptor from here:

Edit the yaml and change the cluster name <YOUR CLUSTER NAME> for the same you set up in the previous steps, also change the image: k8s.gcr.io/cluster-autoscaler:XX.XX.XX with the proper version for your cluster.

Apply the yaml file to deploy the container.

kubectl apply -f cluster-autoscaler-autodiscover.yaml

For troubleshooting you can check the logs of the pod, is running in kube-system namespace.

Please check the official guide for more options.

5. Test it

Create a simple deployment with an nginx image and then add a few replicas.

kubectl create deployment webserver-demo --image=nginx:latest
kubectl scale deployment webserver-demo --replicas=200

# Check the pods if are in "Pending" status
kubectl get pods

# Check the amount of nodes, will take a few minutes until the new node join to the cluster
kubectl get nodes