/var/log/diego

Kubernetes multiple zones and autoscaling
July 6, 2020 - Last updated: July 23, 2020

After a while using Kubernetes, I tried to improve the cluster and try to use all the features from the cloud provider. The following guide is an improvement of the post Kubernetes cluster with Autoscaling on AWS.

The environment

  • AWS (The idea is the same for other cloud providers)
  • Kubernetes
  • Autoscaling groups for the workes for each availability zone (zone-a, zone-b, zone-c).
  • Cluster-Autoscaler (CA)

The cluster will be scale when:

  • There are pods that failed to run in the cluster due to insufficient resources.
  • There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

We have instances groups (nodes groups) per availability zone to avoid some limitations about persistent volumes, I will explain in another post about it.

1. Edit the instance group (node groups)

First, you need to add proper tags to the instance group and the workers nodes to allow the Cluster-Autoscaler can auto-discovered.

Add the following cloud labels:

  • key k8s.io/cluster-autoscaler/enabled, value true.
  • key k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>, value owned.
  • key k8s.io/cluster-autoscaler/node-template/label/zone, value <AVAILABILITY ZONE>.

Add the following node labels:

  • key zone, value <AVAILABILITY ZONE>.

Kubernetes provides the node label "failure-domain.beta.kubernetes.io/zone" but will be deprecated, so I decided to create my own node label "zone" to keep compatibility after update Kubernetes.

In my cluster, I have 3 instance groups, one per availability zone.

Instance group for the zone-a:

kind: InstanceGroup
role: Node
...
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1a"
  nodeLabels:
    zone: "eu-west-1a"
...

Instance group for the zone-b:

kind: InstanceGroup
role: Node
...
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1b"
  nodeLabels:
    zone: "eu-west-1b"
...

Instance group for the zone-c:

kind: InstanceGroup
role: Node
...
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1c"
  nodeLabels:
    zone: "eu-west-1c"
...

2. IAM policies

The Cluster-Autoscaler needs access to certain resources from AWS, such as the autoscaling service.

I recommend use Kiam or Kube2iam to provide IAM policies to your pods.

The easy way but not more secure is assign theses roles to the workers nodes where the Cluster-Autoscaler is running, or another option is create a service user with the IAM roles need it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

3. Deploy the Kubernete Cluster-Autoscaler

For this article, we are going to use the CA with the Auto-Discovery.

Download the yaml descriptor from here:

Edit the yaml and change the cluster name <YOUR CLUSTER NAME> for the same you set up in the previous steps, also change the image: k8s.gcr.io/cluster-autoscaler:XX.XX.XX with the proper version for your cluster.

Apply the yaml file to deploy the container.

kubectl apply -f cluster-autoscaler-autodiscover.yaml

For troubleshooting you can check the logs of the pod, is running in kube-system namespace.

Please check the official guide for more options.

5. Test it

Create a simple deployment with an nginx image and then add a bunch of replicas.

kubectl create deployment webserver-demo --image=nginx:latest
kubectl scale deployment webserver-demo --replicas=200

# Check the pods if are in "Pending" status
kubectl get pods

# Check the amount of nodes, will take a few minutes until the new node join to the cluster
kubectl get nodes

# Check the config map where the Cluster-Autoscaler keep information about the cluster
kubectl describe configmaps cluster-autoscaler-status -n kube-system

Related posts