/var/log/diego

Kubernetes multiple zones and autoscaling
July 6, 2020 - Last updated: April 26, 2021

After a while using Kubernetes, I tried to improve the cluster and try to use all the features from the cloud provider such as the autoscaling groups and multiple availability zones.

The environment

  • AWS (The idea is the same for other cloud providers)
  • Kubernetes
  • Autoscaling groups for the workes for each availability zone (zone-a, zone-b, zone-c).
  • Cluster-Autoscaler (CA)

The cluster will scale when:

  • There are pods that failed to run in the cluster due to insufficient resources.
  • There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

1. Edit the instance group (node groups)

First, you need to add proper tags to the instance group and to the worker's nodes to allow the Cluster-Autoscaler can auto-discovered.

Add the following cloud labels:

  • key k8s.io/cluster-autoscaler/enabled, value true.
  • key k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>, value owned.
  • key k8s.io/cluster-autoscaler/node-template/label/zone, value <AVAILABILITY ZONE>.

Add the following node labels:

  • key zone, value <AVAILABILITY ZONE>.

Kubernetes provides the node label "failure-domain.beta.kubernetes.io/zone" but will be deprecated, so I decided to create my own node label "zone" to keep compatibility after update Kubernetes.

The cluster has 3 instance groups, one per availability zone.

Instance group for the zone-a:

kind: InstanceGroup
role: Node
metadata:
    name: nodes-a
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1a"
  nodeLabels:
    zone: "eu-west-1a"
...

Instance group for the zone-b:

kind: InstanceGroup
role: Node
metadata:
    name: nodes-b
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1b"
  nodeLabels:
    zone: "eu-west-1b"
...

Instance group for the zone-c:

kind: InstanceGroup
role: Node
metadata:
    name: nodes-c
spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
    k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1c"
  nodeLabels:
    zone: "eu-west-1c"
...

2. IAM policies

The Cluster Autoscaler needs access to certain resources from AWS, such as the autoscaling service.

The easy way but not more secure is to assign these roles to the worker's nodes where the Cluster Autoscaler is running. Also, you can use Kiam or Kube2iam to provide IAM policies to your pods (recommended way).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

3. Deploy the Kubernete Cluster Autoscaler

For this article, we are going to use the CA with the Auto-Discovery.

Download the yaml descriptor from here:

Edit the yaml and change the cluster name <YOUR CLUSTER NAME> for the same you set up in the previous steps, also change the image: k8s.gcr.io/cluster-autoscaler:XX.XX.XX with the proper version for your cluster.

Apply the yaml file to deploy the container.

kubectl apply -f cluster-autoscaler-autodiscover.yaml

For troubleshooting you can check the logs of the pod, is running in kube-system namespace.

Please check the official guide for more options.

5. Test it

Create a simple deployment with an nginx image and then increase the replicas.

kubectl create deployment webserver-demo --image=nginx:latest
kubectl scale deployment webserver-demo --replicas=200

# Check the pods if are in "Pending" status
kubectl get pods

# Check the number of nodes, will take a few minutes until the new node join to the cluster
kubectl get nodes

# Check the config map where the Cluster-Autoscaler keep information about the cluster
kubectl describe configmaps cluster-autoscaler-status -n kube-system

Related posts