After a while using Kubernetes, I tried to improve the cluster and try to use all the features from the cloud provider. The following guide is an improvement of the post Kubernetes cluster with Autoscaling on AWS.
The environment
- AWS (The idea is the same for other cloud providers)
- Kubernetes
- Autoscaling groups for the workes for each availability zone (
zone-a
,zone-b
,zone-c
). - Cluster-Autoscaler (CA)
The cluster will be scale when:
- There are pods that failed to run in the cluster due to insufficient resources.
- There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
We have instances groups (nodes groups) per availability zone to avoid some limitations about persistent volumes, I will explain in another post about it.
1. Edit the instance group (node groups)
First, you need to add proper tags to the instance group and the workers nodes to allow the Cluster-Autoscaler can auto-discovered.
Add the following cloud labels:
- key
k8s.io/cluster-autoscaler/enabled
, valuetrue
. - key
k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>
, valueowned
. - key
k8s.io/cluster-autoscaler/node-template/label/zone
, value<AVAILABILITY ZONE>
.
Add the following node labels:
- key
zone
, value<AVAILABILITY ZONE>
.
Kubernetes provides the node label "failure-domain.beta.kubernetes.io/zone" but will be deprecated, so I decided to create my own node label "zone" to keep compatibility after update Kubernetes.
In my cluster, I have 3 instance groups, one per availability zone.
Instance group for the zone-a:
kind: InstanceGroup
role: Node
...
spec:
cloudLabels:
k8s.io/cluster-autoscaler/enabled: "true"
k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1a"
nodeLabels:
zone: "eu-west-1a"
...
Instance group for the zone-b:
kind: InstanceGroup
role: Node
...
spec:
cloudLabels:
k8s.io/cluster-autoscaler/enabled: "true"
k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1b"
nodeLabels:
zone: "eu-west-1b"
...
Instance group for the zone-c:
kind: InstanceGroup
role: Node
...
spec:
cloudLabels:
k8s.io/cluster-autoscaler/enabled: "true"
k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>: "owned"
k8s.io/cluster-autoscaler/node-template/label/zone: "eu-west-1c"
nodeLabels:
zone: "eu-west-1c"
...
2. IAM policies
The Cluster-Autoscaler needs access to certain resources from AWS, such as the autoscaling
service.
I recommend use Kiam or Kube2iam to provide IAM policies to your pods.
The easy way but not more secure is assign theses roles to the workers nodes where the Cluster-Autoscaler is running, or another option is create a service user with the IAM roles need it.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
3. Deploy the Kubernete Cluster-Autoscaler
For this article, we are going to use the CA with the Auto-Discovery.
Download the yaml descriptor from here:
Edit the yaml and change the cluster name <YOUR CLUSTER NAME>
for the same you set up in the previous steps, also change the image: k8s.gcr.io/cluster-autoscaler:XX.XX.XX
with the proper version for your cluster.
Apply the yaml file to deploy the container.
kubectl apply -f cluster-autoscaler-autodiscover.yaml
For troubleshooting you can check the logs of the pod, is running in kube-system
namespace.
Please check the official guide for more options.
5. Test it
Create a simple deployment with an nginx image and then add a bunch of replicas.
kubectl create deployment webserver-demo --image=nginx:latest
kubectl scale deployment webserver-demo --replicas=200
# Check the pods if are in "Pending" status
kubectl get pods
# Check the amount of nodes, will take a few minutes until the new node join to the cluster
kubectl get nodes
# Check the config map where the Cluster-Autoscaler keep information about the cluster
kubectl describe configmaps cluster-autoscaler-status -n kube-system