Persistent volume in Kubernetes cluster with multiple availability zones

August 25, 2020

After a wild using Kubernetes in AWS and set-up persistent volumes via EBS, I faced a problem with evicted pods after they are re-schedule to another node; The issue was the EBS volumes are dedicated by zone, makes sense because the volumes work via networking and are dedicated per datacenter, for the network latency.

Read more

Kubernetes multiple zones and autoscaling

July 6, 2020

After a while using Kubernetes, I tried to improve the cluster and try to use all the features from the cloud provider such as the autoscaling groups and multiple availability zones.

The environment

  • AWS (The idea is the same for other cloud providers)
  • Kubernetes
  • Autoscaling groups for the workes for each availability zone (zone-a, zone-b, zone-c).
  • Cluster-Autoscaler (CA)

The cluster will scale when:

  • There are pods that failed to run in the cluster due to insufficient resources.
  • There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
Read more

Kubernetes: Delete pods older than X days

February 14, 2020

I'm working with Airflow and the Kubernetes operator which deploys from Airflow a container to Kubernetes. The pods created by Airflow have ended so the final status of the pod could be Succeeded or Failed. I was looking for to delete the pods created by Airflow but older than 5 days, so we can still read the logs from the pods until they are deleted.

Read more

Kubernetes API by examples

February 11, 2020

I been playing with the Kubernetes API and I woud like to post and update with the time some examples.

For my case I have a service account which has complete access to a particular namespace, the service account is called demo-user.

To do some test I created a pod and I configured the service account to the pod, so I can get the credentials inside the container and query the API with the service account.

Read more

Airflow: Kubernetes Operator

November 4, 2019

DAG example using KubernetesPodOperator, the idea is run a Docker container in Kubernetes from Airflow every 30 minutes.

Features:

  • Scheduled every 30 minutes.
  • Set environment variable for the pod RULES.
  • Run the pods in the namespace default.
  • Mount a volume to the container. It's just an example mounting the /tmp from host.
  • Set imagePullSecrets for private Docker registry.
Read more