Creating EKS Cluster with Karpenter Auto scaling - Terraform | Kubernetes

Kubernetes has been a disruptive change in IT Infrastructure. It had been widely adopted across the industry

Managed Kubernetes are the best choice if you do not want to face the hurdles of setting up K8s from scratch Unless you want to deep learn Kubernetes.

AWS a market leader in Cloud Infrastructure as a service has a perfectly tailored solution for Kubernetes called EKS - Elastic Kubernetes Services.

Creating EKS Cluster is an easy task and they provide special command-line tools like eksctl to create and manage EKS effortlessly

But Can we create EKS Clusters with eksctl commands for production? What if we want to automate it and customize it and add some IAC(infrastructure as code) standards to it.

When we talk about Infrastructure implementation and automation. Terraform is undeniably a leader and best for multi-cloud solutions and platform-agnostic solutions.

In this article, we are going to see how to create EKS Cluster with Terraform with built-in Karpenter auto-scaling capability

EKS Karpenter Example

So what is Karpenter

Karpenter is a cluster auto scaler that takes care of scaling up and down worker nodes in your cluster for AWS EKS. As of Feb 2022 only AWS is supported by Karpenter

It takes care of Compute provisioning for your EKS cluster and enables you to add on-demand and spot instances to your cluster.

If you have exposure to EKS world or any managed Kubernetes, you might have used another product named cluster autoscaler for the same purpose

While the cluster autoscaler had been solving the purpose and widely accepted. Karpenter comes with various best things to offer.

Here are some of the key items, I can say Karpenter is doing great

  • Creating the right capacity or the computing power based on the number of pods and resources they need. For example. If two pods are pending it can just create t3.medium or small instance automatically. if  twenty pods are pending it can choose the right to compute power like c5.2xlarge  - Cluster auto scaler cannot do this right.
  • Implementation of Karpenter is really easy and well documented - Cluster auto scaler falls behind a little.
  • Since it adds the right compute power for the job. Karpenter offers a lot of cost-saving compared to the Cluster auto scaler.

There are more I can list but these are the ones to be highlighted.


You need to install these tools and configure them before proceeding.

  1. AWS CLI
  2. kubectl - the Kubernetes CLI
  3. terraform - infrastructure-as-code tool made by HashiCorp
  4. helm - the package manager for Kubernetes

Log in to the AWS CLI with a user that has sufficient privileges to create a cluster.

Configuring Terraform Workspace

The steps given below are designed for mac/Linux users.  If you are using windows you can use either Windows Subsystem for Linux or git bash to be able to run these bash commands.

Set Environment variables

Before starting up, we need to set some environment variables like CLUSTER_NAME

export CLUSTER_NAME=gritfy-eks-karpenter

AWS CLI is powering our authentication needed for Terraform to access the AWS account and to create Terraform.

So you can set the following variables according to your requirement

export AWS_DEFAULT_REGION=us-east-2
export AWS_DEFAULT_PROFILE=personal

If you want to use the same default region configured during the aws configure this may not be needed.

The AWS_DEFAULT_PROFILE is to let terraform know which AWS CLI profile to use for authentication and to connect.  This is needed if you are having multiple AWS Named profiles


Create Terraform files

Create a new directory as a workspace and copy the following .tf files file

Create a new file named and copy the following content to the file

this is to define the variables, that are needed for the project. In case no value is provided, it would prompt during the runtime

variable "cluster_name" {
  description = "The name of the cluster"
  type        = string

we can put default values for the variables by using tfvars file but we are not doing that here.


The file

Create a new file named main.tfand copy the following content into that file.

It does three things.

  1. Creates a  vpc with subnets and the cluster-name would be used as a vpc name.
  2. Creating a Kubernetes cluster using eks module. A version of the cluster is defined as 1.21
  3. Instance profile creation with policy attachment. It creates a new instance profile in this format KarpenterNodeInstanceProfile-{cluster-name}
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = var.cluster_name
  cidr = ""

  azs             = ["us-east-2a", "us-east-2b", "us-east-2c"]
  private_subnets = ["", "", ""]
  public_subnets  = ["", "", ""]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  one_nat_gateway_per_az = false

  private_subnet_tags = {
    "${var.cluster_name}" = "owned"
    "" = var.cluster_name

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "<18"

  cluster_version = "1.21"
  cluster_name    = var.cluster_name
  vpc_id          = module.vpc.vpc_id
  subnets         = module.vpc.private_subnets
  enable_irsa     = true

  # Only need one node to get Karpenter up and running
  worker_groups = [
      instance_type = "t3a.medium"
      asg_max_size  = 1

  tags = {
    "" = var.cluster_name

# Instance Profile

data "aws_iam_policy" "ssm_managed_instance" {
  arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"

resource "aws_iam_role_policy_attachment" "karpenter_ssm_policy" {
  role       = module.eks.worker_iam_role_name
  policy_arn = data.aws_iam_policy.ssm_managed_instance.arn

resource "aws_iam_instance_profile" "karpenter" {
  name = "KarpenterNodeInstanceProfile-${var.cluster_name}"
  role = module.eks.worker_iam_role_name

The file

To configure the aws provider right and to define the region and the profile to be used.

provider "aws" {
  region = "us-east-2"
  profile = "personal"


Terraform init and apply - Creating EKS

Now we have all the files copied in place, now it's time to initialize terraform and apply our configuration.

At this point, we assume your directory looks like this

Karpenter-test|⇒ ls -rlt
total 40
-rw-r--r – 1 sarav staff 1497 Feb 6 20:03
-rw-r--r – 1 sarav staff 692 Feb 6 20:03
-rw-r--r – 1 sarav staff 1289 Feb 6 20:03
-rw-r--r – 1 sarav staff 92 Feb 6 20:03
-rw-r--r – 1 sarav staff 64 Feb 6 20:03

Run terraform init

Karpenter-test|⇒ terraform init
Initializing modules...
Downloading terraform-aws-modules/eks/aws 17.24.0 for eks...
- eks in .terraform/modules/eks
- eks.fargate in .terraform/modules/eks/modules/fargate
- eks.node_groups in .terraform/modules/eks/modules/node_groups
Downloading terraform-aws-modules/iam/aws 4.7.0 for iam_assumable_role_karpenter...
- iam_assumable_role_karpenter in .terraform/modules/iam_assumable_role_karpenter/modules/iam-assumable-role-with-oidc
Downloading terraform-aws-modules/vpc/aws 3.11.5 for vpc...
- vpc in .terraform/modules/vpc

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching ">= 1.11.1"...
- Finding hashicorp/cloudinit versions matching ">= 2.0.0"...
- Finding terraform-aws-modules/http versions matching ">= 2.4.1"...
- Finding latest version of hashicorp/helm...
- Finding hashicorp/aws versions matching ">= 2.23.0, >= 3.56.0, >= 3.63.0"...
- Finding hashicorp/local versions matching ">= 1.4.0"...
- Installing hashicorp/kubernetes v2.7.1...
- Installed hashicorp/kubernetes v2.7.1 (signed by HashiCorp)
- Installing hashicorp/cloudinit v2.2.0...
- Installed hashicorp/cloudinit v2.2.0 (signed by HashiCorp)
- Installing terraform-aws-modules/http v2.4.1...
- Installed terraform-aws-modules/http v2.4.1 (self-signed, key ID B2C1C0641B6B0EB7)
- Installing hashicorp/helm v2.4.1...
- Installed hashicorp/helm v2.4.1 (signed by HashiCorp)
- Installing hashicorp/aws v3.74.0...
- Installed hashicorp/aws v3.74.0 (signed by HashiCorp)
- Installing hashicorp/local v2.1.0...
- Installed hashicorp/local v2.1.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Run terraform apply with $CLUSTER_NAME

terraform apply -var cluster_name=$CLUSTER_NAME

You might probably get the following error

Terraform aws_auth error

This is because of authorization. you can resolve it by setting an environment variable.

The Kubeconfig file should have been created already on the current directory with the cluster name. kubeconfig_{cluster-name}

Terraform list

You need to update your KUBECONFIGvariable to refer to this file along with KUBE_CONFIG_PATHvariable.

⇒ export KUBECONFIG=${PWD}/kubeconfig_${CLUSTER_NAME}

After doing this. you can retry and it would be successful this time

Now we have the cluster ready. Now we need to install Karpenter in this EKS Cluster.

Before we can install the Karpenter using the helm chart, we need to do Create an IAM role to give necessary access to Karpenter for managing the nodes like launching nodes, terminating nodes etc.

Creating Karpenter Controller IAM Role

In the same workspace directory, create a new file named and paste the following config

This terraform config does the following things

  • Creating an AWS IAM Role
  • Attaching a policy
  • Authorizing the Service Account to assume the role - IRSA.( IAM Roles for Service Accounts)

We will create the ServiceAccount and connect it to this role during the Helm chart install.

Now we are creating an IAM role karpenter-policy-${var.cluster_name}its an assumable role will later be associated with EKS Service Account using IRSA principle

module "iam_assumable_role_karpenter" {
  source                        = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                       = "4.7.0"
  create_role                   = true
  role_name                     = "karpenter-controller-${var.cluster_name}"
  provider_url                  = module.eks.cluster_oidc_issuer_url
  oidc_fully_qualified_subjects = ["system:serviceaccount:karpenter:karpenter"]

resource "aws_iam_role_policy" "karpenter_controller" {
  name = "karpenter-policy-${var.cluster_name}"
  role = module.iam_assumable_role_karpenter.iam_role_name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
        Action = [
        Effect   = "Allow"
        Resource = "*"

Now we have a terraform config file for the Karpenter controller IAM role.  Let us add one more config file to take care of the karpenter installation using helm.

Installing Karpenter to EKS with Helm and Terraform.

using helm_release resource of terraform we are going to release/install karpenter to our cluster.

we are also configuring the following variable to let Karpenter know the IAM role that it can assume

For this variable, we will be passing the IAM role we have craeted in the config file



Now, Copy the following config and save it as in our workspace directory we have been using so far

resource "helm_release" "karpenter" {
  depends_on       = [module.eks.kubeconfig]
  namespace        = "karpenter"
  create_namespace = true

  name       = "karpenter"
  repository = ""
  chart      = "karpenter"
  version    = "v0.6.0"

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.iam_assumable_role_karpenter.iam_role_arn

  set {
    name  = "controller.clusterName"
    value = var.cluster_name

  set {
    name  = "controller.clusterEndpoint"
    value = module.eks.cluster_endpoint

  set {
    name  = "aws.defaultInstanceProfile"
    value =


so far we have added two files named and Now your directory should look something like this

You can see the new files on the top.

Enable Spot instances in your AWS Account ( If not used spot instances before )

Before we move on to apply these configuration. you need to enable Spot instance support if you have not used Spot instances in your AWS account before

aws iam create-service-linked-role – aws-service-name

If the role has already been successfully created, you will see:

An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: 
Service role name AWSServiceRoleForEC2Spot has been taken in this account, 
please try a different suffix.


Re-initialize Terraform and apply

Since we are adding two new modules with these two new configuration files. we need to reinitialize with terraform init on the same workspace

Note*: Re-initialize will not do any changes to your EKS Cluster which has been already created. so no worries

Now execute the following commands to initialize and apply

terraform init
terraform apply -var cluster_name=$CLUSTER_NAME


Upon Successful, To verify if the IAM role is created fine. you can use the following command. ( we are using jq command for filtering. you must have it installed before using it)

aws iam list-roles – profile personal|jq '.Roles[] | select(.RoleName == "karpenter-controller-gritfy-eks-karpenter")'

you would see some output like this

EKS Karpenter


Connect to your EKS using Kubectl and Validate Karpenter

Since our EKS cluster is ready and the IAM roles are ready.  You should be able to connect to our EKS cluster using kubectl.

since the KUBECONFIG variable is pointing to the current directory the kubectl would be able to use the context and connect to your EKS cluster with no issues

EKS Karpenter

Since you are able to connect to EKS cluster already. you should also be able to list the helm release and karpenter installed

use the following command to list the helm releases. It should show our karpetner helm release

helm list – all-namespaces

AWS Karpenter


Deploying the Provisioner

Provisioner is to instruct the karpenter what type of machines to launch like spot or on-demand etc.

Also to manage the Subnets and the security groups which are necessary for the karpenter to launch the nodes.

A single Karpenter provisioner is capable of handling many different pod shapes. Karpenter makes scheduling and provisioning decisions based on pod attributes such as labels and affinity. In other words, Karpenter eliminates the need to manage many different node groups.

We are going to create a default provisioner. This provisioner configures instances to connect to your cluster’s endpoint and discovers resources like subnets and security groups using the cluster’s name.

We are going with a simple configuration for our provisioner but you can control or extend your provisioner configuration further. refer provisioner api

cat <<EOF | kubectl apply -f -
kind: Provisioner
  name: default
    - key:
      operator: In
      values: ["spot"]
      cpu: 1000
    subnetSelector: ${CLUSTER_NAME}
    securityGroupSelector: ${CLUSTER_NAME}
  ttlSecondsAfterEmpty: 30

You can execute this command in your current terminal and it would create a new provisioner and name it default

As you can see in the configuration, we are using spot instances for our capacity while you can change it to on-demand or use both.

The limits instructs karpenter to create new nodes when the number of CPUs has reached 1000 this way you can set a max limit of nodes that can be created by Karpenter.

Another important element is the selectorthis helps the newly added karpenter nodes to choose the right subnetsand security groups

You need to handle this carefully if you are trying to set up Karpenter in your existing cluster. Sometimes instead of using selector by tags. you can simply go for defining the actual subnets and security groups yourself. ( its an option)

in our case, we are going to choose the subnets and security groups by the tag ${CLUSTER_NAME}

These tags are defined by us in the file for the VPC and EKS configuration. You can go back and refer to confirm.


Listing the Subnets and Security Groups by Tag - to Validate

You can try to mimic this behaviour and try to list security groups and subnets having this tag using AWS CLI

# To List Security Groups
aws ec2 describe-security-groups – query "SecurityGroups[? Tags[? (Key=='') && (Value=='$CLUSTER_NAME')]].GroupId"

# To list Subnets
aws ec2 describe-subnets – query "Subnets[? Tags[? (Key=='') && (Value=='$CLUSTER_NAME')]].SubnetId"

Here is the output of these commands at my end

AWS Karpenter Terraform


Now deploy an application to EKS Cluster

Till there are no deployments in our cluster. So let us deploy something.

we are going to deploy our famous aksarav/tomcat8 image

This image is covered in detail in our previous articles

Now let us deploy it.

This single line command would do the deployment. ( you can create a deployment manifest too and use kubectl apply -f)

kubectl create deployment tomcatinfra – image=saravak/tomcat8

Here is the quick terminal record

As you might have seen in the terminal capture, we deployed and tested the EKS Cluster and tried to scale the deployment and everything works fine.

Now it's a time to test Karpenter


Scale our Deployment and Test the Karpenter

Now it's time to validate, If Karpenter is able to add nodes and auto-scale in case we have pods in pending state and not scheduled due to capacity

Now we are going to scale our tomcatinfra deployment to higher levels to validate and see how Karpenter manages it

After a short time, we will scale it down to see if it brings down the node count

Here is the complete terminal record that shows the Karpenter doing its magic.

As shown in the terminal record. You can see the Karpenter logs adding and removing nodes based on demand and need.

EKS Karpenter Logs

You might find these kinds of logs during the even of Scaling up Nodes

2022-02-06T21:58:32.568Z INFO controller.provisioning Batched 39 pods in 1.761625115s {"commit": "62c4546", "provisioner": "default"}
2022-02-06T21:58:33.034Z INFO controller.provisioning Computed packing of 1 node(s) for 39 pod(s) with instance type option(s) [c4.xlarge c5.xlarge c5a.xlarge c6i.xlarge c5ad.xlarge c5d.xlarge c5n.xlarge m6i.xlarge m5.xlarge t3a.xlarge m5dn.xlarge t3.xlarge m5d.xlarge m4.xlarge m5zn.xlarge m5a.xlarge m5ad.xlarge m5n.xlarge i3.xlarge r4.xlarge] {"commit": "62c4546", "provisioner": "default"}
2022-02-06T21:58:35.834Z INFO controller.provisioning Launched instance: i-0ad3abd70a5de2d2b, hostname:, type: c5.xlarge, zone: us-east-2c, capacityType: spot {"commit": "62c4546", "provisioner": "default"}
2022-02-06T21:58:35.909Z INFO controller.provisioning Bound 39 pod(s) to node {"commit": "62c4546", "provisioner": "default"}

In case of  Scaling down due to no usage on the node. you can find these kind log messages

2022-02-06T22:03:11.685Z INFO controller.node Added TTL to empty node {"commit": "62c4546", "node": ""}
2022-02-06T22:03:41.699Z INFO controller.node Triggering termination after 30s for empty node {"commit": "62c4546", "node": ""}
2022-02-06T22:03:41.725Z INFO controller.termination Cordoned node {"commit": "62c4546", "node": ""}
2022-02-06T22:03:41.935Z INFO controller.termination Deleted node {"commit": "62c4546", "node": ""}



Hope this article covers Everything you need to know about the Karpenter Setup on EKS.

To Summarize this is what we did so far

  • Created EKS Cluster and VPC using Terraform
  • Instance Profile Creation and Policy Attachment
  • Created IAM Roles for Karpenter to Assume roles using IARC
  • Installed Karpenter with Helm release Terraform.
  • Created and Deployed a Provisioner
  • Deployed Sample Application to test EKS Cluster
  • Scaled the Sample application up and down to validate if Karpenter add and remove nodes.

So that's all about it.

Hope it helps. Let us know your feedback or queries on comments.


Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Buy Me a Coffee at

Signup for Exclusive "Subscriber-only" Content