Those who have been using Terraform might already know that you can create multiple numbers of the same resources with the help of count or for_each
But What if we want to create multiple resources with different configurations at the same time.
For example. we can easily create No of EC2 instances with the same set of configurations like AMI, Subnet, MachineType with the help of count
resource "aws_instance" "web" {
ami = "ami-007a18d38016a0f4e"
instance_type = "t3.medium"
count = 5
vpc_security_group_ids = [
"sg-0d8bdc716e7baee9f"
]
But what if we want multiple resources with a different set of configuration and each resource have to created in a specific amount ( 3, 4 etc)
If you think we can use count and for_each under the resource_block and try it. you would see the following error from Terraform.
The "count" and "for_each" meta-arguments are mutually-exclusive, only one should be used to be explicit about the number of resources to be created
Yes. you cannot use count and for_each together under the same resource block.
So how to do it is what We are going to see in this article.
How to use for-each and count together in Terraform
As we already know, The count and for_each are mutually exclusive but we need to use both of them to complete our requirement: creating multiple resources with different sets of configurations and counts.
So. How to get this done.
Without using Count and for_each together we are going to achieve the same result that we are trying to achieve with the help of the following functions/expression
- for - A loop expression to traverse and create dynamic blocks to match the number of instances with dynamic names for each instance
- range - used inside for expression to create a dynamic range according to the number set for the
countlet's say no of instances - flatten - to convert the nested list to a single list
Before we go deep and see how to use each of these expressions to get our requirement completed.
You can download the code from Github and get ready.
Download the Code to begin with
Now it's time to go practical. Though we will be discussing each segment of the code in detail. To begin with. you can download/clone the code from this repository
git clone https://github.com/AKSarav/Terraform-Count-ForEach
A quick look at the files - Decoding the configuration
After you have downloaded the files or cloned the Git repo. You can see three files
- main.tf - the main Terraform configuration files
- variables.tf - For declaring and initializing the variables
- dev.tfvars - A File with variables filled with values.
Let us learn more about each files in detail
We can start with the variables.tf file which has a minimal content.
variables.tf
the variables.tf file is used to declare the list of variables that we might be using on the terraform code.
If the variable is defined/declared here and not provided with a value. Terraform would throw an error.
If the variables is not declared but used on the Terraform configuration. It would throw an error and ask you to declare it.
So this is basically a way to isolate and document all the variables necessary for the terraform module or configuration to work.
In our case, we are declaring only one variable named configuration
variable "configuration" {
description = "The total configuration, List of Objects/Dictionary"
default = [{}]
}
dev.tfvars - A Snapshot of our Infra we are going to create
this tfvars file is where we provide actual values for the variables we declare.
we can have different tfvars file and launch our terraform with customizations every time we launch using --var-file attribute
to be precise. you can have different tfvars file for each environment with different sets of values and name them as dev.tfvars, qa.tfvars, prod.tfvars, dr.tfvars etc to cover different environments like dev,QA, prod,dr accordingly
In our case, we are having a single tfvars file named dev.tfvars with the following content
configuration = [
{
"application_name" : "GritfyApp-dev",
"ami" : "ami-09e67e426f25ce0d7",
"no_of_instances" : "2",
"instance_type" : "t2.medium",
"subnet_id" : "subnet-0f4f294d8404946eb",
"vpc_security_group_ids" : ["sg-0d15a4cac0567478c","sg-0d8749c35f7439f3e"]
},
{
"application_name" : "GrityWeb-dev",
"ami" : "ami-0747bdcabd34c712a",
"instance_type" : "t2.micro",
"no_of_instances" : "1"
"subnet_id" : "subnet-0f4f294d8404946eb"
"vpc_security_group_ids" : ["sg-0d15a4cac0567478c"]
},
{
"application_name" : "OpsGrit-dev",
"ami" : "ami-0747bdcabd34c712a",
"instance_type" : "t3.micro",
"no_of_instances" : "3"
"subnet_id" : "subnet-0f4f294d8404946eb"
"vpc_security_group_ids" : ["sg-0d15a4cac0567478c"]
}
]
As you can see, we have filled the variable named configuration we have declared earlier on the variables.tf file.
we are going to create 6 EC2 instances for three different applications. thanks to count
I have compiled it as a table for you to grasp it real quick.
| Application Name | AMI ID | No Of Instances(Count) | Instance type | subnet_id | security_group_ids |
| GritfyApp | ami-09e67e426f25ce0d7 | 2 | t2.medium | subnet-0f4f294d8404946eb | sg-0d15a4cac0567478c,sg-0d8749c35f7439f3e |
| GritfyWeb | ami-0747bdcabd34c712a | 1 | t2.micro | subnet-0f4f294d8404946eb | sg-0d15a4cac0567478c |
| OpsGrit | ami-0747bdcabd34c712a | 3 | t3.micro | subnet-0f4f294d8404946eb | sg-0d15a4cac0567478c |
As you can see each application has different configurations in terms of count, AMI and instance type and security group
main.tf - The Configuration that does the magic
This is the main configuration file of terraform with all the logic and looping expressions we have talked earlier about.
Let us see what each block on this file is doing, in detail.
provider "aws" {
region = "us-east-1"
profile = "personal"
}
locals {
serverconfig = [
for srv in var.configuration : [
for i in range(1, srv.no_of_instances+1) : {
instance_name = "${srv.application_name}-${i}"
instance_type = srv.instance_type
subnet_id = srv.subnet_id
ami = srv.ami
security_groups = srv.vpc_security_group_ids
}
]
]
}
// We need to Flatten it before using it
locals {
instances = flatten(local.serverconfig)
}
resource "aws_instance" "web" {
for_each = {for server in local.instances: server.instance_name => server}
ami = each.value.ami
instance_type = each.value.instance_type
vpc_security_group_ids = each.value.security_groups
user_data = <<EOF
#!/bin/bash
echo "Copying the SSH Key to the remote server"
echo -e "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvhXuMn9FwsrcK/DkgOlZdQFbY9e0+InX2sdHm8ZF7hGOQvg3CTMdBtMHlALnzqsYlS0aN0puzNF7fWAvUawdGjcSYxKEMlO1CaKPYxEgLTPDdiuYm3DNUutNMOLB0KHSJDk1Vb83UEpXm4vZjAWwHQTgoSsyXA57GcV4+IiTOy+iIIiiB7XzTDjt7ePVOW237HJAENlB/txh0qEl4Gn0eNGykg2E00jN8cOfIf/sKuY2kXBRgSjTjr6HArB4an6+aJpNJMWFFLyk47+NOIepaZhJNuXL39y0kGp/KzTlQw45g+ct92CSoCvySGqSUGN85ofPeYfzwB45yVJ9bMrZpY88TG4kLGAFeAg4DHVxUmJQhbjQOBRL8FDadOZuHmawlBUNeqFFtQ1EAad9Z2FWAZ80htaPysE9coA2VXC559VapIs9fsx2nPStKoB8bPP91rArS4Q9tt077+BgPE3d4IK2GRTYsC1TXzrF6hvGGk9zk+nWpZMqDtW5sQxdxl0k=" >> /home/ubuntu/.ssh/authorized_keys
echo "Changing the hostname to ${each.value.instance_name}"
hostname ${each.value.instance_name}
echo "${each.value.instance_name}" > /etc/hostname
EOF
subnet_id = each.value.subnet_id
tags = {
Name = "${each.value.instance_name}"
}
}
output "instances" {
value = "${aws_instance.web}"
description = "All Machine details"
}
Provider block
the provider block is to tell terraform what Provider to use like AWS GCP etc and additional information to login to the provider API
in our case we are using the AWS CLI Named Profiles.
My profile name is personal and the region I have chosen in us-east-1
provider "aws" {
region = "us-east-1"
profile = "personal"
}
locals block
As discussed earlier, we cannot directly consume our variable configuration using for_each if we do that, Terraform would not let us use count on the same block because both are for loop expressions therefore mutually exclusive to each other.
Having said that. Now we need to find a way to generate repeated resource blocks somewhere outside but not in the aws_instance resource block.
thats what we are going to do in the locals block.
locals {
serverconfig = [
for srv in var.configuration : [
for i in range(1, srv.no_of_instances+1) : {
instance_name = "${srv.application_name}-${i}"
instance_type = srv.instance_type
subnet_id = srv.subnet_id
ami = srv.ami
security_groups = srv.vpc_security_group_ids
}
]
]
}
// We need to Flatten it before using it
locals {
instances = flatten(local.serverconfig)
}
We have two local variables named serverconfig and instances
- serverconfig - is to create a dynamic block of server configuration, in a list of objects
[[{}],[{}],[{}]format - instances - to turn multiple nested lists into a single flat list with flatten
[{},{},{}]
I know it must be a little hard to understand but thanks to terraform console which can help us debug what these variables look like during the runtime.
Refer the following Quick video to understand the actual runtime values of these variables.
resource block
Now the resource block is pretty much typical. except we have a for_each loop set based on the local.instances variable we have created using locals block.
As you know the instances block contains the list of objects [{},{},{}] that represents the configuration of different instances we are going to create.
the key logic part is done by this statement
for_each = {for server in local.instances: server.instance_name => server}
we are passing the for expression's output into for_each
Now to clarify what is the output of our for expression. Let us use terraform console once again
and paste the for expression along with the parenthesis
As presented in the image above. you can see that the for expression is creating a list of objects with instance_name added as a key
{
instance_name-[0-9] = {
},
instance_name-[0-9] = {
}
}
So basically it is creating dynamic blocks of instance configuration with customized configurations like ami, name, instance type etc
Validation and Running the Configuration
If you have just downloaded the code and cloned the repo. you might have to initialize the terraform using the following command
⇒ terraform init
now it's a time that we validate if we have made any syntax issues on our configuration file using the following command
⇒ terraform validate Success! The configuration is valid.
After the validation is successful we can go ahead and create a plan document
Since we have a custom tfvars file we need to use the -var-file during the plan and apply
⇒ terraform plan -var-file=dev.tfvars -out devtfplan.out
Once you are satisfied with the plan
You can apply the changes using the following command and make sure you use the right out file. in our case its devtfplan.out
⇒ terraform apply "devtfplan.out"
Sit back and relax. while the Terraform creates all the instances
Here is the Screen record of Terraform plan and apply execution from my end.
Validate the EC2 instances are Launched
Now head to AWS Management Console and you can see that the EC2 instances are created and waiting for you
You can log in to the machines directly as you have copied your SSH to authorized_keys as part of user_data
Conclusion
As part of this article, we have covered various sub scenarios like
- Create multiple EC2 instances with different sets of configuration
- How to use Count and For_each together
- Debug variables in Terraform and know their values at runtime with Terraform console
- How to process a List of dictionaries in terraform using For and For_each.
- How to use For and For_each on Terraform resources.
Not just for EC2. you can tweak this solution and try it for other AWS resources too.
Hope this helps. If you have any feedback or a better way to do this. Please share it with us in the comments section.
Are you looking for Digital Transformation Partner or DevOps as a Service solution. Try us
Cheers
Sarav AK
Follow me on Linkedin My Profile Follow DevopsJunction onFacebook orTwitter For more practical videos and tutorials. Subscribe to our channel
Signup for Exclusive "Subscriber-only" Content
