Secret Management is an essential part of infrastructure and DevOps. how do you store your Secrets and Environment variables define how safe your infrastructure is
As a good engineering practice, most companies guide their developers to not hard code the secrets and environment variable directly but out of convenience or necessity Engineers tend to commit the secrets/environments to the repositories
It is good to stop them from doing so and employ some constraints and code quality checks to prevent these unintended data from getting committed or entering into the code base.
Another simple efficient way is to encrypt the sensitive data and decrypt it when needed.
Yes. This is a simple ancient encryption and decryption am talking about but not in the ancient ways of creating your own system and keys and sharing the keys amongst the engineers.
I know the management of that KEY itself is going to be tedious and sharing the key between teams and ensuring it's not falling into the wrong hands is another nightmare
If the key gets to the middleman during the sharing or if one of the engineers is compromised then there is no point of security, is not it?
So let's see how to do this ancient technique of encryption/decryption in a modern way
In this article, we are going to discuss an efficient way of encrypting and decrypting files with native AWS KMS keys and IAM
Before we move further, we need to learn about an open-source product we are going to use for these secret operations and it is named
Sops is indeed a short form of Secret Operations.
What is SOPS and how can we use it
SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with the following key mechanisms
- AWS KMS
- GCP KMS
- Azure Key Vault
- PGP. (demo)
Not just these files SOPS can encrypt various other formats also.
When you are using SOPS to encrypt the YAML, JSON, ENV and INI formats - SOPS smartly can encrypt only the values.
In the following diagram, we used the AWS KMS key to encrypt a file
test.yaml with some username and password
You can see the output is encrypted with key-related information after executing
SOPS -e command.
If you look closely, you can see the keys are untouched only values are encrypted.
With a Single command I was able to encrypt these files in an instant and using the AWS KMS key ensures that only selected IAM users can leverage this KEY
sops -e – in-place – kms "arn:aws:kms:us-east-1:<ACCOUNT ID>:key/mrk-KEYFILENAME" test.yaml
AWS KMS and IAM with SOPS
SOPS can use various types of secret encryption mechanisms like PGP, AGE etc where we need to create a Key ourselves and retain it, store it, and share it to be able to decrypt the encrypted files.
Tools such as git-crypt have the same problem - where you have to manage the keys securely.
If the key is lost or tampered the security of the encrypted files would be compromised
Using KMS and IAM brings the perfect balance of Authorization and Authentication in using SOPS at your Organization
With IAM - you ensure only the intended engineers are onboarded to the AWS infrastructure as well and you also define who can use this KMS Key to encrypt and decrypt the files
With KMS Key - You ensure there is no Silos involved and a Single of point of failure such as the Key getting lost or tampered.
As you see in the preceding diagram, When you are using SOPS with AWS KMS two actions happen simultaneously
- With your IAM permission, the KMS key is being fetched
- Once the KMS Key is retrieved, SOPS use it to perform the encryption and/or decryption
Encrypt and Decrypt files with SOPS - using AWS KMS
Now we have clarity on how we are going to use AWS KMS with SOPS, These are the steps to enable SOPS with AWS KMS encryption
Tasks at the AWS console - require admin or elevated privileges
The following steps have to be done on the aws console. To perform these steps you might need admin or elevated privileges
- Create an AWS account with the following IAM permissions
Encrypt, Decrypt, ReEncrypt, GenerateDataKey, DescribeKey
- Get the Access Key ID and Secret of the user created - needed to configure AWS CLI
- Now install and configure AWS CLI with the access key and secret - we got from the previous step ( the values given below are just placeholder)
➜ ~ aws configure AWS Access Key ID [None]: AKIAS790KQGK63WUK6T5 AWS Secret Access Key [None]: kkQEiBjJSKrDkWBLO9G/JJKQWIOKL/CpHjMGyoiJWW Default region name [None]: us-east-1 Default output format [None]:
- Create a
SymmetricKMS key - for Encryption and Decryption
- Add Key administrators and key users - administrators have full access to keys so keep them limited and users are those who can use the key for encryption and decryption
- Add the user created in Step 1, either as a key administrator or as a key user - based on the requirement. in most cases key user is sufficient
- Complete the KMS key creation.
Installation of SOPS in your local PC
Now we have done all the changes required on the AWS front and now it is time to put them into use.
Our first requirement is to install the SOPS on our local PC
Choose the rightful binaries for your PC and install it - from this GitHub link
Using SOPS with KMS - Encryption and Decryption example
With the AWS CLI installed and configured, you will be able to use the SOPS to encrypt and decrypt the files
Here is the test Yaml file we have created for testing.
~|⇒ cat test.yml username: sarav password: secretpassword
Now let us run the encryption command with the KMS key ARN as follows (you need to replace the ARN with yours)
sops -e – in-place – kms "arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KEY-ID>" test.yml
Now the file is encrypted, To validate the file is encrypted and it has used the KMS key we provided just open the file and you will see some content like this
~|⇒ cat test.yml username: ENC[AES256_GCM,data:yTwd8V4=,iv:CDUoeilD7K/oLwOHYYxotRM5NaoHb0ROuHoH57bEBlc=,tag:maP1uemDnm4PB3S/nf0jog==,type:str] password: ENC[AES256_GCM,data:YLk9dYVzBxOjI5ieU30=,iv:l1KoEH02JnGAHeLEkaE6JLP3DVfVJ6/7Uou/SOML2fQ=,tag:/l5YUekg7tdmSIHuAdnJPg==,type:str] sops: kms: - arn: arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KMS-KEY-ID> created_at: "2023-09-10T11:35:49Z" enc: AQICAHhsV+xo/bxJefOk1Gz+1QoGeaNwiYhZtac4EKmVVKaU3AE/ZSl3S6n50iIAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMIJVfLxyEWr0Qf3KtAgEQgDuNpkxS1PDpe9DlsFtaHVs7LJ5sKiYu5FIjaQITtA/xqgACKl55nwlDviBbs/6fxiTWldCl939oVHWuQA== aws_profile: "" gcp_kms:  azure_kv:  hc_vault:  age:  lastmodified: "2023-09-10T11:35:50Z" mac: ENC[AES256_GCM,data:Yo1C7bo/roL+X925EHKbNSS5OW/M9X6FEp4Lt7YtlrV7L0rt9k8YhqqnmecvQN0BquSqA4pPZhT0Rt3JaGiVjjD9yKgKN7LbGVAfd8mE2EUdnJL+Y9cNXeXLx/bAUCq3/jgnPAXbLhpjjS8fwbM3z1nx+uZfdLFA8=,iv:ppxgtxNdYQ7uNHkV4dWMCqAgmO0R+DPdHUkUoulmHGs=,tag:C/WR1zicupSPMpwbLyU65A==,type:str] pgp:  unencrypted_suffix: _unencrypted version: 3.7.3
You can see which mechanism was used to encrypt the file along with the key and hash details
Now you can decrypt the file to view the encrypted content using the
sops -d command
sops -d – in-place – kms "arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KMS-KEY-ID>" test.yml
Here are all the commands and their outputs as executed on my end
That's how SOPS can be used with AWS KMS for securing the files - encryption and decryption.
Signup for Exclusive "Subscriber-only" Content