Backup RDS database snapshots to S3

Relation Database Service (RDS) is a managed database option provided by AWS. It is a great option for those who want to use a relational database without having to manage the underlying infrastructure. RDS provides a number of database engines to choose from including MySQL, PostgreSQL, Oracle, and SQL Server. RDS also provides a number of features including automated backups, point-in-time recovery, and read replicas.

When we talk about backups in RDS, we are talking about two different types of backups. The first is automated backups. Automated backups are enabled during the creation (optionally) and are stored in S3. Users can configure the retention period for automated backups and the window in which the backup occurs.

The second type of backup is a manual snapshot. Manual snapshots are user-initiated and are also stored in S3. Both types of backups are stored in S3 and are accessible via the AWS console or the AWS CLI.

In this article, we will look at how to back up RDS databases to S3 using Boto. Boto is the AWS SDK for Python. It allows you to write Python scripts to interact with AWS resources. We will use Boto to connect to AWS and create a database snapshot. We will then copy the snapshot to S3 and delete the snapshot.

In other words, we are going to export the Snapshot to S3.

Table of Contents

RDS Backup to S3 - An opensource tool/script

I have created a simple tool that will allow you to export the snapshot to S3. The tool is written in Python and uses the Boto SDK to connect to AWS.

The script does the following tasks

Connects to AWS using Boto
Creates a snapshot of the database
Copies the snapshot to S3
Deletes the snapshot once the export is completed

How to Download and Use this Tool

Clone git repository

 git clone https://github.com/AKSarav/RDSbackupS3.git

Create a virtual environment
```
 python3 -m venv venv
```
Install the requirements using the following command
```
 pip install -r requirements.txt
```

Run the script

 python3 rds_backup_to_s3.py [--instance INSTANCE] [--region REGION] [--bucket BUCKET]
[--prefix PREFIX] [--role ROLE] [--kms KMS]

Check the S3 bucket for the snapshot

Prerequisites

The script expects the following arguments.

--instance RDS instance name to backup*
--region AWS region of RDS instance
--bucket S3 bucket name to copy the snapshot to
--prefix S3 prefix ( folder ) to copy the snapshot to
--role IAM role to use to copy the snapshot to S3
--kms key to use to copy the snapshot to S3

Source code

For your convenience, I have pasted the source code of this script below you can quickly copy and use it

But I do insist you clone the repository to get the latest version and not miss any future updates.

import boto3
import datetime
import sys
import time
import argparse


# get the current date and time
now = datetime.datetime.now()

# set the region and create the RDS client
rds = boto3.client('rds', region_name='us-east-1')

if __name__ == '__main__':

    # argparse get inputs
    parser = argparse.ArgumentParser(description='Backup RDS instance')
    parser.add_argument('--instance', help='RDS instance name')
    parser.add_argument('--region', help='AWS region')
    parser.add_argument('--bucket', help='S3 bucket name')
    parser.add_argument('--prefix', help='S3 prefix')
    parser.add_argument('--role', help='IAM role')
    parser.add_argument('--kms', help='KMS key')
    args = parser.parse_args()

    # if args are not passed, throw error
    if None in (args.instance, args.region, args.bucket, args.prefix, args.role, args.kms):
        print('Usage: python backup-rds.py [instance name] [region] [bucket] [prefix] [role] [kms]')
        sys.exit(1)

    # Display all the values given
    print("-"*50)
    print("RDSbackupS3 | Version: 0.0.1 | Github: ")
    print("-"*50)
    for arg in vars(args):
        print("{:<10}: '{}'".format(arg, getattr(args, arg)))
    print("-"*50)


    instance_name = args.instance

    # get the instance id from the instance name
    response = rds.describe_db_instances(DBInstanceIdentifier=instance_name)

    if not response['DBInstances']:
        print('No instance found with name: ' + instance_name)
        sys.exit(1)
        
    instance_id = response['DBInstances'][0]['DBInstanceIdentifier']

    # create the snapshot
    year=str(now.year)
    month=str(now.month)
    day=str(now.day)

    snapshot_name = instance_name + '-manual-' + now.strftime('%Y-%m-%d-%H-%M')
    print('Creating snapshot: ' + snapshot_name)
    response = rds.create_db_snapshot(
        DBSnapshotIdentifier=snapshot_name,
        DBInstanceIdentifier=instance_id,
        Tags=[
            {
                'Key': "BackupDateTime",
                'Value': now.strftime('%Y-%m-%d-%H-%M'),
            },
            {
                'Key': "BackupType",
                'Value': "Manual",
            },
        ]
    )

    if response['ResponseMetadata']['HTTPStatusCode'] != 200:
        print('Error creating snapshot')
        sys.exit(1)

    # print the snapshot name
    print("Snapshot Initiated successfully: "+snapshot_name)  

    # wait for the snapshot to be created
    waiter = rds.get_waiter('db_snapshot_completed')  

    print('Waiting for snapshot to complete')
    waiter.wait(
        DBSnapshotIdentifier=snapshot_name,
        WaiterConfig={
            'Delay': 30,
            'MaxAttempts': 60
        }
    )

    # print the snapshot status
    response = rds.describe_db_snapshots(DBSnapshotIdentifier=snapshot_name)
    snapshot_status = response['DBSnapshots'][0]['Status']
    print('Snapshot status: ' + snapshot_status)

    # if the snapshot fails, print the status message
    if snapshot_status == 'failed':
        print('Snapshot failed: ' + response['DBSnapshots'][0]['StatusMessage'])
        sys.exit(1)

    # if the snapshot succeeds, print the arn
    print('Snapshot created with arn: ' + response['DBSnapshots'][0]['DBSnapshotArn'])


    # Export the Snapshots to S3
    export_response=rds.start_export_task(
        ExportTaskIdentifier='export-'+snapshot_name,
        SourceArn=response['DBSnapshots'][0]['DBSnapshotArn'],
        S3BucketName=args.bucket,
        IamRoleArn=args.role,
        KmsKeyId=args.kms,
        S3Prefix=args.prefix,
    )

    # print the export status
    response = rds.describe_export_tasks(ExportTaskIdentifier='export-'+snapshot_name)
    export_status = response['ExportTasks'][0]['Status']
    print('Export status: ' + export_status)

    export_status = export_status.lower()

    # wait until the export status changes to 'completed'
    while export_status == 'in_progress' or export_status == 'starting':
        i = 0
        response = rds.describe_export_tasks(ExportTaskIdentifier='export-'+snapshot_name)
        export_status = response['ExportTasks'][0]['Status']
        print('Export status: ' + export_status)
        time.sleep(30)
        i = i + 1

        # Lowercase the export status for comparison
        export_status = export_status.lower()

        # break if no of attempts are more than 30 - 15 minutes
        if i > 30:
            break
        if export_status == 'failed' or export_status == 'failed' or export_status == 'canceled':
            break


    if export_status == 'failed':
        print('Export failed: ' + str(response))
        sys.exit(1)
    
    elif export_status == 'canceled':
        print('Export canceled: ' + str(response))
        sys.exit(1)

    elif export_status == 'complete':
        print('Export completed Successfully: ' + str(response))

    else:
        print('Unhandled Export status: ' + export_status)
        print('Response: ' + str(response))
        print('Not removing the snapshot as the export status is not complete')
        sys.exit(1)

    # Delete the snapshot
    print('Deleting snapshot: ' + snapshot_name)
    response = rds.delete_db_snapshot(
        DBSnapshotIdentifier=snapshot_name
    )

    if response['ResponseMetadata']['HTTPStatusCode'] != 200:
        print('Error deleting snapshot')
        sys.exit(1)

    # print the snapshot name
    print("Snapshot deleted successfully: "+snapshot_name)

Sample Output

Here is the execution and the output of this script. Some confidential information are obfuscated

# python3 backup-rds.py  – instance mysqlproddb – region us-east-1 – bucket gritfy-backup-drive – prefix databases/mysql/2023/06/18 – kms arn:aws:kms:us-east-1:XXXXXXXXXXX:key/zzzzzzz-1313-4091-xxxx-z18c81001zz – role arn:aws:iam::XXXXXXXXXXX:role/rds-s3-export-support-role
--------------------------------------------------
RDSbackupS3 | Version: 0.0.1 | Github: https://github.com/AKSarav/RDSbackupS3
--------------------------------------------------
instance  : 'mysqlproddb'
region    : 'us-east-1'
bucket    : 'gritfy-backup-drive'
prefix    : 'databases/mysql/2023/06/18'
role      : 'arn:aws:iam::XXXXXXXXXXX:role/rds-s3-export-support-role'
kms       : 'arn:aws:kms:us-east-1:XXXXXXXXXXX:key/zzzzzzz-1313-4091-xxxx-z18c81001zz'
--------------------------------------------------
Creating snapshot: mysqlproddb-manual-2023-06-18-21-00
Snapshot Initiated successfully: mysqlproddb-manual-2023-06-18-21-00
Waiting for snapshot to complete
Snapshot status: available
Snapshot created with arn: arn:aws:rds:us-east-1:XXXXXXXXXXX:snapshot:mysqlproddb-manual-2023-06-18-21-00
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: INPROGRESS
Export status: COMPLETE
Export completed Successfully:
Deleting snapshot: mysqlproddb-manual-2023-06-18-21-00
Snapshot deleted successfully: mysqlproddb-prime-manual-2023-06-18-21-00

Conclusion

Hope this tool helps you to export your RDS snapshot to S3. If you have any questions or comments please let us know in the comments section

Further References

Cheers
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Signup for Exclusive "Subscriber-only" Content