Automating Amazon RDS Database Backups to S3 Bucket Using Bash Script

Introduction

In the world of cloud computing, automating routine tasks is essential to optimize workflows and ensure consistency. Amazon Web Services (AWS) offers a wide range of services, and one common requirement is to automate the process of creating backups of Amazon RDS (Relational Database Service) databases, storing them on S3, and maintaining a clean backup retention policy.

In this article, we’ll explore a comprehensive bash script that achieves this automation.

The Problem Statement

Creating backups of your critical data is a fundamental practice to safeguard against accidental data loss. When dealing with Amazon RDS instances, managing backups and retaining only the necessary ones can become a complex task. This script aims to simplify the process by:

1. Creating a snapshot of the original RDS instance.
2. Restoring the snapshot to a new RDS instance.
3. Backing up the database on the new instance and saving it to an S3 bucket.
4. Deleting old backups from the S3 bucket to manage storage costs and space efficiently.
5. Deleting the new RDS instance and the snapshot after successful backup.

Assumptions

1. You have AWS Account A and Account B
2. The script will be run on a server in Account A and mysql-client is installed on the server
3. The RDS Engine is MySQL

Prerequisites

1. Basic knowledge of Amazon Web Services (AWS).
2. An AWS account with proper permissions to interact with RDS, S3, and Secrets Manager.
3. An understanding of Bash scripting.
4. AWS CLI and SDK Installation - Make sure you have the AWS CLI and AWS SDK for Bash (awscli and aws-sdk) installed and properly configured on the machine where you plan to run the script.
5. Amazon RDS Database
6. The database credentials stored in Secrete Manager — In each account, ensure that you store the necessary credentials (username and password) for accessing the RDS instances. Each credentials should be saved in the secrete manager using the identifier name as the secrete name. The script will use these credentials to authenticate and perform backup operations.
7. S3 Bucket

The Bash Script: Overview

The provided bash script automates the entire backup and cleanup process for multiple RDS instances. It leverages AWS CLI commands, AWS Secrets Manager, and Amazon S3. Let’s break down the key components and steps of the script:

1. Set up the AWS CLI configuration for the desired region.
2. Define functions to retrieve secret values from AWS Secrets Manager and delete old backups from an S3 bucket.
3. Iterate through a list of RDS instance identifiers, retrieving their credentials from Secrets Manager.
4. Create a snapshot of the RDS instance and restore it to a new instance.
5. Take a backup of the database on the new instance and save it to the specified S3 bucket.
6. Delete old backups from the S3 bucket to keep only the latest backup.
7. Delete the new RDS instance and the snapshot after successful backup and cleanup.

Permissions

Ensure that the AWS IAM user or role used to run the script has the necessary permissions to perform the following actions:

• secretsmanager:GetSecretValue: To retrieve the RDS credentials from AWS Secrets Manager.
• rds:CreateDBSnapshot: To create a snapshot of the RDS instance.
• rds:RestoreDBInstanceFromDBSnapshot: To restore the snapshot to a new RDS instance.
• rds:DescribeDBInstances: To get information about RDS instances, including endpoint details.
• rds:AddTagsToResource: To tag the newly created RDS instance for identification.
• rds:DeleteDBInstance: To delete the new RDS instance after the backup is complete.
• rds:DeleteDBSnapshot: To delete the RDS snapshot after the backup is complete.
• s3:ListBucket and s3:DeleteObject: To list and delete objects in the specified S3 bucket.
• s3:PutObject: To upload the database backup file to the S3 bucket.

For the user/role used on the server (Account A) running the bash script , you need to have these permissions attached to that user/role.

Create a policy with these permissions for example rds_backup_policy and attach it to that role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": [
            "arn:aws:iam::account_number:role/role_name"
         ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "rds:CreateDBSnapshot",
                "rds:DeleteDBSnapshot",
                "rds:DescribeDBSnapshots",
                "rds:RestoreDBInstanceFromDBSnapshot",
                "rds:DeleteDBInstance",
                "rds:DescribeDBInstances",
                "rds:AddTagsToResource"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bucket_name/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::bucket_name"
        },
        {
            "Effect": "Allow",
            "Action": "s3:DeleteObject",
            "Resource": "arn:aws:s3:::bucket_name/*"
        },
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": [
                "secret_arn"
            ]
        }
    ]
}

On the other Account B, create a role for example rds_backup_role and make sure you give Account A Assume role as the Trust relationships. Also you need to create a policy and attached it to that role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "rds:CreateDBSnapshot",
                "rds:DeleteDBSnapshot",
                "rds:DescribeDBSnapshots",
                "rds:RestoreDBInstanceFromDBSnapshot",
                "rds:DeleteDBInstance",
                "rds:DescribeDBInstances",
                "rds:AddTagsToResource"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bucket_name/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::bucket_name"
        },
        {
            "Effect": "Allow",
            "Action": "s3:DeleteObject",
            "Resource": "arn:aws:s3:::bucket_name/*"
        },
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": [
                "secret_arn"
            ]
        }
    ]
}

Detailed Explanation of the Script Sections:

Setting AWS CLI Configuration: The script starts by configuring the AWS CLI with the desired region (in this case, ap-southeast-2). Modify this region as needed for your use case.

# Set AWS CLI configuration
aws configure set region ap-southeast-2

Functions for Secrets Retrieval and Cleanup: The get_secret_value function retrieves secret values from AWS Secrets Manager using the provided secret name. The delete_old_backups function deletes old backup files from the specified S3 bucket, ensuring only the latest backup remains.

# Function to get secret value from AWS Secrets Manager
get_secret_value() {
    secret_name="$1"
    aws secretsmanager get-secret-value --secret-id "$secret_name" --query 'SecretString' --output text
}

# Function to delete old backups from S3 bucket
delete_old_backups() {
    local db_identifier="$1"
    local bucket_name="$2"

    echo "Deleting old backups from S3 bucket: $bucket_name"
    aws s3 ls "s3://$bucket_name/" --recursive | grep "backup-$db_identifier" | sort | head -n -1 | while read -r line; do
        file_name=$(echo "$line" | awk '{print $4}')
        aws s3 rm "s3://$bucket_name/$file_name"
    done

    echo "Old backups cleanup completed for S3 bucket: $bucket_name"
}

RDS Instance Identifiers and S3 Bucket Name: Replace the db_identifiers array with the identifiers of your RDS instances and set your_bucket_name to the desired S3 bucket name.

# List of RDS instance identifiers
declare -a db_identifiers=("rds-db-01" "rds-db-02")

your_bucket_name="niyialimi-rds-test"

Temporary IAM Role Credentials and Assuming Cross-Account IAM Role: The script also takes into consideration if you have your RDS in a different account. Temporary IAM role credentials are assumed using the aws sts assume-role command and exported as environment variables for the AWS CLI to use in subsequent commands.

# Formulate the role ARN to assume role.
account_number="$1"
role_name="$2"
cross_account_role_arn="arn:aws:iam::$account_number:role/$role_name"

# Assume the IAM Role
echo "Assuming IAM Role"
credentials=$(aws sts assume-role --role-arn "$cross_account_role_arn" --role-session-name "CrossAccountSession")

# Set the assumed IAM role credentials for AWS CLI commands
export AWS_ACCESS_KEY_ID=$(echo "$credentials" | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo "$credentials" | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo "$credentials" | jq -r '.Credentials.SessionToken')

Iterating Through RDS Instances: The script iterates through each RDS instance in the db_identifiers array and performs the following actions:

• Retrieves the RDS credentials from AWS Secrets Manager
• Takes a snapshot of the RDS instance.
• Restores the snapshot to a new RDS instance.
• Retrieves the RDS endpoint of the new instance.
• Takes a database backup and saves it to the S3 bucket.

Note that for each tasks, the script tries 2 times for each failure.

# Maximum number of retries for certain AWS CLI commands
max_retries=2

# Loop through the array of RDS instances
for db_identifier in "${db_identifiers[@]}"; do

    echo "Working on RDS instance: $db_identifier"

    timestamp=$(date +%Y%m%d%H%M%S)

    # Construct the AWS Secret Manager secret name
    secret_name="$db_identifier"

    # Get the RDS credentials from AWS Secrets Manager
    retries=0
    while [[ $retries -lt $max_retries ]]; do
        secret_string=$(get_secret_value "$secret_name")
        if [[ $? -eq 0 ]]; then
            break
        fi
        retries=$((retries + 1))
        sleep 5
    done

    if [[ $retries -eq $max_retries ]]; then
        echo "Error: Failed to get secret value for RDS instance: $db_identifier"
        continue
    fi

    db_username=$(echo "$secret_string" | jq -r '.username')
    db_password=$(echo "$secret_string" | jq -r '.password')

    # Take a snapshot of the RDS instance
    retries=0
    while [[ $retries -lt $max_retries ]]; do
        snapshot_id="backup-$db_identifier-$timestamp"
        aws rds create-db-snapshot --db-instance-identifier "$db_identifier" --db-snapshot-identifier "$snapshot_id"
        if [[ $? -eq 0 ]]; then
            break
        fi
        retries=$((retries + 1))
        sleep 5
    done

    if [[ $retries -eq $max_retries ]]; then
        echo "Error: Failed to create snapshot for RDS instance: $db_identifier"
        continue
    fi

    # Wait for the snapshot to be available
    aws rds wait db-snapshot-available --db-snapshot-identifier "$snapshot_id"

    # Restore the snapshot to a new RDS instance
    retries=0
    while [[ $retries -lt $max_retries ]]; do
        new_instance_id="new-instance-$db_identifier-$timestamp"
        subnet_group_name=$(aws rds describe-db-instances --db-instance-identifier "$db_identifier" --query 'DBInstances[0].DBSubnetGroup.DBSubnetGroupName' --output text)
        aws rds restore-db-instance-from-db-snapshot --db-instance-identifier "$new_instance_id" --db-snapshot-identifier "$snapshot_id" --db-subnet-group-name "$subnet_group_name"
        if [[ $? -eq 0 ]]; then
            break
        fi
        retries=$((retries + 1))
        sleep 5
    done

    if [[ $retries -eq $max_retries ]]; then
        echo "Error: Failed to restore snapshot for RDS instance: $db_identifier"
        continue
    fi

    # Wait for the new instance to be available
    aws rds wait db-instance-available --db-instance-identifier "$new_instance_id"

    # Get the actual RDS endpoint of the new instance
    new_instance_endpoint=$(aws rds describe-db-instances --db-instance-identifier "$new_instance_id" --query 'DBInstances[0].Endpoint.Address' --output text)

    # Take a backup of the database on the new instance and save it to S3
    retries=0
    while [[ $retries -lt $max_retries ]]; do
        mysqldump -h "$new_instance_endpoint" -u "$db_username" -p"$db_password" --all-databases | aws s3 cp - "s3://$your_bucket_name/$snapshot_id.sql"
        if [[ $? -eq 0 ]]; then
            break
        fi
        retries=$((retries + 1))
        sleep 5
    done

    if [[ $retries -eq $max_retries ]]; then
        echo "Error: Failed to backup database to S3 for RDS instance: $db_identifier"
    else
        # Delete old backups from S3 bucket
        delete_old_backups "$db_identifier" "$your_bucket_name"
        
        # Delete the new RDS instance
        echo "Deleting the new RDS instance"
        aws rds delete-db-instance --db-instance-identifier "$new_instance_id" --skip-final-snapshot

        # Delete the snapshot
        echo "Deleting the new RDS snapshot"
        aws rds delete-db-snapshot --db-snapshot-identifier "$snapshot_id"

        echo "Backup and cleanup completed for RDS instance: $db_identifier"
    fi
done

Backup Cleanup and Instance Deletion: After a successful backup, the script deletes old backups from the S3 bucket using the delete_old_backups function. It then deletes the new RDS instance and the snapshot associated with it.

Script Completion: Once the script completes its execution for all RDS instances, it clears the temporary assumed IAM role credentials and displays a completion message.

# Clear the temporary assumed IAM role credentials
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN

echo "Script execution completed."

Running the Script

Execute the Script manually:

./rds_backup.sh account_number role_name

You can also schedule this script in a crontab to run daily every 3am:

0 3 * * * /usr/rds_backup.sh account_number role_name > 1/dev/null>&2

Full Script below

In Summary

Automating the process of creating backups, storing them securely, and cleaning up old backups is essential for data management in cloud environments. The Bash script provided in this tutorial streamlines this process for Amazon RDS instances by leveraging AWS CLI commands, AWS Secrets Manager, and Amazon S3. By customizing and implementing this script, you can ensure data resilience and availability in your AWS infrastructure while minimizing manual intervention.

Remember to thoroughly test the script in a controlled environment before deploying it to production. Always ensure that your AWS permissions are properly configured and follow best practices for security and data management.

Do click on the 👏 button if this guide is helpful.