Set up scheduled backups for Amazon DynamoDB using AWS Backup

With customers scaling up their AWS workloads across hundreds, if not thousands of AWS resources, customers have expressed the need to centrally manage and monitor their backups. They want to have a standardized way to manage their backups at scale.

AWS Backup enables you to centralize and automate data protection across AWS services. AWS Backup offers a cost-effective, fully managed, policy-based service that further simplifies data protection at scale. You can schedule periodic or future backups by using AWS Backup; these backup plans include schedules and retention policies for your resources. AWS Backup creates the backups and deletes prior backups based on your retention schedule. Backups of resources are always required in case of disaster recovery. AWS Backup removes the undifferentiated heavy lifting of manually making and deleting on-demand backups by automating the schedule and deletion for you.

In this post, we show you how to schedule periodic backups of an Amazon DynamoDB table using AWS Backup. We create a backup plan where we take daily backups and keep them for a month. Also, we show you how you can restrict people in your organization from deleting backups while still being able to do other operations like creating backups and tables.

AWS Backup overview

AWS Backup is designed to help you centralize and automate data protection across AWS services. AWS Backup enables you to centrally deploy backup policies to configure, manage, and govern your backup activity across your organization’s AWS accounts and resources, including Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Block Storage (Amazon EBS) volumes, Amazon Relational Database Service (Amazon RDS) databases , DynamoDB tables, Amazon Elastic File System (Amazon EFS), Amazon FSx for Lustre, Amazon FSx for Windows File Server, and AWS Storage Gateway volumes.

Let’s understand some AWS Backup terminology:

• Backup vault – A container that you organize your backups in.
• Backup plan – A policy expression that defines when and how you want to back up your AWS resources. The backup plan is attached to a backup vault.
• Resource assignment – This defines which resources should be backed up. You can select resources by tags or by resource ARN.
• Recovery point – A snapshot or backup of a resource backed up by AWS Backup. Each recovery point can be restored with AWS Backup.

Solution overview

Amazon DynamoDB offers two types of backup: point-in-time recovery (PITR) and on-demand. PITR provides continuous backups of your table and enables you to restore your table data to any point in time in the preceding 35 days. If you need to store backups of your data for longer than 35 days, you can use on-demand backup. On-demand provides you a fully consistent snapshot of your table data and stay around forever (even after the table is deleted). However, you may be used to deploying traditional backup solutions in your data centers and want to work with a centralized backup solution. The solution can schedule backups through jobs and handle tasks such as expiring and deleting older backups, monitoring the status of ongoing backups, verifying compliance, and finding and restoring backups, all from a central console. At the same time, you don’t want to manually manage your backups, create your own tools via scripts.

You can schedule and manage backups yourself by using AWS Lambda functions, or use a different solution for each application you have to protect. You also have the option of using AWS Backup, which provides you a more similar experience as your traditional backup solutions and simplifies your backup management by eliminating the need to use a different custom solution for each application you have to protect.

You walk through the following steps:

1. Launch an AWS CloudFormation template to create the required resources.
2. Load data into DynamoDB using a Lambda function.
3. Configure a scheduled backup for the DynamoDB table.
4. Restore the backup to a new DynamoDB table.

The following diagram illustrates the architecture of this solution.

In the next section, we use all these concepts to schedule a backup of a DynamoDB table and restore it with AWS Backup.

Create resources with AWS CloudFormation

You can download the data files and CloudFormation stack file from GitHub.

Use the following steps to create AWS resources to build the solution:

1. Sign in to the AWS Management Console.
2. Choose Launch stack:
3. Enter a stack name and choose Next.
4. Accept all the defaults and choose Next.
5. Acknowledge the creation of AWS Identity and Access Management (IAM) resources and choose Create stack.

Load data to DynamoDB

To replicate a real-life scenario, we simulate an online retail application. For this post, you upload raw data files to an Amazon Simple Storage Service (Amazon S3) bucket and use a Lambda function to load the data to DynamoDB. Complete the following steps:

1. Upload the order_data.csv file to the S3 bucket created by CloudFormation. For instructions, see Step 2: Upload an object to your bucket.
2. On the Lambda console, choose the stack <stack_name>-CsvToDDBLambdaFunction-*.
3. Choose Test.

4. For Event name, enter a name. Remove the “template” event and replace it with {}.

5. Choose Create.
6. Choose Test.

This runs the Lambda function and loads the CSV file order_data.csv to the DynamoDB table. Wait for the succeeded message.

7. To see the data in DynamoDB, on the DynamoDB console, choose the table order_transaction-*.
8. View the data on the Items tab.

Configure a scheduled backup

You must create at least one vault before creating a backup plan or starting a backup job.

Create a backup vault

1. On the AWS Backup console, choose Backup vaults in the navigation pane.
2. Choose Create Backup vault.

3. For Backup vault name, enter a name.

By default, AWS Backup creates an AWS Key Management Service (AWS KMS) key with the alias aws/backup for you.

4. For Encryption key, use the default key or any other key in your account.
5. Choose Create Backup vault.

The following screenshot shows that the backup vault was created successfully.

Now, we need to create a backup plan.

Create a backup plan

1. Choose Backup plans in the navigation pane.
2. Choose Create Backup plan.

3. Select Build a new plan.
4. For Backup plan name, enter a name.
5. For Rule name, enter a name for your rule.

6. For Frequency, choose Daily.

The backup frequency determines how often a backup is created. You can choose a frequency of every 12 hours, daily, weekly, or monthly.

7. For Backup window, select Customize backup window.

The backup window consists of the time that the backup window begins and the duration of the window in hours. Backup jobs are started within this window.

8. Next, we configure the lifecycle, which defines when a backup is transitioned to cold storage and when it expires.

For this post, we configure the backup to move to cold storage after 31 days and expire after 366 days.

Note: As of this writing, only Amazon EFS file system backups can be transitioned to cold storage. The cold storage expression is ignored for the backups of Amazon EBS, Amazon RDS, Aurora, Amazon DynamoDB, and Storage Gateway.

9. For Backup vault¸ choose the vault created earlier.
10. Choose Create plan.

Now we assign the resource to the backup plan. When you assign a resource to a backup plan, that resource is backed up automatically according to the backup plan.

Assign resource to backup plan

1. On the details page for the backup plan, choose Assign resources.

2. For Resource assignment name, enter a name.
3. For IAM role, select Default role.
4. Under Assign resources, choose our DynamoDB table.
5. Choose Assign resources.

You can see the status of your backup job on the Jobs page after your scheduled backup window timeframe. The following screenshot shows that the DynamoDB backup is complete.

You can choose the job to see more details like backup size and resource type.

Restore a backup

After a resource has been backed up at least once, it’s considered protected and is available to be restored using AWS Backup. On the Protected resources page, you can explore details of the resources that are backed up in AWS Backup.

1. Choose our DynamoDB table resource.

2. Select the recovery point ID of the resource.
3. Choose Restore.

4. For New table name, enter a name for your new DynamoDB table.
5. For Restore role, select Default role.
6. Choose Restore backup.

The Restore jobs pane appears. A message at the top of the page provides information about the restore job. You can see the job status in the following screenshot is Running.

After some time, the status changes to Completed.

You can also monitor all the backup and restore jobs on the central dashboard.

You can view the restored table on the DynamoDB console. The following screenshot shows that the table is restored along with the data.

Restrict backup deletion

In some use cases, you may want to allow your developer or administrator to create and delete DynamoDB tables, but not delete the backups.

You can achieve this by creating an IAM policy. The following example IAM policy allows a user to create tables, list tables, delete tables, create backups, and also denies the ability to delete backups:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:CreateTable",
                "dynamodb:CreateBackup",
                "dynamodb:DeleteTable"
            ],
            "Resource": "arn:aws:dynamodb:us-west-1:123456789:table/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "dynamodb:ListTables",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Deny",
            "Action": "dynamodb:DeleteBackup",
            "Resource": "arn:aws:dynamodb:us-west-1:123456789:table/*/backup/*"
        }
    ]
}

For more information about IAM policies, see Using IAM with DynamoDB Backup and Restore.

You can restrict permissions in AWS Backup as well by denying DeleteBackupSelection in the IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "backup:DeleteBackupSelection",
                "backup:CreateBackupSelection",
                "backup:StartBackupJob",
                "backup:CreateBackupPlan",
                "backup:ListBackupSelections",
                "backup:ListRecoveryPointsByBackupVault",
                "backup:GetBackupVaultAccessPolicy",
                "backup:GetBackupSelection"
            ],
            "Resource": [
                "arn:aws:backup:us-west-1:123456789:backup-plan:*",
                "arn:aws:backup:us-west-1:123456789:backup-vault:*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Deny",
            "Action": "backup:DeleteBackupSelection",
            "Resource": "arn:aws:backup:us-west-1:123456789:backup-plan:*"
        }
    ]
}

You can apply the policy to the role and assign the role to the IAM group. The users that belong to this IAM group inherit the permissions.

To test this, let’s say that a user tries to delete the backup in AWS Backup.

The user gets an access denied error due to insufficient permissions to delete the backup.

Clean up the resources

To avoid incurring future charges, follow these steps to remove the example resources:

1. Delete the backup policy, backup plans, and recovery points. For instructions, see Clean up resources.
2. Delete the resources you created with the CloudFormation template by deleting the stack instances from your stack set.

Conclusion

In this blog post, we described how to use AWS Backup to take frequent backups from your DynamoDB tables. We have provided a detailed step-by-step guide of how AWS Backup works to schedule and monitor your DynamoDB backups. Also, we provided the steps to restore the DynamoDB table from a restore point using AWS Backup.

To learn more about AWS Backup, check out the Developer Guide.

About the Author

Dhiraj Thakur is a Solutions Architect with Amazon Web Services. He works with AWS customers and partners to provide guidance on enterprise cloud adoption, migration, and strategy. He is passionate about technology and enjoys building and experimenting in analytics and AI/ML space.