Improve Backups with S3

Perhaps the most important aspect of any system is the backup strategy. All sysadmins would agree that without good backups, it’s hard to sleep at night. Over the years, we have used many different backup strategies at Sourcetoad. My current favorite one is based on Amazon Web Services’s S3 and the various options it can provide.

S3 has a few options that make it a very compelling backup system. Combining these options allows one to create a relatively bullet-proof backup solution.

Versioning

Adding versioning to a bucket means that as objects in the bucket are deleted or overwritten, the original data is not lost. Instead, a version is created with the old data. The new data becomes the current version. Enabling versioning on a bucket is as simple as going into the bucket properties and selecting “Enable Versioning” under the versioning dropdown. Note that once versioning is enabled, it cannot be disabled.

Object Lifecycle Management

Object Lifecycle Management allows objects to automatically be deleted after a certain period of time. Keep in mind that with versioning enabled, deleting the object will not remove it, just place a delete marker. The previous version will remain. However, on versioned buckets, Lifecycle rules for previous versions can be defined. This will allow the previous versions to be permanently removed after a certain period of time as well.

For cases where long-term backups are needed, Object Lifecycle Management also can be used to automatically transition objects to lower-cost storage options (either Glacier or Infrequently Accessed Storage Class).

A good example of a Object Lifecycle Management rule for shorter-term disaster recovery backup might be to delete objects 30 days after they are uploaded, and permanently delete old versions 30 days after that. This can be set by going into the bucket properties, and adding a rule under the Lifecycle dropdown.

IAM Permissions

Standard best-practice when working with AWS permissions to assign a user the minimum permissions it needs in order to accomplish what it needs to do. Using the the above options allows the creation of a user with only one permission: PutObject on the backup bucket.

The user does not need permission to delete items from the bucket, because Object Lifecycle Management is handling the removal of old backup data. In fact, if the backup user does get compromised, it will not be able to remove the backups. The PutObject permission does allow overwriting of one object with another, but, with versioning enabled on the bucket, the old version will remain. The user does not have any permissions to act on non-current versions of objects.

Finally, without the GetObject or ListBucket permission, the user cannot retrieve previous backups, or even see a listing of them.

Here is an example of an IAM policy that gives this one single permission.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StmtXXXXXXXXX",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::XXXXXXXXX/*"
            ]
        }
    ]
}

Going Beyond

This article barely scratches the surface of what is possible with S3 and backup. Many different, much more complex scenarios can be addressed. AWS’s other object storage service, Glacier, even offers a “Vault Lock” feature that can be used to create backups that no one (not even the root account holder) can delete before they reach a certain age.

However, even a basic understanding of some of the features of S3 allows a user to create a fairly robust backup system, and sleep better at night.