Cross region replication was introduced a little ago and it can be used to cope with company’s compliance and meet DR (Disaster Recovery) / BCP (Business Continuity Program) demands. Users now can configure a replicatioin configuration in their buckets and write rules how to replicate objects under the buckets.
When we write a cloudformation template to create replication buckets and replication configuration across regions, there is a constrain to do this with a single cloudformation file because cloudformation stack is tied with region. Cloudformation stack can be created in a particular region.
In this case we can’t create a destination bucket and bucket configuration from a source region where we create a stack of source bucket. In this blog, the author packed lambda execution role and lambda function to create a destination bucket by lambda.
I prefer to use 2 separate cloudformation templates rather than it. Because created lambda execution role and lambda will not be used for any other purpose and embedded lambda code in cloudformation file seems complicated and not be maintainable.
In this guide, it shows how to write 2 cloudformation templates for S3 cross region replication across regions with encryption configuration of buckets. First create a destination bucket in us-east-1 and the second create a source bucket in ap-northeast-1 by cloudformation.
Create a destination bucket
First let’s create a destination bucket because this destination bucket must exist when we create a source bucket to point out the bucket’s name.
The bucket has AES256 default encryption and the lifecycle policy to delete older versioned objects after 21 days. Versioning and lifecycle policy must be retained in a destination bucket. At a glance, these are the configuration I had in my bucket configuration.
- Versioning is enabled
- Lifecycle policy configured
- SSE-S3 is used for encryption (the default encryption)
Now you can run aws cloudformation command to create a destination bucket as below.
$ aws cloudformation create-stack --stack-name s3-replication --template-body file://01_s3_destination_bucket.yaml --capabilities CAPABILITY_NAMED_IAM
{
"StackId": "arn:aws:cloudformation:us-east-1:<account id>:stack/s3-replication/<object id>"
}
Create a source bucket
Next we need to create a source bucket where we replicate objects to the created destination bucket. The bucket configuration is actually the same except the replication configuration. If we use the default encryption SSE-S3, encrypted objects can be replicated in nature.
We need at least 1 rule in a replication configuration that has the destination bucket information, the IAM role’s arn that has appropriate accesses to the buckets. Needed configuration is explained in the documentation here. Here’s the example replication configuration as a snippet.
You may notice that we have IAM role associated with this replication configuration. Why do we need this? Because this IAM role allows a source bucket to handle replications and replicate objects to a destination bucket.
Besides the default encryption, If you decided to use KMS CMS encryption you need to also allow S3 to operate both KMS CMS keys to decrypt and encrypt objects across regions. You need to refer to 2 documentations to create such IAM policy appropriately.
Setting up permissions for replication
This bucket must have the same configuration, SSE-S3 encryption and the lifecycle policy to delete older versioned objects after 21 days. Versioning must be enabled at both end for s3 cross region replication. Here’s the sample bucket configuration in cloudformation template.
- Versioning is enabled
- Lifecycle policy configured
- SSE-S3 is used for encryption (the default encryption)
Now you can run aws cloudformation command to create a source bucket as below.
$ aws cloudformation create-stack --stack-name s3-replication --template-body file://02_s3_original_bucket.yaml --capabilities CAPABILITY_NAMED_IAM
{
"StackId": "arn:aws:cloudformation:ap-northeast-1:<account id>:stack/s3-replication/<object id>"
}
Cool. We can now test if replication is functional or not by just copying a file to the source bucket. If you use aws cli, you can use aws s3 cp command to copy a local file to s3 bucket.
$ aws s3 cp testfile s3://primary-test-1-dev-bucket