AWS bucket replication lab

Cloud Data and Risk Management Lab

Goals:

understand the importance of protecting data
setup S3 bucket replication

Lab Outline

preparation
S3 bucket versioning and replication
Cleanup

S3 bucket versioning and replication

Create S3 “source” bucket with bucket versioning
Create S3 “destination” bucket with bucket versioning
Setup bucket replication rules

To test:

Upload a file to “source” bucket
Watch it replicate to the “destination” bucket

The “permission magic” under the hood:

review the IAM role that allows S3 to assume replication rights

Preparation

connect to the AWS console
connect to your EC2 instance

Create terraform configuration

cd
cd terraform
mkdir s3
cd s3

now let’s edit the terraform manifest

nano main.tf

!! REMEMBER to replace with your group name!

Example: “group10”

provider "aws" {
  region = "ap-southeast-1"
}

# Define the remote backend for the terraform state database

terraform {
  backend "s3" {
      bucket = "nsrc-noc"
      key    = "terraform/deploy-lab/<group_name>/s3_replication_terraform_state_file"
      region = "ap-southeast-1"
  }
  required_providers {
    aws  = {
      source = "hashicorp/aws"
    }
  }

}


# Create source bucket
resource "aws_s3_bucket" "source_bucket" {
  bucket = "<group_name>-source"
  force_destroy = true
}

resource "aws_s3_bucket_versioning" "source" {
  bucket = aws_s3_bucket.source_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Create destination bucket
resource "aws_s3_bucket" "destination_bucket" {
  bucket = "<group_name>-destination"
  force_destroy = true
}

resource "aws_s3_bucket_versioning" "destination" {
  bucket = aws_s3_bucket.destination_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}


resource "aws_s3_bucket_replication_configuration" "replication" {
  bucket = aws_s3_bucket.source_bucket.id

  ## this role has already been defined for you
  ## we will review later

  role = "arn:aws:iam::058264411872:role/service-role/s3crr_role_for_nsrc-noc" # Specify ARN of IAM role for replication

  rule {
    id     = "rule-id"
    status = "Enabled"

    destination {
      bucket = aws_s3_bucket.destination_bucket.arn
    }

  }
  depends_on = [aws_s3_bucket_versioning.source]
}

Apply terraform configuration

terraform init

then

terraform plan

!!! NOTE: Remember review the plan carefully!

You should see an output similar to this:

Terraform will perform the following actions:

  # aws_s3_bucket.destination_bucket will be created
  + resource "aws_s3_bucket" "destination_bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "noc_destination"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)
    }

  # aws_s3_bucket.source_bucket will be created
  + resource "aws_s3_bucket" "source_bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "noc_source"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)
    }

# aws_s3_bucket_replication_configuration.replication will be created
  + resource "aws_s3_bucket_replication_configuration" "replication" {
      + bucket = (known after apply)
      + id     = (known after apply)
      + role   = "arn:aws:iam::058264411872:role/service-role/s3crr_role_for_nsrc-noc"

      + rule {
          + id     = "rule-id"
          + status = "Enabled"

          + destination {
              + bucket = (known after apply)
            }
        }
    }

  # aws_s3_bucket_versioning.destination will be created
  + resource "aws_s3_bucket_versioning" "destination" {
      + bucket = (known after apply)
      + id     = (known after apply)

      + versioning_configuration {
          + mfa_delete = (known after apply)
          + status     = "Enabled"
        }
    }

  # aws_s3_bucket_versioning.source will be created
  + resource "aws_s3_bucket_versioning" "source" {
      + bucket = (known after apply)
      + id     = (known after apply)

      + versioning_configuration {
          + mfa_delete = (known after apply)
          + status     = "Enabled"
        }
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Let’s apply the terraform plan

terraform apply

Let’s see what terraform created

terraform state list

you should see something like this:

aws_s3_bucket.destination_bucket
aws_s3_bucket.source_bucket
aws_s3_bucket_replication_configuration.replication
aws_s3_bucket_versioning.destination
aws_s3_bucket_versioning.source

Let’s take a look at the replication configuration we created:

terraform state show aws_s3_bucket_replication_configuration.replication

You should see something like this:

resource "aws_s3_bucket_replication_configuration" "replication" {
    bucket = "noc-source"
    id     = "noc-source"
    role   = "arn:aws:iam::058264411872:role/service-role/s3crr_role_for_nsrc-noc"

    rule {
        id       = "rule-id"
        priority = 0
        status   = "Enabled"

        destination {
            bucket = "arn:aws:s3:::noc-destination"
        }
    }
}

The output is easy to understand:

the “source” bucket has a replication rule that copies everything uploaded to it to the “destination” bucket

Verify replication works

Now, let’s verify if our replication setup works as intended!

Go to the S3 console, and locate your “source” bucket
Click on the bucket name, and click on “Create folder”:

On the screen that opens specify the name of the folder you want to create:

then scroll down and click “Create folder”:

once the folder is created you can go back Click to move to the folder you just created.

click on ’Add files”, then choose any file from your laptop that doesn’t contain any sensitive information

click “Upload”

once you upload the file you should see a green-background message “Upload succeeded”

then you should the file you just uploaded, something like this:

now go back to the S3 console and
find the “destination” bucket
click on it
you should see a folder identical to the one you created in the source bucket (meaning replication works)
then click on the folder name to see its content, and you should see the file that you uploaded to the “source” bucket that just replicated to the “destination”

Note that we could also use the AWS cli to check that what we created and uploaded to the “source” made it the the “destination”

Remember to replace by your group name and by the name of the folder you created

Example: “group10”

this is the folder in the “source” bucket:

aws s3 ls s3://<group_name>-source/<folder>/

this is the folder in the “destination” bucket:

aws s3 ls s3://<group_name>-destination/<folder>/

Example:

aws s3 ls s3://noc-source/test-replication/
2024-02-22 01:11:43          0
2024-02-22 01:12:08     270592 cloud-deployment.pdf
2024-02-22 01:13:59      62387 public-cloud-management.odp


aws s3 ls s3://noc-destination/test-replication/
2024-02-22 01:11:43          0
2024-02-22 01:12:08     270592 cloud-deployment.pdf
2024-02-22 01:13:59      62387 public-cloud-management.odp

If you can see something like that, you successfully completed the exercise. Congrats!!

Cleanup

Now let’s cleanup the buckets we created, to test terraform removing resources (end of resource lifecycle)

terraform destroy

Watch the destruction process. Once completed, if you run

terraform plan

The output should look like terraform is planning to create the resources. That means it removed the resources from both AWS and the terraform state database.

In other words, the resources were destroyed

Optional / Reference: Examine the role

You can find the role which was granted to the source S3 bucket (to allow it to replicate to another S3 bucket) under IAM Roles in the AWS web interface. With a little work, you can also extract the information from aws cli.

# 1. List the attached managed policies for the role
aws iam list-attached-role-policies --role-name s3crr_role_for_nsrc-noc
{
    "AttachedPolicies": [
        {
            "PolicyName": "s3crr_for_nsrc-noc_80e96a",
            "PolicyArn": "arn:aws:iam::058264411872:policy/service-role/s3crr_for_nsrc-noc_80e96a"
        }
    ]
}

# 2. For each attached managed policy, get the default policy version
aws iam get-policy --policy-arn arn:aws:iam::058264411872:policy/service-role/s3crr_for_nsrc-noc_80e96a
{
    "Policy": {
        "PolicyName": "s3crr_for_nsrc-noc_80e96a",
        "PolicyId": "ANPAQ3EGUL3QMB2B3ZWVW",
        "Arn": "arn:aws:iam::058264411872:policy/service-role/s3crr_for_nsrc-noc_80e96a",
        "Path": "/service-role/",
        "DefaultVersionId": "v2",
        "AttachmentCount": 1,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2024-02-22T08:36:47+00:00",
        "UpdateDate": "2024-02-22T08:43:43+00:00",
        "Tags": []
    }
}

# 3. Get the JSON policy document of the default version
aws iam get-policy-version --policy-arn arn:aws:iam::058264411872:policy/service-role/s3crr_for_nsrc-noc_80e96a --version-id v2
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "s3:ListBucket",
                        "s3:GetReplicationConfiguration",
                        "s3:GetObjectVersionForReplication",
                        "s3:GetObjectVersionAcl",
                        "s3:GetObjectVersionTagging",
                        "s3:GetObjectRetention",
                        "s3:GetObjectLegalHold"
                    ],
                    "Effect": "Allow",
                    "Resource": [
                        "arn:aws:s3:::*",
                        "arn:aws:s3:::*/*"
                    ]
                },
                {
                    "Action": [
                        "s3:ReplicateObject",
                        "s3:ReplicateDelete",
                        "s3:ReplicateTags",
                        "s3:ObjectOwnerOverrideToBucketOwner"
                    ],
                    "Effect": "Allow",
                    "Resource": [
                        "arn:aws:s3:::*/*",
                        "arn:aws:s3:::*/*"
                    ]
                }
            ]
        },
        "VersionId": "v2",
        "IsDefaultVersion": true,
        "CreateDate": "2024-02-22T08:43:43+00:00"
    }
}

# 4. List inline policies embedded directly in the role (not managed policies)
aws iam list-role-policies --role-name s3crr_role_for_nsrc-noc
{
    "PolicyNames": []
}

# 5. If there were any, get the inline policy document for each inline policy name
aws iam get-role-policy --role-name <role_name> --policy-name <policy_name>