Configure bucket synchronization (PUT bucket replication)

Content Platform for Cloud Scale S3 Console Guide
Version
2.4.x
Audience
anonymous
Part Number
MK-HCPCS009-04

You can configure S3 bucket sync-to and sync-from settings.

Note:
  • If you use the AWS command-line interface to configure bucket synchronization, use at least aws-cli v1.16.211 and aws-sdk 1.11.610.
  • Configuration rules should be provided to AWS CLI from a file, rather than inline. This is to avoid problems with double quote characters in some terminals.

HTTP request syntax (URI) 

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

Request structure

A rule consists of up to 1000 prefixes and tag-value pairs. You can configure up to 1000 rules per bucket. Separate tag-value pairs in the rule using the keywords "And": or "Or":.

The content of the configuration JSON file is:

aws --endpoint-url https://company.com s3api put-bucket-replication --bucket "bucket name" --replication-configuration \
'{
    "Role": "",
    "Rules": [{
        "ID": "string",
        "Filter": {
            "Prefix": "string",
            "Tag": {
                "Key": "string",
                "Value": "string"
            }
        },
        "Status": "Enabled",
        "Destination": {
            "Bucket": "<a string with several parameters>",
            "Account": "<a string with several parameters>",
            "StorageClass": "<a string with several parameters>"
        }
      },
      {
        "ID": "string",
        "Filter": {
            "Prefix": "string",
            "Tag": {
                "Key": "string",
                "Value": "string"
            }
        },
        "Status": "Enabled",
        "Destination": {
            "Bucket": "<a string with several parameters>",
            "Account": "",
            "StorageClass": "<a string with several parameters>"
      }
    }]
}'
Note: S3 parameters not shown are not required, not supported, and if specified should be left empty.
Account Parameter Required Type Description
Role Yes N/A Not supported; leave empty.
Rules Yes N/A Container for a list of one or more rules. Supports up to 1000 rules.
ID No String

Unique identifier for rule, up to 255 characters.

All rules must specify the same bucket.
Status Yes N/A Values: Enabled or Disabled. The rule is ignored if status is set to Disabled.
Filter Yes N/A Container for prefixes and tags. Each rule can have one prefix, and up to 1000 tags. See AWS for more details on syntax.
Priority Yes Integer Not supported; ignored.
DeleteMarkerReplication.Status No String Not supported; if provided, leave as Disabled.
Prefix No String Prefix (one per rule). Up to 1024 characters.
Key No String Tag key (up to 1000 per rule). Up to 128 characters.
Value No String Tag value. Up to 256 characters.
Status Yes Boolean Enabled or Disabled. If Disabled, rule is ignored.
Destination.Bucket Yes Base64-encoded JSON

External S3 bucket access settings.

  • For bucket sync-to, the settings to access the external bucket.
  • For bucket sync-from, the settings to access the external bucket and the SQS queue settings.

You can't specify the same bucket name and host as both source and destination.
Destination.Account Yes String Must include the empty string "Account": "", in order for the destination rule to function correctly.
Destination.StorageClass No String An optional destination storage class override to use when synchronizing objects. If not provided, this value should be left empty.
Destination.AccessControlTranslation.Owner No String Not supported; leave empty.

Bucket sync-to structure

Bucket sync-to settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded JSON structure.

The syntax inside the bucket parameter for the sync-to setting is:

{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    },
  'notifications': {
    'type': 'type', 
    'region': 'region', 
    'queue': 'queue', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key'
    }
}
Parameter Required Type Description
version Yes String 1.0.
host Yes IP address Host IP address.
type Yes String Destination storage class: AMAZON_S3 or GENERIC_S3.
region Yes String The S3 region.
remoteBucketName Yes String The name of the bucket, from 3 to 63 characters long, containing only lowercase characters (a-z), numbers (0-9), periods (.), or hyphens (-). The bucket must already exist.
accessKey Yes Base64 encoded string The S3 access key credentials to the external S3 bucket.
secretKey Yes Base64 encoded string The S3 secret key credentials to the external S3 bucket.
port Yes integer Host port.
authVersion Yes String AWS Signature version: V2 or V4.
usePathStyleAlways Yes Boolean Path-style URLs for bucket access: true or false.

Bucket sync-from structure

Bucket sync-from settings include both a bucket address and a notification queue. The settings are defined by a set of parameters and passed in the value of Rules.Destination.Bucket as a Base64-encoded string.

The syntax inside the bucket parameter for sync-from setting is:

"{
  'version': 'version', 
  'action': 'sync-from', 
  'externalBucket': {
    'host': 'host', 
    'type': 'type', 
    'region': 'region', 
    'remoteBucketName': 'bucket_name', 
    'accessKey': 'B64_key', 
    'secretKey': 'B64_key', 
    'port': 'port', 
    'authVersion': 'auth_version', 
    'usePathStyleAlways': '[true|false]'
    }
}"
Parameter Required Type Description
version Yes String Enter 1.0.
host Yes IP address Host IP address.
type Yes String Destination storage class: AMAZON_S3 or GENERIC_S3.
region Yes String The S3 region.
remoteBucketName Yes String The name of the bucket, from 3 to 63 characters long, containing only lowercase characters (a-z), numbers (0-9), periods (.), or hyphens (-). The bucket must already exist.
accessKey Yes Base64 encoded string The S3 access key credentials to the external S3 bucket.
secretKey Yes Base64 encoded string The S3 secret key credentials to the external S3 bucket.
port Yes integer Host port.
authVersion Yes String AWS Signature version: V2 or V4.
usePathStyleAlways Yes Boolean Path-style URLs for bucket access: true or false.
Destination.type Yes String Always set as AWS_SQS.
Destination.region Yes String Region of your AWS_SQS queue.
Destination.queue Yes String Name of your AWS_SQS queue.
Destination.accessKey Yes Base64 encoded string accessKey for permissions to read from your AWS_SQS queue.
Destination.secretKey Yes Base64 encoded string secretKey for permissions to read from your AWS_SQS queue.

Response structure

None.

Example

Request example:

aws --endpoint-url https://10.08.1019 s3api put-bucket-replication --bucket "hcpcs_bucket" --replication-configuration file://rules.json

Configuration rules.json:

{
    "ID": "sync_rule2_for_music",
    "Filter": {
      "Prefix": "/music/october/",
      "Tag": {
        "Key": "target",
        "Value": "cloud"
        }
      }
    },
    "Status": "Enabled",
    "Destination": {
      "Bucket": "{
        'version' : '1.0',
        'action' : 'sync_from',
        'externalBucket' : {
          'type' : 'AMAZON_S3',
          'region' : 'us-east-1',
          'remoteBucketName' : 'bluebucket',
          'authVersion' : 'V4',
          'usePathStyleAlways' : 'true',
          'accessKey' : 'access_key',
          'secretKey' : 'secret_key'
          },
        "notifications" : {
          "type" : "AMAZON_SQS",
          "region" : "us-east-1",
          "queue" : "testQueue",
          "accessKey" : "access_key",
          "secretKey" : "secret_key"
          }
        },
      }
    }
  }]
}