Setting up ongoing periodic data replication - FSx for OpenZFS
Architecture overviewRequired permissionsStep 1: Initializing and deploying the application Step 2: Monitoring periodic replication
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Setting up ongoing periodic data replication

With ongoing periodic data replication, you can set up a schedule that automatically takes a snapshot of a source volume and performs an incremental replication of that snapshot on a destination volume at a certain interval, for example every 15 minutes. You can schedule ongoing periodic data replication between two volumes on FSx for OpenZFS file systems on the same account, within or across Amazon Web Services Regions, by using the solution provided in this section.

Architecture overview

Deploying this solution builds the following resources in the Amazon Web Services Cloud.

Architecture diagram showing Serverless Application Module deploying an EventBridge Scheduler schedule, a Lambda function, and an SNS queue.

The diagram illustrates the following periodic replication workflow.

  1. Amazon Serverless Application Model (SAM) automates the deployment of the FSx for OpenZFS periodic replication solution. For more information about Amazon SAM, see What is the Amazon Serverless Application Model (Amazon SAM)? in the Amazon Serverless Application Model User Guide.

  2. The SAM template deploys an Amazon EventBridge scheduler, an Amazon Lambda function, an Amazon SNS queue, and an IAM role. The IAM role gives the Lambda function permission to call the necessary Amazon FSx API operations.

  3. The EventBridge scheduler runs on a schedule you specify as a cron pattern during the initial deployment. For more information about cron patterns, see Creating an Amazon EventBridge rule that runs on a schedule in the Amazon EventBridge User Guide. The scheduler invokes a Lambda function that calls the Amazon FSx CreateSnapshot API operation to create a snapshot of the source volume.

  4. Once the snapshot is available, the Lambda function calls the Amazon FSx CopySnapshotAndUpdateVolume API operation to start replicating the source snapshot data to the destination volume.

  5. The Lambda function sends a notification message to the Amazon SNS queue when replication starts, if you choose to be notified during the initial deployment. A notification is always sent when a snapshot cannot be created or the replication cannot be initiated.

Required permissions

The following permissions are required to use the custom snapshot schedule Amazon CloudFormation template.

  • AmazonS3FullAccess

  • AWSCloudFormationFullAccess

  • AmazonEventBridgeFullAccess

  • IAMFullAccess

  • AmazonSNSFullAccess

  • AWSKeyManagementServicePowerUser

  • AWSLambda_FullAccess

For more information about using IAM to set up permissions, see How Amazon FSx for OpenZFS works with IAM.

Step 1: Initializing and deploying the application

The following procedure configures and deploys the periodic replication solution. It takes about five minutes to deploy. Before you begin this step, make sure that you have the ID of the source and destination volumes that you would like to initiate the replication between. For more information on these resources, see Creating a volume, Creating a snapshot, and Using on-demand data replication.

Note

Implementing this solution incurs billing for the associated Amazon services. For more information, see the pricing details pages for those services.

To launch the periodic replication solution stack

  1. Follow the instructions on the Replicate FSx-OpenZFS volumes across file systems page to download the serverless pattern.

  2. For Parameters, review the following parameters for the template and modify them for the needs of your periodic replication. This solution uses the following default values.

    Parameter

    Default

    Description
    Source volume ID No default value The ID of the source volume from which data will be periodically replicated.
    Destination volume ID No default value The ID of the destination volume that will become a replica of the source volume.
    CronSchedule [0 0/6 **?*] (every six hours) The schedule to replicate data from the source volume to the destination volume.
    SnapshotName fsx-scheduled-snapshot The name for the scheduled snapshots that will be taken of the source volume. Appears in the Snapshot Name column of the Amazon FSx Console.
    Snapshot retention (days) 7 The number of days to keep user-initiated snapshots. The Lambda function deletes user-initiatted snapshots that are kept after this number of days.
    SuccessNotification Yes Choose whether to be notified when the replication is successfully initiated. A notification is always sent when a snapshot fails to create or the replication fails to start.
    Email No default value The email address that you would like notifications to be sent to.
    CopyStrategy INCREMENTAL_COPY The CopyStrategy parameter for the CopySnapshotAndUpdateVolume API operation. For more information, see CopySnapshotAndUpdateVolume in the Amazon FSx API.
    Options None The Options parameter for the CopySnapshotAndUpdateVolume API operation. For more information, see CopySnapshotAndUpdateVolume in the Amazon FSx API.

  3. In the Amazon SAM CLI, run the following command to deploy the resources specified in the SAM template.

    sam deploy --guided \
--stack-name fsxz-periodic-replication \
--template-file fsx-openzfs-periodic-replication.yaml \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM

    You will be asked if you would like to update any parameters.

  4. Choose Enter to deploy the template.

Step 2: Monitoring periodic replication

You can monitor the status of the periodic replication workflow using the Amazon FSx Console, Amazon CLI, and API. For more information on how to monitor periodic replication using the Amazon FSx Console, see Monitoring progress of on-demand data replication.

To use the Amazon CLI or API to track the progress of your replication, call the describe-volumes CLI command or the DescribeVolumes API operation to view the AdministrativeActions array for the destination volume. The following example shows the response for an incremental copy on-demand data replication task.

"AdministrativeActions": [
   {
    "AdministrativeActionType": "VOLUME_UPDATE_WITH_SNAPSHOT",
    "ProgressPercent": 100,
    "RequestTime": 1699997847.438,
    "Status": "COMPLETED",
    "TargetVolumeValues": {
    "OpenZFSConfiguration": {
        "RecordSizeKiB": 128,
        "DataCompressionType": "ZSTD",
        "DeleteIntermediateSnaphots": true,
        "DeleteClonedVolumes": false,
        "DeleteIntermediateData": true,
        "SourceSnapshotARN": "arn:aws:fsx:us-east-1:609492434915:snapshot/fsvol-0e1ab09de954a352f/fsvolsnap-01dda47dcbb24ddd0",
        "DestinationSnapshot": "fsvolsnap-0afef62088c7c9060"
        }
    },
    "TotalTransferBytes": 44144,
    "RemainingTransferBytes": 0
   },