Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Working with older deployment types

This section applies to file systems with Scratch 1 deployment type, and also to file systems with Scratch 2 or Persistent 1 deployment types that do not use data repository associations.

Link your file system to an Amazon S3 bucket

When you create an Amazon FSx for Lustre file system, you can link it to a durable data repository in Amazon S3. Before you create your file system, make sure that you have already created the Amazon S3 bucket to which you are linking. In the Create file system wizard, you set the following data repository configuration properties in the optional Data Repository Import/Export pane.

You can have a 1:1 mapping where Amazon FSx exports data from your FSx for Lustre file system back to the same directories on the S3 bucket that it was imported from. To have a 1:1 mapping, specify an export path to the S3 bucket without any prefixes when you create your file system.

Using this method, you can include an import prefix when you specify the import path, and it doesn't impact a 1:1 mapping for exports.

Creating file systems linked to an S3 bucket

The following procedures walk you through the process of creating an Amazon FSx file system linked to an S3 bucket using the Amazon Management Console and Amazon Command Line Interface (Amazon CLI).

Console

1. Open the Amazon FSx console at https://console.amazonaws.cn/fsx/.

2. From the dashboard, choose Create file system.

3. For the file system type, choose FSx for Lustre, and then choose Next.

4. Provide the information required for the File system details and Network and security sections. For more information, see Step 1: Create your FSx for Lustre file system.

5. You use the Data repository import/export panel to configure a linked data repository in Amazon S3. Select Import data from and export data to S3 to expand the Data Repository Import/Export section and configure the data repository settings.

   

6. Choose how Amazon FSx keeps your file and directory listing up to date as you add or modify objects in your S3 bucket. When you create your file system, your existing S3 objects appear as file and directory listings.

   • Update my file and directory listing as objects are added to my S3 bucket: (Default) Amazon FSx automatically updates file and directory listings of any new objects added to the linked S3 bucket that do not currently exist in the FSx file system. Amazon FSx does not update listings for objects that have changed in the S3 bucket. Amazon FSx does not delete listings of objects that are deleted in the S3 bucket.

     Note

     The default import preferences setting for importing data from a linked S3 bucket using the CLI and API is NONE. The default import preferences setting when using the console is to update Lustre as new objects are added to the S3 bucket.

   • Update my file and directory listing as objects are added to or changed in my S3 bucket: Amazon FSx automatically updates file and directory listings of any new objects added to the S3 bucket and any existing objects that are changed in the S3 bucket after you choose this option. Amazon FSx does not delete listings of objects that are deleted in the S3 bucket.

   • Update my file and directory listing as objects are added to, changed in, or deleted from my S3 bucket: Amazon FSx automatically updates file and directory listings of any new objects added to the S3 bucket, any existing objects that are changed in the S3 bucket, and any existing objects that are deleted in the S3 bucket after you choose this option.

   • Do not update my file and directly listing when objects are added to, changed in, or deleted from my S3 bucket - Amazon FSx only updates file and directory listings from the linked S3 bucket when the file system is created. FSx does not update file and directory listings for any new, changed, or deleted objects after choosing this option.

7. Enter an optional Import prefix if you want to import only some of the file and directory listings of data in your S3 bucket into your file system. The import prefix defines where in your S3 bucket to import data from. For more information, see Automatically import updates from your S3 bucket.

8. Choose one of the available Export prefix options:

   • A unique prefix that Amazon FSx creates in your bucket: Choose this option to export new and changed objects using a prefix generated by FSx for Lustre. The prefix looks like the following: /FSxLustrefile-system-creation- timestamp. The timestamp is in UTC format, for example FSxLustre20181105T222312Z.

   • The same prefix that you imported from (replace existing objects with updated ones): Choose this option to replace existing objects with updated ones.

   • A prefix you specify: Choose this option to preserve your imported data and to export new and changed objects using a prefix that you specify. To achieve a 1:1 mapping when exporting data to your S3 bucket, choose this option and leave the prefix field blank. FSx will export data to same directories it was imported from.

9. (Optional) Set Maintenance preferences, or use the system defaults.

10. Choose Next, and review the file system settings. Make any changes if needed.

11. Choose Create file system.

Amazon CLI

The following example creates an Amazon FSx file system linked to the amzn-s3-demo-bucket, with an import preference that imports any new, changed, and deleted files in the linked data repository after the file system is created.

Note

The default import preferences setting for importing data from a linked S3 bucket using the CLI and API is NONE, which is different from the default behavior when using the console.

To create an FSx for Lustre file system, use the Amazon FSx CLI command create-file-system, as shown below. The corresponding API operation is CreateFileSystem.

$ aws fsx create-file-system \
--client-request-token CRT1234 \
--file-system-type LUSTRE \
--file-system-type-version 2.10 \
--lustre-configuration
AutoImportPolicy=NEW_CHANGED_DELETED,DeploymentType=SCRATCH_1,ImportPath=s
3://amzn-s3-demo-bucket/,ExportPath=s3://amzn-s3-demo-bucket/export,
PerUnitStorageThroughput=50 \
--storage-capacity 2400 \
--subnet-ids subnet-123456 \
--tags Key=Name,Value=Lustre-TEST-1 \
--region us-east-2

After you successfully create the file system, Amazon FSx returns the file system description as JSON, as shown in the following example.

{
    "FileSystems": [
        {
            "OwnerId": "owner-id-string",
            "CreationTime": 1549310341.483,
            "FileSystemId": "fs-0123456789abcdef0",
            "FileSystemType": "LUSTRE",
            "FileSystemTypeVersion": "2.10",
            "Lifecycle": "CREATING",
            "StorageCapacity": 2400,
            "VpcId": "vpc-123456",
            "SubnetIds": [
                "subnet-123456"
            ],
            "NetworkInterfaceIds": [
                "eni-039fcf55123456789"
            ],
            "DNSName": "fs-0123456789abcdef0.fsx.us-east-2.amazonaws.com",
            "ResourceARN": "arn:aws:fsx:us-east-2:123456:file-system/fs-0123456789abcdef0",
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "Lustre-TEST-1"
                }
            ],
            "LustreConfiguration": {
                "DeploymentType": "PERSISTENT_1",
                "DataRepositoryConfiguration": {
                    "AutoImportPolicy": "NEW_CHANGED_DELETED",
                    "Lifecycle": "UPDATING",
                    "ImportPath": "s3://amzn-s3-demo-bucket/",
                    "ExportPath": "s3://amzn-s3-demo-bucket/export",
                    "ImportedFileChunkSize": 1024
                },
                "PerUnitStorageThroughput": 50
            }
        }
    ]
}

Viewing a file system's export path

You can view a file system's export path using the FSx for Lustre console, the Amazon CLI, and the API.

Console

1. Open the Amazon FSx console at https://console.amazonaws.cn/fsx/

2. Choose File system name or File system ID for the FSx for Lustre file system for which you want to view the export path.

   The file system details page appears for that file system.

3. Choose the Data repository tab.

   The Data repository integration panel appears, showing the import and export paths.

CLI

To determine the export path for your file system, use the describe-file-systems Amazon CLI command.

aws fsx describe-file-systems

Look for the ExportPath property under LustreConfiguration in the response.

{
    "OwnerId": "111122223333",
    "CreationTime": 1563382847.014,
    "FileSystemId": "",
    "FileSystemType": "LUSTRE",
    "Lifecycle": "AVAILABLE",
    "StorageCapacity": 2400,
    "VpcId": "vpc-6296a00a",
    "SubnetIds": [
        "subnet-1111111"
    ],
    "NetworkInterfaceIds": [
        "eni-0c288d5b8cc06c82d",
        "eni-0f38b702442c6918c"
    ],
    "DNSName": "fs-0123456789abcdef0.fsx.us-east-2.amazonaws.com",
    "ResourceARN": "arn:aws:fsx:us-east-2:267731178466:file-system/fs-0123456789abcdef0",
    "Tags": [
        {
          "Key": "Name",
          "Value": "Lustre System"
        }
    ],
	"LustreConfiguration": {
    "DeploymentType": "SCRATCH_1",
    "DataRepositoryConfiguration": {
    "AutoImportPolicy": " NEW_CHANGED_DELETED",
    "Lifecycle": "AVAILABLE",
    "ImportPath": "s3://amzn-s3-demo-bucket/",
    "ExportPath": "s3://amzn-s3-demo-bucket/FSxLustre20190717T164753Z",
    "ImportedFileChunkSize": 1024
    }
  },
  "PerUnitStorageThroughput": 50,
  "WeeklyMaintenanceStartTime": "6:09:30"
}

Data repository lifecycle state

The data repository lifecycle state provides status information about the file system's linked data repository. A data repository can have the following Lifecycle states.

You can view a file system's linked data repository lifecycle state using the Amazon FSx console, the Amazon Command Line Interface, and the Amazon FSx API. In the Amazon FSx console, you can access the data repository Lifecycle state in the Data Repository Integration pane of the Data Repository tab for the file system. The Lifecycle property is located in the DataRepositoryConfiguration object in the response of a describe-file-systems CLI command (the equivalent API action is DescribeFileSystems).

Automatically import updates from your S3 bucket

By default, when you create a new file system, Amazon FSx imports the file metadata (the name, ownership, timestamp, and permissions) of objects in the linked S3 bucket at file system creation. You can configure your FSx for Lustre file system to automatically import metadata of objects that are added to, changed in, or deleted from your S3 bucket after file system creation. FSx for Lustre updates the file and directory listing of a changed object after creation in the same manner as it imports file metadata at file system creation. When Amazon FSx updates the file and directory listing of a changed object, if the changed object in the S3 bucket no longer contains its metadata, Amazon FSx maintains the current metadata values of the file, rather than using default permissions.

You can set import preferences when you create a new file system, and you can update the setting on existing file systems using the FSx management console, the Amazon CLI, and the Amazon API. When you create your file system, your existing S3 objects appear as file and directory listings. After you create your file system, how do you want to update it as the contents of your S3 bucket are updated? A file system can have one of the following Import preferences:

When you set the import preferences to update your file system file and directory listings based on changes in the linked S3 bucket, Amazon FSx creates an event notification configuration on the linked S3 bucket named FSx. Do not modify or delete the FSx event notification configuration on the S3 bucket—doing so prevents the automatic import of new or changed file and directory listings to your file system.

When Amazon FSx updates a file listing that has changed on the linked S3 bucket, it overwrites the local file with the updated version, even if the file is write-locked. Similarly, when Amazon FSx updates a file listing when the corresponding object has been deleted on the linked S3 bucket, it deletes the local file, even if the file is write- locked.

Amazon FSx makes a best effort to update your file system. Amazon FSx cannot update the file system with changes in the following situations:

Either of these conditions cause the data repository lifecycle state to become Misconfigured. For more information, see Data repository lifecycle state.

Prerequisites

The following conditions are required for Amazon FSx to automatically import new, changed, or deleted files from the linked S3 bucket:

Types of file changes supported

Amazon FSx supports importing the following changes to files and folders that occur in the linked S3 bucket:

Updating import preferences

You can set a file system's import preferences when you create a new file system. For more information, see Linking your file system to an Amazon S3 bucket.

You can also update a file system's import preferences after it is created using the Amazon Management Console, the Amazon CLI, and the Amazon FSx API, as shown in the following procedure.

Console

1. Open the Amazon FSx console at https://console.amazonaws.cn/fsx/.

2. From the dashboard, choose File systems.

3. Select the file system that you want to manage to display the file system details.

4. Choose Data repository to view the data repository settings. You can modify the import preferences if the lifecycle state is AVAILABLE or MISCONFIGURED. For more information, see Data repository lifecycle state.

5. Choose Actions, and then choose Update import preferences to display the Update import preferences dialog box.

6. Select the new setting, and then choose Update to make the change.

CLI

To update import preferences, use the update-file-system CLI command. The corresponding API operation is UpdateFileSystem.

After you successfully update the file system's AutoImportPolicy, Amazon FSx returns the description of the updated file system as JSON, as shown here:

{
    "FileSystems": [
        {
            "OwnerId": "111122223333",
            "CreationTime": 1549310341.483,
            "FileSystemId": "fs-0123456789abcdef0",
            "FileSystemType": "LUSTRE",
            "Lifecycle": "UPDATING",
            "StorageCapacity": 2400,
            "VpcId": "vpc-123456",
            "SubnetIds": [
                "subnet-123456"
            ],
            "NetworkInterfaceIds": [
                "eni-039fcf55123456789"
            ],
            "DNSName": "fs-0123456789abcdef0.fsx.us-east-2.amazonaws.com",
            "ResourceARN": "arn:aws:fsx:us-east-2:123456:file-system/fs-0123456789abcdef0",
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "Lustre-TEST-1"
                }
            ],
            "LustreConfiguration": {
                "DeploymentType": "SCRATCH_1",
                "DataRepositoryConfiguration": {
                    "AutoImportPolicy": "NEW_CHANGED_DELETED",
                    "Lifecycle": "UPDATING",
                    "ImportPath": "s3://amzn-s3-demo-bucket/",
                    "ExportPath": "s3://amzn-s3-demo-bucket/export",
                    "ImportedFileChunkSize": 1024
                }
                "PerUnitStorageThroughput": 50,
                "WeeklyMaintenanceStartTime": "2:04:30"
            }
        }
    ]
}