Deployment options for FSx for Lustre file systems
FSx for Lustre provides a high performance, parallel file system that stores data across multiple network file servers to maximize performance and reduce bottlenecks. These servers have multiple disks. To spread load, Amazon FSx shards file system data into smaller chunks and spreads them across disks and servers using a process called striping. For more information about FSx for Lustre data striping, see Striping data in your file system.
It's a best practice to link a highly durable long-term data repository residing on Amazon S3 with your FSx for Lustre high-performance file system.
In this scenario, you store your datasets on the linked Amazon S3 data repository. When you create your FSx for Lustre file system, you link it to your S3 data repository. At this point, the objects in your S3 bucket are listed as files and directories on your FSx file system. Amazon FSx then automatically copies the file contents from S3 to your Lustre file system when a file is accessed for the first time on the Amazon FSx file system. After your compute workload runs, or at any time, you can use a data repository task to export changes back to S3. For more information, see Using data repositories with Amazon FSx for Lustre and Using data repository tasks to export changes.
File system deployment options for FSx for Lustre
FSx for Lustre file systems use the persistent deployment type.
Encryption of data at rest is automatically enabled when you create an Amazon FSx for Lustre file system, regardless of the deployment type you use. Scratch 2 and persistent file systems automatically encrypt data in transit when they are accessed from Amazon EC2 instances that support encryption in transit. For more information on encryption, see Data encryption in Amazon FSx for Lustre.
Persistent file systems
Persistent file systems are designed for longer-term storage and workloads. The file servers are highly available, and data is automatically replicated within the same Availability Zone in which the file system is located. The data volumes attached to the file servers are replicated independently from the file servers to which they are attached.
Amazon FSx continuously monitors persistent file systems for hardware failures, and automatically replaces infrastructure components in the event of a failure. On a persistent file system, if a file server becomes unavailable, it's replaced automatically within minutes of failure. During that time, client requests for data on that server transparently retry and eventually succeed after the file server is replaced. Data on persistent file systems is replicated on disks, and any failed disks are automatically replaced transparently.
Use persistent file systems for longer-term storage and for throughput-focused workloads that run for extended periods or indefinitely, and that might be sensitive to disruptions in availability.
The following diagram shows the architecture for an Amazon FSx for Lustre persistent file system, with replicated, highly available file servers and data volumes within a single Availability Zone.
Persistent deployment types automatically encrypt data in transit when they are accessed from Amazon EC2 instances that support encryption in transit.
Persistent deployment types can be built on Lustre v2.10 or v2.12, and support SSD (solid state drive) and HDD (hard disk drive) storage types. For SSD storage, the throughput per unit of storage is either 50, 100, or 200 MB/s per tebibyte (TiB). For HDD, throughput per unit of storage is 12 or 40 MB/s per tebibyte.
Persistent_1 deployment types are available in all Amazon Web Services Regions.
