Attaching a custom file system to a domain or user profile - Amazon SageMaker
PrerequisitesAttaching with the Amazon CLI
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Attaching a custom file system to a domain or user profile

When you create a domain, Amazon SageMaker automatically associates it with an Amazon Elastic File System (Amazon EFS) volume that SageMaker creates for you. You also have the option to associate the domain with a custom Amazon EFS file system that you've created in your Amazon Web Services account. This file system is available to any users who belong to the domain when they use Amazon SageMaker Studio. Users can attach the file system to any space that they create for the supported applications: JupyterLab and Code Editor. Then, after running the space and starting the application, they can access any data, code, or other artifacts that the file system contains.

If you don't want to permit all of the users for a domain to access the file system, you can attach it to a specific user profile instead. If you do that, the file system is available only in spaces that the associated user creates.

You can attach a custom file system by using the Amazon SageMaker API, the Amazon SDKs, or the Amazon CLI. You can't attach a custom file system by using the SageMaker console.

Prerequisites

Before you can attach a custom Amazon EFS file system to a domain, you must meet the following requirements:

  • You have an Amazon EFS file system in your Amazon Web Services account. For the steps to create one, see Create your Amazon EFS file system in the Amazon Elastic File System User Guide.

  • Before Studio can access your file system, it must have a mount target in each of the subnets that you associate with the domain. For more information about assigning mount targets to subnets, see Creating and managing mount targets and security groups in the Amazon Elastic File System User Guide.

  • For each mount target, you must add the security group that Amazon SageMaker created in your Amazon Web Services account when you created the domain. The security group name has the format security-group-for-inbound-nfs-domain-id.

  • Your IAM permissions must allow you to use the elasticfilesystem:DescribeMountTargets action. For more information about this action, see Actions, resources, and condition keys for Amazon Elastic File System in the Service Authorization Reference.

Attaching a custom file system with the Amazon CLI

To attach a custom file system to a domain or user profile with the Amazon CLI, you pass a CustomFileSystemConfigs definition when you use any of the following commands:

Example create-domain command with a custom file system

The following example attaches a file system to a new domain.

aws sagemaker create-domain --domain-name domain-name \
--vpc-id vpc-id --subnet-ids subnet-ids --auth-mode IAM \
--default-user-settings file://default-user-settings.json \
--default-space-settings "ExecutionRole=execution-role-arn"

In this example, the file default-user-settings.json has the following settings, which include the CustomPosixUserConfig and CustomFileSystemConfigs keys.

{
    "ExecutionRole": "execution-role-arn",
    "CustomPosixUserConfig":
    {
        "Uid": UID,
        "Gid": GID
    },
    "CustomFileSystemConfigs":
    [
        {
            "EFSFileSystemConfig":
            {
                "FileSystemId": "file-system-id",
                "FileSystemPath": "/"
            }
        }
    ]
}

This example configuration has the following keys:

ExecutionRole

The default execution role for the users of the domain.

CustomPosixUserConfig

The default POSIX identities that are used for file system operations. You can use these settings to apply your existing POSIX permission structure to the user profiles that access the custom file system. At a POSIX permissions level, you can control which users can access the file system and which files or data they can access.

You can also apply CustomPosixUserConfig settings when you create a user profile by using the create-user-profile command. The settings that you apply to a user profile override those that you apply to the associated domain.

Note

You can apply CustomPosixUserConfig settings when you use the create-domain and create-user-profile commands. However, you can't apply these settings when you do the following:

  • Use the update-domain command for a domain that is already associated with any user profiles. You can apply these settings only to domains that have no user profiles.

  • Use the update-user-profile command. To apply these settings to profile that you've already created, delete the profile, and create a new one that has the updated settings.

Uid

The POSIX user ID. The default is 200001.

Gid

The POSIX group ID. The default is 1001.

CustomFileSystemConfigs

Settings for custom file systems (only Amazon EFS file systems are supported).

You can also apply CustomFileSystemConfigs settings to a user profile when you use the create-user-profile or update-user-profile commands. The user profile will have access to those file systems as well as any that you attach to their domain.

EFSFileSystemConfig

Settings for custom Amazon EFS file systems.

FileSystemId

The ID of your Amazon EFS file system.

FileSystemPath

The path to the file system directory that is accessible to the domain users in their spaces in Studio. Permitted users can access only this directory and below. The default path is the file system root: /.

SageMaker creates a symbolic link at the following path: /home/sagemaker-user/custom-file-systems/file-system-type/file-system-id. With this, the domain users can navigate to the custom file system from within their home directory, /home/sagemaker-user.

After you attach a custom file system to a domain, the domain users can attach the file system to a space when they use the create-space command.

Example create-space command with a custom file system

The following example attaches a file system to a new space.

aws sagemaker create-space \
--space-name space-name \
--domain-id domain-id \
--ownership-settings "OwnerUserProfileName=user-profile-name" \
--space-sharing-settings "SharingType=Private" \
--space-settings file://space-settings.json

In this example, the file space-settings.json has the following settings, which include the CustomFileSystems configuration with the FileSystemId key.

{
    "AppType": "JupyterLab",
    "JupyterLabAppSettings":
    {
        "DefaultResourceSpec":
        {
            "InstanceType": "ml.t3.xlarge"
        }
    },
    "CustomFileSystems":
    [
        {
            "EFSFileSystem":
            {
                "FileSystemId": "file-system-id"
            }
        }
    ]
}