FSx 使用 kubectl 部署来自亚马逊 S3 和亚马逊的自定义微调模型

1. 在环境中选择您的区域。

   export REGION=<region>

2. 初始化集群名称。这标识了您的模型将在哪个 HyperPod 集群中部署。

   注意

   请与您的集群管理员核实，确保已为该角色或用户授予权限。您可以运行 !aws sts get-caller-identity --query "Arn" 来检查您在终端使用的角色或用户。

   # Specify your hyperpod cluster name here
   HYPERPOD_CLUSTER_NAME="<Hyperpod_cluster_name>"
   
   # NOTE: For sample deployment, we use g5.8xlarge for deepseek-r1 1.5b model which has sufficient memory and GPU
   instance_type="ml.g5.8xlarge"

3. 初始化集群命名空间。您的集群管理员应已在命名空间中创建一个 hyperpod-inference 服务账户。

   cluster_namespace="<namespace>"

4. 使用下列选项之一创建 CRD：

   Using Amazon FSx as the model source

   1. 设置 SageMaker 端点名称。

      export SAGEMAKER_ENDPOINT_NAME="deepseek15b-fsx"

   2. 配置要使用的亚马逊 FSx 文件系统 ID。

      export FSX_FILE_SYSTEM_ID="fs-1234abcd"

   3. 以下是使用 Amazon FSx 和 DeepSeek 模型创建终端节点的 yaml 文件示例。

      注意

      对于启用了 GPU 分区的集群，请nvidia.com/gpu替换为相应的 MIG 资源名称，例如。nvidia.com/mig-1g.10gb有关更多信息，请参阅 使用 MIG 提交任务。

      cat <<EOF> deploy_fsx_cluster_inference.yaml
      ---
      apiVersion: inference.sagemaker.aws.amazon.com/v1
      kind: InferenceEndpointConfig
      metadata:
        name: lmcache-test
        namespace: inf-update
      spec:
        modelName: Llama-3.1-8B-Instruct
        instanceType: ml.g5.24xlarge
        invocationEndpoint: v1/chat/completions
        replicas: 2
        modelSourceConfig:
          fsxStorage:
            fileSystemId: $FSX_FILE_SYSTEM_ID
          modelLocation: deepseek-1-5b
          modelSourceType: fsx
        worker:
          environmentVariables:
          - name: HF_MODEL_ID
            value: /opt/ml/model
          - name: SAGEMAKER_PROGRAM
            value: inference.py
          - name: SAGEMAKER_SUBMIT_DIRECTORY
            value: /opt/ml/model/code
          - name: MODEL_CACHE_ROOT
            value: /opt/ml/model
          - name: SAGEMAKER_ENV
            value: '1'
          image: 763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.4.0-tgi2.3.1-gpu-py311-cu124-ubuntu22.04-v2.0
          modelInvocationPort:
            containerPort: 8080
            name: http
          modelVolumeMount:
            mountPath: /opt/ml/model
            name: model-weights
          resources:
            limits:
              nvidia.com/gpu: 1
              # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1
            requests:
              cpu: 30000m
              memory: 100Gi
              nvidia.com/gpu: 1
              # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1
      EOF

   Using Amazon S3 as the model source

   1. 设置 SageMaker 端点名称。

      export SAGEMAKER_ENDPOINT_NAME="deepseek15b-s3"

   2. 配置模型所在的 Amazon S3 存储桶位置。

      export S3_MODEL_LOCATION="deepseek-qwen-1-5b"

   3. 以下是使用 Amazon S3 和 DeepSeek 模型创建终端节点的 yaml 文件示例。

      注意

      对于启用了 GPU 分区的集群，请nvidia.com/gpu替换为相应的 MIG 资源名称，例如。nvidia.com/mig-1g.10gb有关更多信息，请参阅 使用 MIG 提交任务。

      cat <<EOF> deploy_s3_inference.yaml
      ---
      apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1
      kind: InferenceEndpointConfig
      metadata:
        name: $SAGEMAKER_ENDPOINT_NAME
        namespace: $CLUSTER_NAMESPACE
      spec:
        modelName: deepseek15b
        endpointName: $SAGEMAKER_ENDPOINT_NAME
        instanceType: ml.g5.8xlarge
        invocationEndpoint: invocations
        modelSourceConfig:
          modelSourceType: s3
          s3Storage:
            bucketName: $S3_MODEL_LOCATION
            region: $REGION
          modelLocation: deepseek15b
          prefetchEnabled: true
        worker:
          resources:
            limits:
              nvidia.com/gpu: 1
              # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1
            requests:
              nvidia.com/gpu: 1
              # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1
              cpu: 25600m
              memory: 102Gi
          image: 763104351884.dkr.ecr.us-east-2.amazonaws.com/djl-inference:0.32.0-lmi14.0.0-cu124
          modelInvocationPort:
            containerPort: 8000
            name: http
          modelVolumeMount:
            name: model-weights
            mountPath: /opt/ml/model
          environmentVariables:
            - name: PYTHONHASHSEED
              value: "123"
            - name: OPTION_ROLLING_BATCH
              value: "vllm"
            - name: SERVING_CHUNKED_READ_TIMEOUT
              value: "480"
            - name: DJL_OFFLINE
              value: "true"
            - name: NUM_SHARD
              value: "1"
            - name: SAGEMAKER_PROGRAM
              value: "inference.py"
            - name: SAGEMAKER_SUBMIT_DIRECTORY
              value: "/opt/ml/model/code"
            - name: MODEL_CACHE_ROOT
              value: "/opt/ml/model"
            - name: SAGEMAKER_MODEL_SERVER_WORKERS
              value: "1"
            - name: SAGEMAKER_MODEL_SERVER_TIMEOUT
              value: "3600"
            - name: OPTION_TRUST_REMOTE_CODE
              value: "true"
            - name: OPTION_ENABLE_REASONING
              value: "true"
            - name: OPTION_REASONING_PARSER
              value: "deepseek_r1"
            - name: SAGEMAKER_CONTAINER_LOG_LEVEL
              value: "20"
            - name: SAGEMAKER_ENV
              value: "1"
            - name: MODEL_SERVER_TYPE
              value: "vllm"
            - name: SESSION_KEY
              value: "x-user-id"
      EOF

   Using Amazon S3 as the model source

   1. 设置 SageMaker 端点名称。

      export SAGEMAKER_ENDPOINT_NAME="deepseek15b-s3"

   2. 配置模型所在的 Amazon S3 存储桶位置。

      export S3_MODEL_LOCATION="deepseek-qwen-1-5b"

   3. 以下是使用 Amazon S3 和 DeepSeek 模型创建终端节点的 yaml 文件示例。

      cat <<EOF> deploy_s3_inference.yaml
      ---
      apiVersion: inference.sagemaker.aws.amazon.com/v1
      kind: InferenceEndpointConfig
      metadata:
        name: lmcache-test
        namespace: inf-update
      spec:
        modelName: Llama-3.1-8B-Instruct
        instanceType: ml.g5.24xlarge
        invocationEndpoint: v1/chat/completions
        replicas: 2
        modelSourceConfig:
          modelSourceType: s3
          s3Storage:
            bucketName: bugbash-ada-resources
            region: us-west-2
          modelLocation: models/Llama-3.1-8B-Instruct
          prefetchEnabled: false
        kvCacheSpec:
          enableL1Cache: true
      #    enableL2Cache: true
      #    l2CacheSpec:
      #      l2CacheBackend: redis/sagemaker
      #      l2CacheLocalUrl: redis://redis.redis-system.svc.cluster.local:6379
        intelligentRoutingSpec:
          enabled: true
        tlsConfig:
          tlsCertificateOutputS3Uri: s3://sagemaker-lmcache-fceb9062-tls-6f6ee470
        metrics:
          enabled: true
          modelMetrics:
            port: 8000
        loadBalancer:
          healthCheckPath: /health
        worker:
          resources:
            limits:
              nvidia.com/gpu: "4"
            requests:
              cpu: "6"
              memory: 30Gi
              nvidia.com/gpu: "4"
          image: lmcache/vllm-openai:latest
          args:
            - "/opt/ml/model"
            - "--max-model-len"
            - "20000"
            - "--tensor-parallel-size"
            - "4"
          modelInvocationPort:
            containerPort: 8000
            name: http
          modelVolumeMount:
            name: model-weights
            mountPath: /opt/ml/model
          environmentVariables:
            - name: PYTHONHASHSEED
              value: "123"
            - name: OPTION_ROLLING_BATCH
              value: "vllm"
            - name: SERVING_CHUNKED_READ_TIMEOUT
              value: "480"
            - name: DJL_OFFLINE
              value: "true"
            - name: NUM_SHARD
              value: "1"
            - name: SAGEMAKER_PROGRAM
              value: "inference.py"
            - name: SAGEMAKER_SUBMIT_DIRECTORY
              value: "/opt/ml/model/code"
            - name: MODEL_CACHE_ROOT
              value: "/opt/ml/model"
            - name: SAGEMAKER_MODEL_SERVER_WORKERS
              value: "1"
            - name: SAGEMAKER_MODEL_SERVER_TIMEOUT
              value: "3600"
            - name: OPTION_TRUST_REMOTE_CODE
              value: "true"
            - name: OPTION_ENABLE_REASONING
              value: "true"
            - name: OPTION_REASONING_PARSER
              value: "deepseek_r1"
            - name: SAGEMAKER_CONTAINER_LOG_LEVEL
              value: "20"
            - name: SAGEMAKER_ENV
              value: "1"
            - name: MODEL_SERVER_TYPE
              value: "vllm"
            - name: SESSION_KEY
              value: "x-user-id"
      EOF

1. 通过enableL2Cache将enableL1Cache和设置为启用 KV 缓存true。然后，设置l2CacheSpec为redis并使用 Redis 集群 URL l2CacheLocalUrl 进行更新。

     kvCacheSpec:
       enableL1Cache: true
       enableL2Cache: true
       l2CacheSpec:
         l2CacheBackend: <redis | tieredstorage>
         l2CacheLocalUrl: <redis cluster URL if l2CacheBackend is redis >

   注意

   如果 redis 集群与集 HyperPod 群不在同一 Amazon VPC 内，则无法保证传输中的数据会得到加密。

   注意

   如果选择了分层存储，CacheLocalUrl 则不需要 l2。

2. 通过enabled将true设置为 under 来启用智能路由intelligentRoutingSpec。您可以在下方指定要使用的路由策略routingStrategy。如果未指定路由策略，则默认为prefixaware。

   intelligentRoutingSpec:
       enabled: true
       routingStrategy: <routing strategy to use>

3. 通过设置enabled为小true于，启用路由器指标和缓存指标metrics。该port值必须与下的containerPort值相同modelInvocationPort。

   metrics:
       enabled: true
       modelMetrics:
         port: <port value>
       ...
       modelInvocationPort:
         containerPort: <port value>

1. 从集群 ARN 中获取用于 kubectl 身份验证的 Amazon EKS HyperPod 集群名称。

   export EKS_CLUSTER_NAME=$(aws --region $REGION sagemaker describe-cluster --cluster-name $HYPERPOD_CLUSTER_NAME \
     --query 'Orchestrator.Eks.ClusterArn' --output text | \
     cut -d'/' -f2)
   aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $REGION

2. 使用以下选项之一部署 InferenceEndpointConfig 模型：

   Deploy with Amazon FSx as a source

   kubectl apply -f deploy_fsx_luster_inference.yaml

   Deploy with Amazon S3 as a source

   kubectl apply -f deploy_s3_inference.yaml

1. 检查是否已成功部署模型。

   kubectl describe InferenceEndpointConfig $SAGEMAKER_ENDPOINT_NAME -n $CLUSTER_NAMESPACE

2. 检查是否已成功创建端点。

   kubectl describe SageMakerEndpointRegistration $SAGEMAKER_ENDPOINT_NAME -n $CLUSTER_NAMESPACE

3. 测试已部署的端点以验证其是否正常工作。此步骤可确认模型已部署成功且能处理推理请求。

   aws sagemaker-runtime invoke-endpoint \
     --endpoint-name $SAGEMAKER_ENDPOINT_NAME \
     --content-type "application/json" \
     --body '{"inputs": "What is AWS SageMaker?"}' \
     --region $REGION \
     --cli-binary-format raw-in-base64-out \
     /dev/stdout

清除资源

1. 删除推理部署和关联的 Kubernetes 资源。这将停止正在运行的模型容器并移除 SageMaker端点。

   kubectl delete inferenceendpointconfig $SAGEMAKER_ENDPOINT_NAME -n $CLUSTER_NAMESPACE

2. 确认已成功完成清理。

   # # Check that Kubernetes resources are removed
   kubectl get pods,svc,deployment,InferenceEndpointConfig,sagemakerendpointregistration -n $CLUSTER_NAMESPACE

   # Verify SageMaker endpoint is deleted (should return error or empty)
   aws sagemaker describe-endpoint --endpoint-name $SAGEMAKER_ENDPOINT_NAME --region $REGION

问题排查

如果部署无法按预期工作，请使用以下调试命令。

1. 检查 Kubernetes 部署状态。

   kubectl describe deployment $SAGEMAKER_ENDPOINT_NAME -n $CLUSTER_NAMESPACE

2. 检查 InferenceEndpointConfig 状态以查看高级部署状态和任何配置问题。

   kubectl describe InferenceEndpointConfig $SAGEMAKER_ENDPOINT_NAME -n $CLUSTER_NAMESPACE

3. 检查所有 Kubernetes 对象的状态。全面了解命名空间中所有相关的 Kubernetes 资源。这可让您快速了解当前正在运行的内容以及可能缺失的内容。

   kubectl get pods,svc,deployment,InferenceEndpointConfig,sagemakerendpointregistration -n $CLUSTER_NAMESPACE