Amazon Deep Learning Containers for vLLM with EFA Support on SageMaker - Amazon Deep Learning Containers
ChangelogSecurity AdvisoryPython 3.12 SupportInstance Type SupportAmazon Regions supportBuild and TestKnown Issues
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Deep Learning Containers for vLLM with EFA Support on SageMaker

Amazon Deep Learning Containers (DLCs) now support vLLM images that are optimized for large language model serving on Amazon SageMaker. The vLLM DLC provides a production-ready environment for deploying and serving LLMs with advanced features like PagedAttention for efficient memory management and continuous batching. With vLLM's advanced features and optimizations pre-configured, this specialized container offers an ideal starting point for high-performance, scalable and efficient LLM serving on SageMaker.

All software components in this container are scanned for security vulnerabilities and updated or patched in accordance with Amazon Security best practices. A list of all available Amazon DLCs can be found in our github repo. Get started quickly with Amazon DLCs using the getting-started section in our developer guide. To ensure you're using the latest DLC releases, we invite you to subscribe to our DLC notification mechanism. For guide on how to use vLLM, checkout vLLM documentation.

Changelog

To learn about latest changes in vLLM DLC, checkout the changelog.

A list of available containers can be found on GitHub.

vLLM v0.11.1 and above utilizes CUDA 12.9 which is only compatible with Nvidia Driver 535 and above (550 preferred). To deploy the container on SageMaker platform, please specify al2-ami-sagemaker-inference-gpu-3-1 as the ProductionVariant.

Security Advisory

All software components in these images are scanned for security vulnerabilities and updated or patched in accordance with Amazon Security best practices.

Amazon recommends that customers monitor critical security updates in the Amazon Security Bulletin.

Python 3.12 Support

Python 3.12 is supported.

Instance Type Support

The containers support x86_64 instance types.

Amazon Regions support

The containers are available in the following regions:

Region

Code

US East (N. Virginia)

us-east-1

US East (Ohio)

us-east-2

US West (N. California)

us-west-1

US West (Oregon)

us-west-2

Asia Pacific (Hong Kong)

ap-east-1

Asia Pacific (Mumbai)

ap-south-1

Asia Pacific (Hyderabad)

ap-south-2

Asia Pacific (Tokyo)

ap-northeast-1

Asia Pacific (Seoul)

ap-northeast-2

Asia Pacific (Osaka)

ap-northeast-3

Asia Pacific (Singapore)

ap-southeast-1

Asia Pacific (Sydney)

ap-southeast-2

Asia Pacific (Jakarta)

ap-southeast-3

Asia Pacific (Melbourne)

ap-southeast-4

Asia Pacific (Malaysia)

ap-southeast-5

Asia Pacific (Thailand)

ap-southeast-7

Canada (Central)

ca-central-1

Canada (Calgary)

ca-west-1

EU (Frankfurt)

eu-central-1

EU (Zurich)

eu-central-2

EU (Ireland)

eu-west-1

EU (London)

eu-west-2

EU (Paris)

eu-west-3

EU (Milan)

eu-south-1

EU (Spain)

eu-south-2

EU (Stockholm)

eu-north-1

Middle East (Bahrain)

me-south-1

Middle East (UAE)

me-central-1

Israel (Tel Aviv)

il-central-1

SA (Sau Paulo)

sa-east-1

AF South (Cape Town)

af-south-1

Mexico (Central)

mx-central-1

China (Beijing)

cn-north-1

China (Ningxia)

cn-northwest-1

Build and Test

  • Built on: c5.18xlarge

  • Tested on: p4d.24xlarge, p5.48xlarges

  • Tested with deepseek-ai/DeepSeek-R1-Distill-Qwen-32B model, single-node and multi-node serving configurations

Known Issues

No known issues so far