Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Deep Learning Containers for vLLM ARM64 with EFA Support on EC2, ECS, and EKS

Amazon Deep Learning Containers (DLCs) now support vLLM images that are optimized for large language model serving. The vLLM DLC provides a production-ready environment for deploying and serving LLMs with built-in support for EFA (Elastic Fabric Adapter). With vLLM's advanced features and optimizations pre-configured, this specialized container offer an ideal starting point for high-performance, scalable and efficient LLM serving for various use cases, from single-node to multi-node deployments.

All software components in this container are scanned for security vulnerabilities and updated or patched in accordance with Amazon Security best practices. A list of all available Amazon DLCs can be found in our github repo. Get started quickly with Amazon DLCs using the getting-started section in our developer guide. To ensure you're using the latest DLC releases, we invite you to subscribe to our DLC notification mechanism. If you are looking for a DLC to use with SageMaker, please refer to this documentation. For guide on how to use vLLM, checkout vLLM documentation.

Changelog

To learn about latest changes in vLLM DLC, checkout the changelog.

A list of available containers can be found on GitHub.

Security Advisory

Amazon recommends that customers monitor critical security updates in the Amazon Security Bulletin.

Python 3.12 Support

Python 3.12 is supported.

Instance Type Support

The containers support ARM64 instance types.

Amazon Regions support

The containers are available in the following regions:

Build and Test

Known Issues

No known issues so far