Improve network performance between EC2 instances with ENA Express

Benefits of ENA Express

• Increases the maximum bandwidth a single flow can use from 5 Gbps up to 25 Gbps within the Availability Zone, up to the aggregate instance limit.

• Reduces tail latency of network traffic between EC2 instances, especially during periods of high network load.

• Detects and avoids congested network paths.

• Handles some tasks directly in the network layer, such as packet reordering on the receiving end, and most retransmits that are needed. This frees up the application layer for other work.

• Processes can benefit from increased maximum single flow bandwidth from 5 Gbps up to 25 Gbps within the same Availability Zone, up to the aggregate instance limit. For example, if a specific instance type supports up to 12.5 Gbps, the single flow bandwidth is also limited to 12.5 Gbps.

• Longer running processes should experience reduced tail latency during periods of network congestion.

• Processes can benefit from a smoother and more standard distribution for network response times.

• Both sending and receiving instance types are supported. See the Supported instance types for ENA Express table for more information.

• Both sending and receiving instances must have ENA Express configured. If there are differences in the configuration, you can run into situations where traffic defaults to standard ENA transmission. The following scenario shows what can happen.

  Scenario: Differences in configuration

  Instance	ENA Express Enabled	UDP uses ENA Express
  Instance 1	Yes	Yes
  Instance 2	Yes	No

  In this case, TCP traffic between the two instances can use ENA Express, as both instances have enabled it. However, since one of the instances does not use ENA Express for UDP traffic, communication between these two instances over UDP uses standard ENA transmission.

• The sending and receiving instances must run in the same Availability Zone.

• The network path between the instances must not include middleware boxes. ENA Express doesn't currently support middleware boxes.

• (Linux instances only) To utilize full bandwidth potential, use driver version 2.2.9 or higher.

• (Linux instances only) To produce metrics, use driver version 2.8 or higher.

• If your instance uses jumbo frames, run the following command to set your maximum transmission unit (MTU) to 8900.

  [ec2-user ~]$ sudo ip link set dev eth0 mtu 8900

• Increase the receiver (Rx) ring size, as follows:

  [ec2-user ~]$ ethtool -G device rx 8192

• To maximize ENA Express bandwidth, configure your TCP queue limits as follows:

  1. Set the TCP small queue limit to 1MB or higher. This increases the amount of data that's queued for transmission on a socket.

     sudo sh -c 'echo 1048576 > /proc/sys/net/ipv4/tcp_limit_output_bytes'

  2. Disable byte queue limits on the eth device if they're enabled for your Linux distribution. This increases data queued for transmission for the device queue.

     sudo sh -c 'for txq in /sys/class/net/eth0/queues/tx-*; do echo max > ${txq}/byte_queue_limits/limit_min; done'

     Note

     The ENA driver for the Amazon Linux distribution disables byte queue limits by default.