Configuring distributed KV caching
Distributed key-value (KV) caching has two configuration surfaces: the parameter group of the cluster, and the LMCache connector settings that you set in your inference engine. For the cluster settings, see Cluster requirements. For general information about managing node-based clusters, see Managing your node-based cluster in ElastiCache.
Parameters
ElastiCache runs the large object module in tiered mode with Elastic Fabric Adapter (EFA), and manages its settings for you. You can't change them, and the CONFIG and MODULE LIST commands aren't available. To view the module state, use INFO largeobj.
You can change the following parameter with a custom parameter group in the valkey9 family. The parameter group must also set cluster-enabled to yes.
| Parameter | Recommended value | Description |
|---|---|---|
maxmemory-policy | allkeys-lru | Controls which keys can be evicted. For more information, see Eviction and memory management. |
For the parameters of the valkey9 family, see Valkey 9.1 and Valkey 9.0 parameter changes.
Eviction and memory management
When a node is full, a write can evict other keys to make space. For large objects, the node is full when its Non-Volatile Memory Express (NVMe) storage is full. For values that you write with SET, the node is full when its memory is full. The maxmemory-policy parameter controls which keys can be evicted.
| maxmemory-policy | Keys that can be evicted |
|---|---|
allkeys-* | Any key |
volatile-* | Only keys with a time to live (TTL). Keys with no TTL are never evicted, so writes fail when the node is full. |
noeviction | None. The write fails with an OOM error. |
If you don't choose a parameter group, the cluster uses the default parameter group of the valkey9 family. As for other node types, this group uses volatile-lru. We recommend a custom parameter group with allkeys-lru, or the ttl_seconds field of LMCache. Choose one of the following options before you create the cluster:
Use the default parameter group – Set the
ttl_secondsfield of the LMCache Valkey L2 adapter, so that every LMCache key has a TTL and can be evicted. Set the TTL to more than the longest session.Use a custom parameter group with
allkeys-lru– LMCache keys don't need a TTL. Use a dedicated cluster, becauseallkeys-lrucan evict the keys of other applications.
Objects are always evicted in least recently used (LRU) order, whatever the policy. After the cluster fills, make sure that the Evictions metric increases. For more information about eviction, see Memory and eviction
LMCache connector settings
LMCache in multiprocess mode has two parts on each GPU host: the LMCache server, which you configure with lmcache server options and a --l2-adapter JSON object with "type": "valkey", and the engine connector. For vLLM, you configure the connector with the --kv-transfer-config JSON object. The following table describes the settings that connect them to the cluster. For a complete example, see Step 8: Connect your inference engine.
| Setting | Value | Description |
|---|---|---|
|
|
|
|
|
|
|
Suggested for LMCache 0.5.8rc1 or later: about half of the memory of the non-uniform memory access (NUMA) node where LMCache runs |
|
|
(flag) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
From Amazon Secrets Manager |
|
|
|
|
|
|
|
|
More than the longest session |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Environment variable. Required for every engine and LMCache process, so that all hosts compute the same cache keys. |
Every GPU host that shares the cluster must use the same model and revision, TP layout, KV cache data type, LMCache version, --hash-algorithm, and PYTHONHASHSEED=0. A difference causes cache misses or wrong KV state, with no error.
For all LMCache server options and Valkey L2 adapter fields, see Valkey L2 adapter
Request routing
You can keep your existing request router. Request routers don't count the KV state in the cluster, and you don't need to change your routing to use it. Route requests to GPU hosts in the Availability Zone of the cluster. For more information, see Cache placement and sharing.