View a markdown version of this page

Troubleshooting distributed KV caching - Amazon ElastiCache
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting distributed KV caching

The BLOB.RDMA_* commands return standard Valkey error replies. Client libraries with remote direct memory access (RDMA) support handle most of them, such as session setup and memory region errors. If these errors continue, make sure that you use a supported client version. For the full list of server errors, see Errors at run time on the valkey.io website.

Distributed KV caching errors and symptoms
Error or symptomCauseAction
ERR RDMA read failed or ERR RDMA write failed from BLOB.RDMA_SET or BLOB.RDMA_GET, after BLOB.RDMA_HELLO succeeds The node can't reach the client over Elastic Fabric Adapter (EFA). Common causes are a missing self-referencing inbound or outbound rule on the security group, the security group missing from one of the EFA interfaces of the client, or a client in a different Availability Zone. Check both self-referencing rules, attach the security group to every efa and efa-only interface on the client, and make sure that the client is in the Availability Zone of the cluster. For more information, see Step 2: Create a security group and Step 6: Set up the client hosts.
fi_mr_reg: Cannot allocate memory, or the client can't open the efa-direct fabric The memlock limit of the client is too low, or the instance type doesn't support EFA RDMA write. Set memlock to unlimited, and use an instance type that supports RDMA write. For more information, see Step 6: Set up the client hosts.
OOM command not allowed when used disk space > 'largeobj.disk-maxmemory'. or OOM command not allowed when used memory > 'maxmemory'., or writes fail and the Evictions metric doesn't increase The node is full, and eviction can't free enough space. With a volatile-* policy, such as volatile-lru in the default parameter group, keys with no time to live (TTL) are never evicted. If maxmemory-policy is noeviction, change it to allkeys-lru, and use a dedicated cluster. With a volatile-* policy, set a TTL on your keys (for LMCache, the ttl_seconds field of the Valkey L2 adapter), or change the policy to allkeys-lru. Keys that LMCache wrote before you set ttl_seconds have no TTL. If eviction is already on, the cluster needs more storage. For more information, see Eviction and memory management and Sizing and scaling.
ERR disk staging buffer pool exhausted; retry later Too many reads and writes are in progress on the node at the same time. This is a temporary concurrency limit, not a lack of memory. Retry the request. If the error continues, decrease the number of concurrent requests to each node.
Objects disappear before they are read again The node evicts objects to make space. Check the Evictions and LargeObjectDiskUtilization metrics. If both are high, the cluster needs more storage. For more information, see Sizing and scaling.
Registering the L1 arena (n MiB) with the fabric failed in the LMCache log With the rdma transport, each LMCache worker registers the whole L1 tier with its EFA device. The registered memory is more than the device allows, or the locked-memory limit of the LMCache server is too low. Decrease num_workers or --l1-size-gb, or back the L1 tier with 2 MiB huge pages (--l1-use-hugepages, and keep --no-l1-use-lazy). For huge pages, reserve --l1-size-gb × 512 pages of 2 MiB on the non-uniform memory access (NUMA) node of the EFA devices before LMCache starts, for example in /sys/devices/system/node/noden/hugepages/hugepages-2048kB/nr_hugepages. Reserve the pages soon after boot, because free memory becomes fragmented. If the pool is too small, LMCache stops at start. Set the locked-memory limit of the LMCache server to unlimited. For more information, see Step 8: Connect your inference engine.
The LMCache log shows transport=tcp, or LargeObjectBasedCmds stays at 0 The transport field of the LMCache Valkey L2 adapter isn't set to rdma, or the installed LMCache version doesn't support the rdma transport. Set "transport": "rdma", and install LMCache 0.5.8rc1 or later. For more information, see Inference engine connectors.
All objects on a shard are gone The node was replaced, patched, or rebooted, or it failed. This is expected. For more information, see Resilience and data persistence. No action is necessary. Treat a missing key as a cache miss, and reload or recompute the data.

For general ElastiCache troubleshooting, see Common troubleshooting steps and best practices with ElastiCache.