Troubleshooting FAQs

The connection is inactive. This is common for streaming operations, where data is not being written to or from the wire for a period of time, so an intermediary party detects the connection as dead and closes it. To prevent this, be sure your application actively downloads or uploads data.

You've closed the HTTP or SDK client. Be sure not to close resources while they are in use.

A misconfigured proxy. Try to bypass any proxies that you've configured to see if it fixes the problem. If this fixes the issue, the proxy is closing your connection for some reason. Research your specific proxy to determine why it's closing the connection.

The configured connection timeout is too low. By default, the connection timeout is 2 seconds in the Amazon SDK for Java 2.x. If you set the connection timeout too low, you may get this error. The recommended connection timeout is 1 second if you make only in-region calls and 3 seconds if you make cross-region requests.

A misconfigured proxy. Try to bypass any proxies that you configured to see if it fixes the problem. If this fixes the issue, the proxy is the reason for the connection timeout. Research your specific proxy to determine why that is happening

Connection leak. You can investigate this by checking LeasedConcurrency , AvailableConcurrency, and MaxConcurrency metrics. If LeasedConcurrency increases until it reaches MaxConcurrency but never decreases, there may be a connection leak. A common cause of a leak is because a streaming operation—such as a S3 getObject method—is not closed. We recommend that your application read all data from the input stream as soon as possible and close the input stream afterwards. The following chart shows what SDK metrics might look like for connection leak.

Connection pool starvation. This can happen if your request rate is too high and the connection pool size that has been configured cannot meet the request demand. The default connection pool size is 50, and when the connections in the pool reach the maximum, the HTTP client queues incoming requests until connections become available. The following chart shows what SDK metrics might look like for connection pool starvation.

To mitigate this issue, consider taking any of the following actions.

By increasing the maximum number of connections, client throughput can increase (unless the network interface is already fully utilized). However, you can eventually hit operation system limitations on the number of file descriptors used by the process. If you already fully use your network interface or cannot further increase your connection count, try increasing the acquire timeout. With the increase, you gain extra time for requests to acquire a connection before timing out. If the connections don't free up, the subsequent requests will still timeout.

If you are unable to fix the issue by using the first two mechanisms, slow the request rate by trying the following options.

I/O Threads are too busy. This only applies if you are using an asynchronous SDK client with NettyNioAsyncHttpClient. If the AvailableConcurrency metric is not low—indicating that connections are available in the pool—but ConcurrencyAcquireDuration is high, it might be because I/O threads are not able to handle the requests. Be sure you are not passing Runnable:run as a future completion executor and performing time-consuming task in the response future completion chain since this can block an I/O thread. If that is not the case, consider increasing the number of I/O threads by using the eventLoopGroupBuilder method. For reference, the default number of I/O threads for a NettyNioAsyncHttpClient instance is twice the number of CPU cores of the host.

High TLS handshake latency. If your AvailableConcurrency metric is near 0 and LeasedConcurrency is lower than MaxConcurrency, it might be because the TLS handshake latency is high. The following chart shows what SDK metrics might look like for high TLS handshake latency.

For HTTP clients offered by the Java SDK that are not based on CRT, try enabling TLS logs to troubleshoot TLS issues. For the Amazon CRT-based HTTP client, try enabling Amazon CRT logs. If you see that the Amazon endpoint seems to take a long time to perform a TLS handshake, you should contact the affected service.

the class does not exist in the classpath because the JAR is missing or the wrong version of the JAR is on the classpath.

the class failed to load because its static initializer threw an exception.

A proxy or intermediary party modifies the request. For example, a proxy or load balancer might modify a header, path or query string that was signed by the SDK.

The service and SDK differ in the way they encode the request when each generates the string to sign.

The SDK client was closed prematurely. The SDK only closes the connection pool when the associated client is closed. Be sure not to close resources while they are in use.

A java.lang.Error was thrown. Errors such as OutOfMemoryError cause an Apache HTTP connection pool to shut down. Examine your logs for error stack traces. Also review your code for places where it catches Throwables or Errors but swallows the output that prevents the error from surfacing. If your code does not report errors, rewrite the code so information is logged. The logged information helps determine the root cause of the error.

You attempted to use the credentials provider returned from DefaultCredentialsProvider#create() after it was closed. DefaultCredentialsProvider#create returns a singleton instance, so if it's closed and your code calls the resolveCredentials method, the exception is thrown after cached credentials (or token) expire.

Check your code for places where the DefaultCredentialsProvider is closed, as shown in the following examples.

Create a new, non-singleton instance by calling DefaultCredentialsProvider.builder().build() if your code has closed the singleton instance and you need to resolve credentials by using a DefaultCredentialsProvider.