Neptune Best Practices Using openCypher and Bolt - Amazon Neptune
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Neptune Best Practices Using openCypher and Bolt

Follow these best practices when using the openCypher query language and Bolt protocol with Neptune. For information about using openCypher in Neptune, see Accessing the Neptune Graph with openCypher.

Prefer directed to bi-directional edges in queries

When Neptune performs query optimizations, bi-directional edges make it difficult to create optimal query plans. Sub-optimal plans require the engine to do unnecessary work and result in poorer performance.

Therefore, use directed edges rather than bi-directional ones whenever possible. For example, use:

MATCH p=(:airport {code: 'ANC'})-[:route]->(d) RETURN p)

instead of:

MATCH p=(:airport {code: 'ANC'})-[:route]-(d) RETURN p)

Most data models don't actually need to traverse edges in both directions, so queries can achieve significant performance improvements by making the switch to using directed edges.

If your data model does require traversing bi-directional edges, make the first node (left-hand side) in the MATCH pattern the node with the most restrictive filtering.

Take the example, "Find me all the routes to and from the ANC airport". Write this query to start at the ANC airport, like this:

MATCH p=(src:airport {code: 'ANC'})-[:route]-(d) RETURN p

The engine can perform the minimal amount of work to satisfy the query, because the most restricted node is placed as the first node (left-hand side) in the pattern. The engine can then optimize the query.

This is far preferable than filtering the ANC airport at the end of the pattern, like this:

MATCH p=(d)-[:route]-(src:airport {code: 'ANC'}) RETURN p

When the most restricted node is not placed first in the pattern, the engine must perform additional work because it can't optimize the query and has to perform additional lookups to arrive at the results.

Neptune does not support multiple concurrent queries in a transaction

Although the Bolt driver itself allows concurrent queries in a transaction, Neptune does not support multiple queries in a transaction running concurrently. Instead, Neptune requires that multiple queries in a transaction be run sequentially, and that the results of each query be completely consumed before the next query is initiated.

The example below shows how to use Bolt to run multiple queries sequentially in a transaction, so that the results of each one are completely consumed before the next one begins:

final String query = "MATCH (n) RETURN n"; try (Driver driver = getDriver(HOST_BOLT, getDefaultConfig())) { try (Session session = driver.session(readSessionConfig)) { try (Transaction trx = session.beginTransaction()) { final Result res_1 = trx.run(query); Assert.assertEquals(10000, res_1.list().size()); final Result res_2 = trx.run(query); Assert.assertEquals(10000, res_2.list().size()); } } }

Create a new connection after failover

In case of a failover, the Bolt driver can continue connecting to the old writer instance rather than the new active one, because the DNS name resolved to a specific IP address.

To prevent this, close and then reconnect the Driver object after any failover.

Connection handling for long-lived applications

When building long-lived applications, such as those running within containers or on Amazon EC2 instances, instantiate a Driver object once and then reuse that object for the lifetime of the application. The Driver object is thread safe, and the overhead of initializing it is considerable.

Connection handling for Amazon Lambda

Bolt drivers are not recommended for use within Amazon Lambda functions, because of their connection overhead and management requirements. Use the HTTPS endpoint instead.

Close driver objects when you're done

Be sure to close the client when you are finished with it, so that the Bolt connections are closed by the server and all resources associated with the connections are released. This happens automatically if you close the driver using driver.close().

If the driver is not closed properly, Neptune terminates all idle Bolt connections after 20 minutes, or after 10 days if you are using IAM authentication.

Neptune supports no more than 1000 concurrent Bolt connections. If you don't explicitly close connections when you're done with them, and the number of live connections reaches that limit of 1000, any new connection attempts fail.

Use explicit transaction modes for reading and writing

When using transactions with Neptune and the Bolt driver, it is best to explicitly set the access mode for both read and write transactions to the right settings.

Read-only transactions

For read-only transactions, if you don't pass in the appropriate access mode configuration when building the session, the default isolation level is used, which is mutation query isolation. As a result, it's important for read-only transactions to set the access mode to read explicitly.

Auto-commit read transaction example:

SessionConfig sessionConfig = SessionConfig .builder() .withFetchSize(1000) .withDefaultAccessMode(AccessMode.READ) .build(); Session session = driver.session(sessionConfig); try { (Add your application code here) } catch (final Exception e) { throw e; } finally { driver.close() }

Read transaction example:

Driver driver = GraphDatabase.driver(url, auth, config); SessionConfig sessionConfig = SessionConfig .builder() .withDefaultAccessMode(AccessMode.READ) .build(); driver.session(sessionConfig).readTransaction( new TransactionWork<List<String>>() { @Override public List<String> execute(org.neo4j.driver.Transaction tx) { (Add your application code here) } } );

In both cases, SNAPSHOT isolation is achieved using Neptune read-only transaction semantics.

Because read replicas only accept read-only queries, any query submitted to a read replica runs under SNAPSHOT isolation semantics.

There are no dirty reads or non-repeatable reads for read-only transactions.

Read-only transactions

For mutation queries, there are three different mechanisms to create a write transaction, each of which is illustrated below:

Implicit write transaction example:

Driver driver = GraphDatabase.driver(url, auth, config); SessionConfig sessionConfig = SessionConfig .builder() .withDefaultAccessMode(AccessMode.WRITE) .build(); driver.session(sessionConfig).writeTransaction( new TransactionWork<List<String>>() { @Override public List<String> execute(org.neo4j.driver.Transaction tx) { (Add your application code here) } } );

Auto-commit write transaction example:

SessionConfig sessionConfig = SessionConfig .builder() .withFetchSize(1000) .withDefaultAccessMode(AccessMode.Write) .build(); Session session = driver.session(sessionConfig); try { (Add your application code here) } catch (final Exception e) { throw e; } finally { driver.close() }

Explicit write transaction example:

Driver driver = GraphDatabase.driver(url, auth, config); SessionConfig sessionConfig = SessionConfig .builder() .withFetchSize(1000) .withDefaultAccessMode(AccessMode.WRITE) .build(); Transaction beginWriteTransaction = driver.session(sessionConfig).beginTransaction(); (Add your application code here) beginWriteTransaction.commit(); driver.close();
Isolation levels for write transactions
  • Reads made as part of mutation queries are run under READ COMMITTED transaction isolation.

  • There are no dirty reads for reads made as part of mutation queries.

  • Records and ranges of records are locked when reading in a mutation query.

  • When a range of the index has been read by a mutation transaction, there is a strong guarantee that this range will not be modified by any concurrent transactions until the end of the read.

Mutation queries are not thread safe.

For conflicts, see Conflict Resolution Using Lock-Wait Timeouts.

Mutation queries are not automatically retried in case of failure.

Retry logic for exceptions

For all exceptions that allow a retry, it is generally best to use an exponential backoff and retry strategy that provides progressively longer wait times between retries so as to better handle transient issues such as ConcurrentModificationException errors. The following shows an example of an exponential backoff and retry pattern:

public static void main() { try (Driver driver = getDriver(HOST_BOLT, getDefaultConfig())) { retriableOperation(driver, "CREATE (n {prop:'1'})") .withRetries(5) .withExponentialBackoff(true) .maxWaitTimeInMilliSec(500) .call(); } } protected RetryableWrapper retriableOperation(final Driver driver, final String query){ return new RetryableWrapper<Void>() { @Override public Void submit() { log.info("Performing graph Operation in a retry manner......"); try (Session session = driver.session(writeSessionConfig)) { try (Transaction trx = session.beginTransaction()) { trx.run(query).consume(); trx.commit(); } } return null; } @Override public boolean isRetryable(Exception e) { if (isCME(e)) { log.debug("Retrying on exception.... {}", e); return true; } return false; } private boolean isCME(Exception ex) { return ex.getMessage().contains("Operation failed due to conflicting concurrent operations"); } }; } /** * Wrapper which can retry on certain condition. Client can retry operation using this class. */ @Log4j2 @Getter public abstract class RetryableWrapper<T> { private long retries = 5; private long maxWaitTimeInSec = 1; private boolean exponentialBackoff = true; /** * Override the method with custom implementation, which will be called in retryable block. */ public abstract T submit() throws Exception; /** * Override with custom logic, on which exception to retry with. */ public abstract boolean isRetryable(final Exception e); /** * Define the number of retries. * * @param retries -no of retries. */ public RetryableWrapper<T> withRetries(final long retries) { this.retries = retries; return this; } /** * Max wait time before making the next call. * * @param time - max polling interval. */ public RetryableWrapper<T> maxWaitTimeInMilliSec(final long time) { this.maxWaitTimeInSec = time; return this; } /** * ExponentialBackoff coefficient. */ public RetryableWrapper<T> withExponentialBackoff(final boolean expo) { this.exponentialBackoff = expo; return this; } /** * Call client method which is wrapped in submit method. */ public T call() throws Exception { int count = 0; Exception exceptionForMitigationPurpose = null; do { final long waitTime = exponentialBackoff ? Math.min(getWaitTimeExp(retries), maxWaitTimeInSec) : 0; try { return submit(); } catch (Exception e) { exceptionForMitigationPurpose = e; if (isRetryable(e) && count < retries) { Thread.sleep(waitTime); log.debug("Retrying on exception attempt - {} on exception cause - {}", count, e.getMessage()); } else if (!isRetryable(e)) { log.error(e.getMessage()); throw new RuntimeException(e); } } } while (++count < retries); throw new IOException(String.format( "Retry was unsuccessful.... attempts %d. Hence throwing exception " + "back to the caller...", count), exceptionForMitigationPurpose); } /* * Returns the next wait interval, in milliseconds, using an exponential backoff * algorithm. */ private long getWaitTimeExp(final long retryCount) { if (0 == retryCount) { return 0; } return ((long) Math.pow(2, retryCount) * 100L); } }