How to automatically Retry DynamoDB Write on ProvisionedThroughputExceededException

Upasana | July 27, 2018 | 3 min read | 1,314 views | Amazon DynamoDB

Amazon DynamoDB requires us to provision read and write capacity in advance, otherwise if exceeded will start throwing ProvisionedThroughputExceededException. So lets say you have configured write capacity of 1 record per second, and all of a sudden lots of write requests queue up, you will start receiving this exception in your code. Please be noted that Amazon DynamoDB maintains a sliding window buffer of 5 minutes, i.e. if you do not use any write capacity in last 5 minutes, you can use it in one go, so effectively a spike of 300 writes can be handled gracefully if you have un-utilized quota of last 5 minutes

ProvisionedThroughputExceededException

Your request rate is too high. The AWS SDKs for DynamoDB automatically retry requests that receive this exception. Your request is eventually successful, unless your retry queue is too large to finish. Reduce the frequency of requests and use exponential backoff. For more information, go to Error Retries and Exponential Backoff in the Amazon DynamoDB Developer Guide.

Gracefully handling the failures using Spring Retry with exponential back-off

Spring Retry module provides us a convenient mechanism to handle retry execution with exponential back offs.

You need to make the below changes in your project’s build.gradle file:

build.gradle

compile group: 'org.springframework.retry', name: 'spring-retry', version: '1.1.2.RELEASE'

Now add @EnableRetry annotation to your main Application class.

/src/main/java/com/foo/Application.java

@SpringBootApplication
@EnableRetry    (1)
public class Application {

    public static void main(String[] args) {
        new SpringApplicationBuilder(Application.class).run(args);
    }

}

1	This line is important

Finally we need to apply @Retryable annotation to the method that we want to retry automatically in case of certain exceptions (ProvisionedThroughputExceededException in this case).

/src/main/java/com/foo/ViewsService.java

private static final Logger logger = LoggerFactory.getLogger(ViewService.class);

@Retryable(maxAttempts = 5, value = {ProvisionedThroughputExceededException.class},
        backoff = @Backoff(value = 5000, multiplier = 2))           (1)
public void incrementViews(String app, String id, long delta) {
    customDao.incrementViews(app, id, delta);
}

@Recover
public void recover(ProvisionedThroughputExceededException exception) {
    logger.error("All retries failed - recover from exception ", exception);
}

1	method `incrementViews` will be retried 5 times (at 5, 10, 20, 40, 80 seconds interval) when `ProvisionedThroughputExceededException` is thrown by Dao layer.

That’s all we need to do. Exponential backoff will make sure that subsequent retries succeed.

Exponential backoff

An exponential back-off is an algorithm that uses a pre-defined process to gradually, multiplicatively decrease the rate of a given operation to find a more acceptable rate.

Other options worth considering

There are other options worth considering when you are integrating AWS DynamoDB in your application, which will increase fault tolerance of your application. Here they are:

Using SQS for queuing all requests

Amazon SQS is a simple queue service that can be added to your application to make it more resilient to failures. What you can do is put the write requests into SQS and a separate job can poll records from SQS at a limited rate and insert them into Dynamo Db without the fear of having ProvisionedThroughputExceededException. But this approach may not work well where your client needs a synchronous response to the requests.