How to Prevent Duplicate Data with Idempotency Key

What is idempotency ?

Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

When 2 identical requests are sent almost at the same time, your service must handle them accordingly, for example, if a web app sends a duplicate request to book (due to various reasons) a movie ticket, the service must understand which request should be accepted and only process one.

How do we achieve idempotency?

One way to mitigate a duplicate request is by making the clients are sending an idempotency key, basically it’s the identity of a request.

Usually, the key is placed in a request header e.g. x-idempotency-key. To ensure uniqueness the client can send a UUID as a key.

The logic is quite simple

isDuplicateRequest(key) {
    foundKey = repo.getKey(key)
    if (foundKey) {
        repo.saveKey(key)
        return true
    }
    return false
}

Now the pseudo code above will get the job done (almost). If the requests are coming within 2 milliseconds difference, it will trigger a race condition which can result in incorrect validation.

take a look at the code below

isDuplicateRequest(key) {
    foundKey = repo.getKey(key)
    if (foundKey) {
        repo.saveKey(key)  //This process might take more than 10 milliseconds
        return true
    }
    return false
}

if the request comes fast enough it will not detect the previous ones, so how do we make sure that every request is validated correctly?

Preventing a race condition

Rather than fetching known keys, validating them, and saving them after, just save them directly. The datastore needs to have a unique constraint for example in RDBMS a column needs to have a unique constraint.

In theory, we can store the keys in any type of database but ideally, we need the database to be blazingly fast so typically we store it in a transient database such as Redis.

Key expiration

An idempotency key will have an expiration time, usually 24 to 48 hours, to achieve this we can combine Redis’s set with its item score.

public Boolean isUniqueKey(String key) {
        String redisKey = "idempotency_key";
        RedisCommands<String, String> commands = redisConnection.sync();
        Long keyAdded = commands.zadd(redisKey, new Date().getTime() + (1000 * 60 * 60 * 24), key);
        String sNow = String.valueOf(new Date().getTime());
        commands.zremrangebyscore(redisKey, "-inf", sNow);
        return keyAdded > 0;
}

Now the function will guarantee to validate the uniqueness of a key.

Conclusion

The uniqueness of a request is a crucial part of data integrity. An efficient flow should be placed and should not affect the quality of your service.