Pattern for integrating primary and secondary data stores in a sharded data domain

Abstract

Example systems and methods for integrating a sharded primary data store (e.g., a source-of-truth relational database management system), a secondary data store (e.g., external cache) and an external global index are described. The approach implements a modified read-through/write-through data access pattern that integrates read and write flows in order to support high-concurrency environments while maintaining immediate consistency between all three stores. Writes are handled using a three-phase flow that avoids concurrency-related race conditions and the need to block in the secondary store in order to maintain cross-store consistency. Reads are never dirty and will repair the secondary store as needed, presenting an immediately consistent view of data to application consumers.