Amazon Technologies, Inc.

Sandesh Sanjeev

Ankit Kumar

Meng Li

Rachit Jain

Munjal Doshi

Dan Popick

David M Hulme

Sharad Sridhar

Techniques for incremental stream switching in a distributed system with flip events are described. A special-purpose flip event is inserted into an event stream that indicates, to recipients of the stream, whether they are to switch to use of an alternate event stream. The flip event may identify which other event stream to use, how to obtain events of that event stream, when to switch to the other event stream, and/or whether that particular recipient is to switch. The recipients may be selectively or iteratively migrated using flip events targeting different recipients, limiting the blast radius of a switch. Via use of a tracking data structure, event replay and event skips can be avoided.

Incremental stream switching in distributed systems

Features are disclosed for managing multiple hosts that stream sequential data to nodes for propagation. The hosts can include a primary host and a failover host that each store a replica of the sequential data. A computing device can monitor the hosts and identify the occurrence of a failover event for the primary host. For example, the computing device may determine the schema has been modified. The computing device can further determine to failover to the failover host based on the failover event. The computing device can insert a flip task into the stream of sequential data for the primary host. The insertion of the flip task may be automatic. Further, each node reading from the stream of sequential data for the primary host may encounter the flip task and failover to the failover host at the same logical position but a different absolute position.

Managing failover between data streams

Growing community of inventors

Seattle, WA, United States of America

Sandesh Sanjeev