Google Inc.

Haifeng Jiang

Ashish Gupta

Manpreet Singh

Rajagopal Ananthanarayanan

Monica D Chawathe

Tianhao Qiu

Sumit Das

Venkatesh Basker

Namit Sikka

Robert C Keller

Fenglin Liao

Shan He

Yuri Vasilevski

Vinayak Ganeshan

Navin Reginald Melville

Systems and methods are disclosed herein for providing fault tolerant processing of events. The system includes multiple consensus computers configured to communicate with one another and multiple event processors configured to process data such as events. Each consensus computer is further configured to receive a request to process a unit of data from an event processor. A consensus computer communicates with at least one other consensus computer to reach consensus as to whether the unit of data has previously been assigned to an event processor for processing. Then, a consensus computer sends a message to the event processor that sent the inquiry including instructions to either process the unit of data or not process the unit of data. Because the consensus computers determine whether a unit of data has previously been assigned to an event processor, the system ensures that an event is not processed more than once.

Disaster-proof event data processing

A method for handling data includes receiving a data record. The method further includes adding the count value of the data record to a count value of the top row of a stack when the data record field values are included among the field values of the top row. The method also includes rolling up the top row when less than all of the data record field values are included among the top row field values and the count value of the top row is less than a threshold value. The method further includes outputting the top row and inserting the data record onto the stack as the new top row. The method may also include removing the top row and adding its count value to the count value of a new top row. A system for handling data includes a streaming data handler and a sorter.

Streaming roll-ups on statistical data using a stack

Aspects of the disclosure relate generally to reducing latency for consensus in geographically distributed disaster-safe persistent data-store systems. These distributed systems may include registry system having redundant storage for maintaining the system status. Each registry system may include a server and a storage component. Consensus may be achieved by querying all of other servers of the registry system. In one example, the consensus data may be sharded into independent small groups. This may allow for multiple consensus transactions to be generated and run in parallel, which, in turn may reduce the latency. In addition, or alternatively, requests to a server to write or otherwise change the data-store may be batched at the server side. Thus, for the consensus, the server need only communicate with the other servers only once per batch. This may also reduce the latency of the distributed system.

Latency reduction in distributed computing systems

Aspects of the disclosure relate generally to a distributed event joining system for joining primary and secondary events based on a common key. In one example, a continuous stream of primary events may be submitted by a plurality of ad serving computers for storage at a cache layer. Each primary event may be associated with a primary event key. After a time T, the stream of primary events may be sent to long term memory. The system may also receive secondary events, each associated with a primary key. The primary key may be used to identify a primary event in the cache layer. If the primary event is not available in the cache layer, the long term memory may be searched for the primary key. Paired events may then be submitted to a distributed joined event storage system.

Multi-level caching event lookup

Aspects of the disclosure relate generally to a distributed event joining system for joining primary and secondary events based on a common key. In one example, a continuous stream of primary events may be stored at a cache layer. Each primary event may be associated with a primary event key which may be used to generate an index map for the cache layer. After a time T, the stream of primary events may be sent to long term memory. The system may also receive secondary events, each associated with a primary key. The primary key may be used to identify a primary event in the cache layer. If the primary event is not available in the cache layer, the long term memory may be searched for the primary key. Paired events may then be submitted to a distributed joined event storage system.

Real-time log joining on a continuous stream of events that are approximately ordered

A device may store information associated with a group of items in a database. The information associated with a particular item may include a group of versions of a particular attribute. A particular version of the particular attribute may include a value associated with the particular attribute and a timestamp. The device may receive a query that specifies a time; determine that the query is associated with the particular item; and determine, based on the specified time, which version of the particular attribute is associated with the query. The device may determine which version of the particular attribute is associated with the query by identifying, based on the timestamps associated with the versions of the particular attribute, a version of the particular attribute that is a newest version, of the versions of the particular attribute that are associated with timestamps that are before or concurrent with the specified time.

Versioned database cache

A log file processing system includes a data storage device and a processing circuit configured to retrieve log file data from the data storage device and to create a bundle including a plurality of data chunks using the log file data. The processing circuit is configured to assign the bundle to a first processing pipeline of a set of multiple processing pipelines in response to a request from the first processing pipeline. The processing circuit is configured to store a global available timestamp for the bundle upon assigning the bundle to the first processing pipeline. The global timestamp is a time threshold before which the bundle is ineligible to be assigned to any of the multiple processing pipelines.

Growing community of inventors

Sunnyvale, CA, United States of America

Haifeng Jiang