Lendingclub Corporation

Lendingclub Bank, National Association

Oracle International Corporation

Igor Petrunya

Yana Nikitina

Prateek Demla

Dmitry Kondratenko

Hyunsuk Han

Fernando Lores

Wira Pradjinata

David Taubler

John Daum

Techniques a provided for performing multi-system operations in which changes are asynchronously committed in multiple systems. Metadata about the multi-system operation is injected into the commit logs of one system involved in a multi-system operation. An event stream is generated based on the commit logs of the one system, and is used to drive the operations that one or more other systems need to perform as part of the multi-system operation. A reconciliation system reads the logs of all systems involved in the multi-system operation and determines whether the multi-system operation completed successfully. Techniques are also provided for using machine learning to generate models of normal execution of different types of operations, detect anomalies, pre-emptively send expectation messages, and automatically suggest and/or apply fixes.

Techniques for generating pre-emptive expectation messages

Embodiments provide a batching system that conforms message batches to publication constraints and also to message ordering requirements. An output array of messages is formed from messages received from a plurality of input streams, in which the messages are ordered. The output array preserves the ordering of the messages found in the source input streams. Messages are added from a head of the output array to a batch until addition of a next message to the batch would violate a particular batch processing constraint imposed on the batch. According to embodiments, one or more additional messages are included in the current batch when addition of the one or more additional messages to the batch (a) does not violate the particular batch processing constraint, and (b) continues to preserve the ordering of the messages, in the batch, with respect to the respective ordering of each of the plurality of input streams.

Message stream processor microbatching

Techniques for automatically addressing anomalous behavior

Techniques are provided for recording, in an audit log data store, log records that satisfy one or more audit log rules. Audit log rules may be associated with one or more context attributes. Specifically, based on the context attribute of a given rule, embodiments store, in the audit log data store, additional log records that are associated with the context attribute value from log records that satisfied the rule. Because a context attribute may span multiple systems that implement a multi-system operation, the information in the audit log data store may include cross-system contextual information for changes that are of interest to administrators. The audit log data store may be efficiently queried to provide information regarding multi-system operations because of the targeted nature of the audit log data gathering techniques. Automatically-generated indexes on audit log data provide additional efficiency gains for executing queries over the audit log data.

Multi-system operation audit log

Embodiments facilitate efficient recovery from an inter-stream messaging system failure by using heartbeat messages (HMs) to act as watermarks for message recovery. Embodiments insert HMs into each of the input streams at configurable regular intervals. The inter-stream router determines that a message being routed is a HM, and corresponding output HMs are generated based on the input HM. Embodiments insert a respective output HM into each of the output streams. Information indicating which output HMs have been processed from the output streams is tracked. After a failure of the inter-stream router, embodiments identify a target HM for each input stream, which is the latest HM sourced from the respective input stream that was processed from all of the output streams. After the inter-stream router restarts, the router initiates message routing, from each input stream, at the location of the respective input stream's target HM within the input stream.

Batch checkpointing for inter-stream messaging system

Techniques a provided for communication graph tracking in multi system operation in heterogeneous database systems, including retrieving two or more database commit logs from two or more corresponding heterogeneous databases. For each of the commit logs, a subset of information is extracted, including a correlation identifier, and event stream entries are produced based on that information and are stored. The event stream entries are then retrieved and event database entries are created based on the event stream entries. A request for a communication graph related to a particular correlation identifier is received and the event database is searched for matching entries. Those entries are used to generate a displayable communication graph for the particular correlation identifier, which is then provided in response to the request.

Communication graph tracking of multi-system operations in heterogeneous database systems

When a particular event record requires a particular service, but that service has failed for the event, an error record that includes an aggregate identifier (AgID) is stored for the event in an error queue. Storing an error record in the error queue causes the service to notify a remediation team to fix the failed event. All events with the AgID in the error record are put on hold and an error record is created for each of these events. The remediation team generates a fixed version of the event record and causes the system to retry the failed service for the event based on the fixed version. If the fixed version of the event is successfully processed, then any other events with the same AgID that have error records in the error queue, are routed in order of enqueue time through the event processor to be handled.

Growing community of inventors

Pleasant Hill, CA, United States of America

Igor Petrunya