Sap Se

Daniel Bos

Tobias Karpstein

Hui Li

Stefan Rau

Florian Maier

Peter Schoenau

Joao Pedro Duro Reis

Nico Gaertner

Dan Liu

Marcus Roland Schwarz

Provided are systems and methods for transforming data from a format of a source system into a format of a target system based on a dedicated virtual machine. In one example, a method may include storing programming code which comprises logic for transforming data from a source system to a target format of a target system, compiling the programming code into bytecode, generating a work order which includes a payload of data from the source system and the bytecode, reading the work order via a virtual machine and executing the bytecode on the payload of data via the virtual machine, wherein the virtual machine transforms the payload of data from the source format to the target format based on the bytecode, and loading the transformed payload of data into the target system.

Virtual machine for ETL transformations

Disclosed herein are system, method, and computer program product embodiments for accessing a semantic change data capture (CDC). An embodiment operates by receiving a request for changes made to a source system from a consumer at a target system. The embodiment determines a high watermark stored in a consumer registration table based on the consumer. The embodiment further determines a set of open transactions on an enhanced data table. The embodiment then determines a set of changes made to the source system based on the high watermark and the set of open transactions. The embodiment transmits the set of changes to the consumer at the target system. The embodiment then receives a confirmation from the consumer at the target system that the set of changes were committed to the target system. The embodiment then updates the high watermark in the consumer registration table based on the confirmation.

Multiple subscriber data extraction for a change data capture (CDC)

Disclosed herein are system, method, and computer program product embodiments for automatically generating events. In some embodiments, a server receives a request to subscribe to an event type for an application. The server identifies an entity in a data model corresponding to the event type. The entity is associated with application data of the application. Furthermore, the server subscribes to receive alerts associated with any change to the entity in the data model. The server receives an alert indicating a change to the entity in the data model. The alert includes the change to the entity. Moreover, the server generates an event corresponding to the event type based on the change to the entity and transmits the event to an event infrastructure.

Automatically generating events

Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Dynamic filter and projection push down

Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.

Data replication system

Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system. An embodiment operates receiving a data change for a data extraction from a producer job at the source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at the target system. The embodiment generates the next package from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification.

Look-ahead staging for accelerated data extraction

Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Look-ahead staging for time-travel reconstruction

Provided is a framework including a generic communication layer that facilitates a separation between communication connections for different communication technologies and an application interfacing with the communication connections. The framework generally includes a remote interface layer including communication interfaces that generically communicate with a client, an application layer that includes applications represented by a set of services that use the communication interfaces to communicate with a client, and a service layer that includes one or more services. The framework includes mechanisms to map parameters of a specific communication technology to a standardized parameter structure that can be used by different service-based applications. In some aspects, the disclosed framework functions to encapsulate each service of an application in a configuration that is communication protocol specific independent of the implementation of the service, where the configuration specifies how the framework is to map services of the application to a specific communication protocol.

Growing community of inventors

Beijing, China

Daniel Bos