Sap Se

Gordon Gaumnitz

Lars Dannecker

Robert Schulze

Ivan T Bowman

Dan Farrar

Yu Cheng

Daniel James Farrar

Boyi Ni

Jonathan Dees

Arnaud Lacurie

Ingo Mueller

A system and method for faster access for compressed time series data. A set of blocks are generated based on a table stored in a database of the data platform. The table stores data associated with multiple sources of data provided as consecutive values, each block containing index vectors having a range of the consecutive values. A block index is generated for each block having a field start vector representing a starting position of the block relative to the range of consecutive values, and a starting value vector representing a value of the block at the starting position. The field start vector of the block index is accessed to obtain the starting position of a field corresponding to a first block and to the range of the consecutive values of the first block. The starting value vector is then determined from the block index to determine an end and a length of the field of the first block.

Faster access for compressed time series data: the block index

Disclosed is a system and method for improving database memory consumption and performance using compression of time stamp columns. A number of time stamps of a time series is received. The time stamps have a start time, and are separated by an equal increment of time that defines an interval. The start time and interval are stored in a dictionary of a column store of a database. An index is generated in the column store of the database, the index having a number of index vectors. Using the index vectors, each time stamp of the number of time stamps can be calculated from the start time stored in the dictionary and the position in the time series based on the interval stored in the dictionary.

Compressing time stamp columns

A system and method include storing a table of time series data in a database of a data platform, the table of time series data representing a set of time series blocks. Each time series block of the set of time series blocks has a time series of equally-incremented time intervals and a run length. Each time interval of the time series is associated with one or more values. The run length has a starting position with at least one starting value and an ending position with at least one ending value. The starting position and the at least one starting value is stored for each time series block in a column store of the database. Then, a compressed index is generated in the column store of the database for each time series block, the compressed index comprising the starting position and the at least one starting value.

Linear run length encoding: compressing the index vector

Time series data may be represented with multiple representations, optionally using a variety of storage approaches, and the plurality of representations may be indexed using a representation index, which includes a start row identifier, a representation identifier, and an offset within the representation for each segment of one or more rows in the time series data column.

Indexing flexible multi-representation storages for time series data

An advisor creates configurations for generating multi-representations of time series data based upon detected characteristics such as length, interval, minimums, data types, etc., as well as configurations provided by a user. In an offline mode the advisor may further consider a previous time series workload. In an on-line mode the advisor may adapt multi-representation configurations with respect to ongoing changes in a current time series workload. The advisor may reference a cost model including values quantifying various dimensions (e.g., compression technique, accuracy, covered time period, storage medium, memory consumption, speed) of the multi-representations for optimization purposes. Configurations created by the advisor may be input to a storage engine to generate and store the multi-representations according to goals for data aging, operation execution pattern optimization, and ease of access to time series data located in hot zones. The advisor may be implemented with an engine of an in-memory database.

Advisor generating multi-representations of time series data

A flexible approach for storing time series data, utilizes multiple representations in order to achieve optimization amongst various dimensions such as covered time period, accuracy, compression model, and storage medium. A first aspect selectively provides different representations for various portions of a time series along the time axis (horizontal). In this horizontal aspect, a first compression model may be applied to store data occurring during one period in a time series, while a different compression model is applied to store data during another period. In a separate (vertical) aspect, time series data over a same time period may be saved in multiple representations using different compression models. While increasing demands on memory, such vertical storage can afford efficient access for specific purposes (i.e., analytical). Storing time series data according to horizontal and/or vertical distributions, may be useful for applications such as data aging and the optimization of operator execution patterns.

Multi-representation storage of time series data

Linear run length encoding is described. A system and method include storing a table of time series data in a database of a data platform, the table of time series data representing a set of time series blocks. Each time series block of the set of time series blocks has a time series of equally-incremented time intervals and a run length. Each time interval of the time series is associated with one or more values. The run length has a starting position with at least one starting value and an ending position with at least one ending value. The starting position and the at least one starting value is stored for each time series block in a column store of the database. Then, a compressed index is generated in the column store of the database for each time series block, the compressed index comprising the starting position and the at least one starting value.

A system includes multiple memory modules arranged and configured to store data and at least one processor that is operably coupled to the memory modules. The at least one processor is arranged and configured to select an access pattern of a forecast model, determine a storage layout model based on the identified access pattern of the forecast model, and store values in an order defined by the storage layout model using at least one of the memory modules. The order of the stored values enables sequential access to the stored values for use in the forecast model. Implementations of one or more features of the system may be performed by a computer-implemented method and/or a computer program product.

Forecast-model-aware data storage for time series data

Aggregating input into hashtables using just-in-time compilation of compilable code in response to a database query. Compilable code can be generated that is configured to cause a programmable processor to produce one or more hashmaps based on the input database. The one or more hashmaps can correspond to each individual thread from the input. The compilable code can be configured to cause the one or more processors to insert the hashmaps into a scheduler. Compilable code can be generated that is configured to: aggregate elements from the one or more hashmaps into buckets of elements having the same partition identity; rehash the buckets of elements having the same partition identity to reduce the number of groups within the bucket; facilitate the merger of all non-empty elements from each target-partition into a merged-partition.

Aggregating database entries by hashing

A database can receive a query from a remote computing system. The database can include (i) a linear run length encoded compressed column, based on an original column of time series data partitioned into runs containing consecutive values and generated by run length encoding, (ii) a run index comprising at least one run index value having a run index position, the at least one run index value identifying runs in the original column, and (iii) an offsets column identifying the run index positions corresponding to the runs that contain a desired value. Using the run index, data responsive to the query can be identified. The identified data responsive to the query can be transmitted by the database to the remote computing system.

Growing community of inventors

Walldorf, Germany

Gordon Gaumnitz