International Business Machines Corporation

Other

John Marland Garth

James Alan Ruddy

Craig Alan Friske

Bryan Frederick Smith

Paramesh Sampatrai Desai

Laura Michiko Kunioka-Weis

Ka Chun Ng

Christian Michel

Arnold T Banzon

Bituin B Vizconde

Thomas Abel Beavin

James Z Teng

Julie Ann Watts

David R Schwartz

Jay A Yothers

Akira Shibamiya

Christina Marie Lee

Mary C Lehner

Li-Mey P Lee

Robert L Huddleston

Neal R Eisenberg

William James Franklin

Frances Hwa Villafuerte

Charles S Tribolet

Ted Eric Blank

John Richard Lyle

Kathryn Ruth Zeidenstein

Julie Chen

Daniel Keith Courter

Koshy John

David R Betten

David Elmer Raiman

Timm Zimmermann

Kevin Daniel Gougherty

Techniques are described for assigning collections of records to parallel processing tasks for sorting index keys of the records. In one example, a method includes determining an order of the collections of records in a descending order by byte length of an index key of each of the collections of records. The method further includes modifying the order of the collections of records in a descending order, for collections of records with index keys of equal byte length, by number of records per collection of each of the collections of records with index keys of same byte length. The method further includes assigning the collections of records to a plurality of parallel processing tasks in the order of the collections of records. The method may combine low elapsed processing times by balancing the amount of work in each assigned processing task with reduced computing resource overhead for each processing task.

Sorting database collections for parallel processing

A method, apparatus and program storage device for managing buffers during online reorganization. An adaptive buffer is provided having a dynamically adjustable boundary, the adaptive buffer processes log records and pointers associated with the log records during online reorganization of a database. Adaptive switching is provided between a first and a second task during the processing of data log records and index log records during the online reorganization of the database.

Method, apparatus and program storage device for managing buffers during online reorganization

A method, apparatus and program storage device for determining an optimal number of tasks during reorganization of a database system with memory and processor constraints. The reorganization system identifies memory constraints, identifies processing capabilities and determines a number of reorganization processes and a combination of reorganization processes that provides a minimum reorganization time based upon the identified memory and processing constraints. The identified memory and processing capabilities identify a domain representing a set of rules applicable to the identified memory and processing capabilities. The set of rules of the identified domain are applied to determine an optimal number and combination of reorganization processes that minimizes a reorganization time.

Method, apparatus and program storage device for determining an optimal number of tasks during reorganization of a database system with memory and processor constraints

An apparatus, system, and method for determining the maximum supported degree of parallel sort operations in a multi-processor computing environment. An allocation module allocates a minimum number of sort files to a sort operation for each data source that participates in the parallel sort. The allocation module attempts to allocate sort files of one-half the sort operation data source file size, and iteratively reduces the sort file size requests in response to determinations that sort files of the requested size are not available. After allocation, a parallel operation module determines whether there is sufficient virtual storage to execute the sort operations in parallel. If there is not, the parallel operations module collapses the two smallest sort operations, thereby reducing the degree of parallelism by one, and repeats the request. The parallel operation module repeats the process until the sorts are executed or the process fails for lack of virtual storage.

Apparatus, system, and method for deterministic file allocations for parallel operations

One or more indexes are built on data (concurrent with manipulation of data) wherein the index build operation is done without restricting the manipulation of data. The index is brought to consistency with the data based upon processing recovery log records to simulate index manipulations during data insert, update, or delete operations.

Method and apparatus for building one or more indexes on data concurrent with manipulation of data

A base table and related auxiliary table spaces are reorganized concurrently via a database utility. The database utility determines which auxiliary tables are related to the base table and automatically includes their respective auxiliary table in the same invocation of the utility. The reorganization is performed via allocated shadow data sets. The original data sets are switched with the newly built shadow data sets including the LOB shadows.

System and method for concurrently reorganizing logically related LOB table spaces

The invention relates to a system and method for predicting the elapsed time required to execute a database utility command. A base of historical data is maintained containing information relating to the elapsed time required for previous database utility commands to execute. The elapsed time for a database utility command provides a consolidated measurement of all factors that affect processing time, including amount and type of data stored as well as changes within the computer system environment. Upper and lower control limits are calculated using statistical process control techniques to predict database utility command execution times. The technique may be used to validate and/or edit database utility command files containing multiple database utility commands, based upon whether the commands contained within are likely to fully execute within a specified batch window.

System and method for predicting execution time of a database utility command

A method, apparatus, and article of manufacture of a computer-implemented parallel database loading system. The optimum number of tasks to be processed by the system is determined by identifying the memory constraints of the system, by identifying available processing capabilities, and by determining a number of load and sort processes to be started in parallel based on the identified memory constraints and processing capabilities. Optimizing the number of load and sort processes increases overall system processing speed.

Technique for determining an optimal number of tasks in a parallel database loading system with memory constraints

A technique for loading data into a data store connected to a computer. Under control of a main process, multiple agent load processes are started for loading data in parallel. The main process awaits receipt of a checkpoint signal from each agent load process. Then, upon receiving the checkpoint signal from each load process, the main process performs a checkpoint.

Technique for establishing a point of consistency in a parallel database loading system

Disclosed is a system, method, and program for extracting data from a record in a first format and inserting data into a record in a second format during an insert operation. A determination is made of location information for data in at least one record in at least one column in the first format involved in the insert operation from which data will be extracted and of location information for data in at least one record in at least one column in the second format involved in the insert operation into which the data extracted from a corresponding column in the first format will be inserted. For each column from which data will be extracted, the determined location information is used to generate at least one instruction to move the data in each column in the record in the first format to each corresponding column of one record in the second format. The location information of the columns in records in the first and second formats involved in the data movement operations is encoded in the at least one instruction. The instruction may be repeated for multiple records in the first format from which data will be extracted into multiple records in the second format.

Method, system, and program for extracting data from database records using dynamic code

A technique for identifying changes in a data store connected to a computer. Initially, one or more interval changes are measured. Each interval change indicates an amount of change in the data store at an interval. Next, a data store change is estimated that indicates an amount of change in the data store across all of the intervals using each interval change.

Method of estimating an amount of changed data over plurality of intervals of time measurements

A method, apparatus, and article of manufacture for a computer-implemented building indexes system. Indexes are built for a database that is stored in a data storage device coupled to a computer. An amount of available memory is determined. An amount of memory for use in transmitting data between extract, sort, and index build tasks is determined. Then, a number of sort tasks to be used to build indexes is determined based on the determined amount of available memory, the determined amount of memory for use in transmitting data between tasks, and task memory requirements.

Determining the optimal number of tasks for building a database index

A method, apparatus, and article of manufacture for a computer-implemented repartitioning system. Data is repartitioned in a database stored on a data storage device connected to a computer. First, it is detected that a partitioning scheme for the data has been altered. Next, partitions that would be affected by the altered partitioning scheme are identified. Then, the identified partitions are reorganized based on the altered partitioning scheme.

Repartitioning data

Data is loaded to a data structure while an image copy of the data structure is concurrently formed. After data records are obtained from one or more data sources, each of the data records is associated with one of multiple pages. As each page is completed, it is written to a primary data structure. Also upon completion of each page in the primary data structure, concurrently with storage of the page in the primary data structure, an image copy of the page is made in an image copy data set. Partially or completely failed primary data structures may also be restored using image copies made according to the invention. Timestamps mark the time at which each page are made; image copy pages include the same timestamp as their respective primary pages. Upon detecting a data structure failure, each page of the image copy is sequentially reviewed. For each image page being reviewed, the page's timestamp is compared with the timestamp of the corresponding page from the primary data structure. The page is copied from the image copy to the primary data structure only if the timestamp comparison indicates that the image page is more recent than the page from the primary data structure.

System for data structure loading with concurrent image copy

A versioned-data management system (VDMSMS) is provided with a method for determining candidates for promotion in association with a user task. The VDMS maintains a list of parts changed as a result of a user task, referred to as a promote group. Before performing any changes the user informs the VDMS which user task is to be associated with the changes. All parts subsequently changed are automatically placed by the VDMS in the promote group associated with the given user task. When the parts associated with a user task for a given variant level are promoted, an iterative process is performed to determine the complete set of parts that must be promoted in support of the requested promote. Initially the set includes those parts in the promote group at the given variant level. Then all the other promote groups are searched to determine whether they contain any of the parts at the given variant level that are already in this set. For any that do, the parts in those promote groups for the given variant level are added to the set of parts to be promoted.

Growing community of inventors

Gilroy, CA, United States of America

John Marland Garth