National Instruments Corporation

James A Grey

Jon Bellin

Scott A Rust

Douglas Melamed

Erik Crank

Ronald Byrd

Scott Richardson

Patrick Williams

Hjalmar Perez

Daniel Elizalde

Test executive system and method of use. The system includes a test executive engine, configured to execute at least one test executive sequence to test at least one unit under test (UUT), a process model that specifies one or more function sequences for pre-test or post-test functionality for the test executive sequences, and a plug-in framework, configured to selectively incorporate one or more process model plug-in instances in the process model. Each process model plug-in instance specifies at least one respective function sequence for pre-test or post-test functionality for the test executive sequences.

Test executive system with processing of results data in chunks

System and method for performing a multiple tests on each of one or more units, where each of the tests requires a respective resource of a plurality of resources. A first test is performed on a unit using a first resource. During performance of the first test, a search is made for a second test, requiring a second resource, where the second resource is not currently being used. If the second test is found, the second test is performed on the unit, or a second unit, using the second resource, substantially concurrently with at least a portion of the first test being performed on the unit. Performing a test includes locking the respective resource to exclude use by other tests, including acquiring the resource, and unlocking the resource upon completion of the test, including releasing the resource for use in performing the respective test on another of the units.

Auto-scheduling of tests

A system and method for automatically detecting heap corruption errors and memory leak errors caused by user-supplied code modules that are called by steps of a test executive sequence. The test executive sequence may first be created by including a plurality of test executive steps in the test executive sequence and configuring at least a subset of the steps to call user-supplied code modules. The test executive sequence may then be executed on a host computer under control of a test executive engine. For each step that calls a user-supplied code module, the test executive engine may perform certain actions to automatically detect whether the user-supplied code module causes a heap corruption error and/or automatically detect whether the user-supplied code module causes a memory leak error.

Test executive system with memory leak detection for user code modules

A system and method for detecting and/or preventing execution problems caused by user-supplied code modules that are called by steps of a test executive sequence. The test executive sequence may include a plurality of test executive steps. At least a subset of the steps may be configured to call user-supplied code modules. The test executive sequence may be executed on a host computer under control of a test executive engine. For each step that calls a user-supplied code module, the test executive engine may perform certain actions to detect and/or prevent execution problems caused by the user-supplied code module. For example, the test executive engine may be operable to automatically detect stack corruption caused by the user-supplied code module, may be operable to utilize stack safety buffers to avoid stack corruption problems caused by the user-supplied code module, and/or may include features for aiding a user in determining whether the user-supplied code module contains an uninitialized local variable.

Test executive with stack corruption detection

Test executive which provides heap validity checking and memory leak detection for user code modules

A system and method for isolating execution of user-supplied code modules that are called by steps of a test executive sequence. The test executive sequence may first be created by including a plurality of test executive steps in the test executive sequence and configuring at least a subset of the steps to call user-supplied code modules. The test executive sequence may then be executed on a host computer under control of a test executive engine. For each step that calls a user-supplied code module, the test executive engine may invoke the user-supplied code module for execution in a separate process other than the process in which the test executive engine executes.

Test executive with external process isolation for user code modules

A test executive sequence may be created by including a plurality of test executive steps in the test executive sequence in response to user input to a sequence editor. At least a subset of the test executive steps in the test executive sequence may be configured to call user-supplied code modules. In one embodiment, one or more expressions may be configured for one or more steps in the test executive sequence, where the one or more expressions evaluate to result values to be logged when the step is executed during execution of the test executive sequence. In another embodiment, one or more parameters of one or more user-supplied code modules called by steps in the test executive sequence may be configured to be logged when the test executive sequence is executed.

Test executive system with automatic expression logging and parameter logging

System and method for synchronizing software execution in a computer system. Threads or processes executing in the computer system are synchronized via various types of synchronization objects. User input indicating a desire to specify a synchronization object operation is received, specifying a particular type of synchronization object on which to perform the operation, in response to which a graphical user interface, is displayed which is useable to specify a plurality of synchronization object operations, e.g., that are specific to a specified type of synchronization object. The graphical user interface enables the user to select a desired operation to perform on a synchronization object. The user may also specify additional configuration information for the operation. In response to the specified synchronization object operation, program instructions that are executable to perform the synchronization object operation may be programmatically created, and may subsequently be executed to perform the specified synchronization object operation.

System and method for synchronizing software execution

A system and method for asynchronous execution of a test executive subsequence. A first test executive sequence (calling sequence) may be created and configured to asynchronously call a second test executive sequence (subsequence). In one embodiment, a 'subsequence call' step may be included in the first test executive sequence, wherein the subsequence call step is operable to asynchronously call the second test executive sequence when the subsequence call step is executed. In response to executing the first test executive sequence, the second test executive sequence may also be executed, asynchronously from the first test executive sequence. The user may specify various options affecting execution of the second test executive sequence (the subsequence). For example, the user may specify various aspects of the execution environment or execution location for the subsequence.

System and method enabling asynchronous execution of a test executive subsequence

A system and method for synchronizing execution of multiple processes or threads executing to perform tests of one or more units under test. One or more synchronization steps may be included in a test executive sequence, in response to user input requesting inclusion of the synchronization steps. Each synchronization step may be configured to perform a synchronization operation, in response to user input specifying the synchronization operation. The test executive sequence may then be executed multiple times concurrently, e.g., by multiple threads or processes, and the one or more synchronization steps in the test executive sequence may coordinate the execution of the multiple threads or processes to test the unit(s) under test.

System and method for synchronizing execution of a test sequence

A system and method for enabling execution stop and re-start of a test executive sequence or hierarchy of test executive sequences. Execution progress of a test executive sequence or test executive sequence hierarchy may be periodically stored. This may comprise performing or taking 'snapshots' of the execution at various points during the execution. Performing a snapshot may comprise saving all data needed to restore and re-start the execution at the respective point. The criteria of when and where to perform the snapshots may be any of various criteria and may be specified in any of various ways.

System and method enabling execution stop and restart of a test executive sequence(s)

A method for creating a computer program to be executed by a plurality of threads, in which the method utilizes a technique for execution synchronization referred to herein as a batch synchronization section. According to this technique, a plurality of threads may be associated with one another as a 'batch' of threads. Each thread in the plurality (batch) of threads may execute the computer program simultaneously. The batch synchronization section may specify a portion of the computer program for which the execution of the portion by the plurality of threads is to be synchronized. In one embodiment different types of batch synchronization sections may be specified, wherein each type of batch synchronization section performs a different type of execution synchronization. In one embodiment the method may enable execution synchronization behavior for multiple concurrent executions of a test executive test sequence to be specified. The test sequence may include one or more batch synchronization sections. Multiple threads may each execute an instance of the test sequence to concurrently to test a group of units under test, and the batch synchronization sections may coordinate the execution of the multiple threads where necessary.

System and method for synchronizing execution of a batch of threads

Individual test executive sequences may correspond to individual sub-components of a hierarchical system. To test a system sub-component directly without testing 'upper' sub-components in the hierarchy, a first test executive sequence corresponding to that system sub-component may be executed directly, without requiring execution of test executive sequences above the first test executive sequence in the hierarchy, or requiring only partial execution of test executive sequences above the first test executive sequence in the hierarchy. Another embodiment of the invention comprises a system and method for enabling propagation or inheritance of test executive sequence local variables.

System and method enabling hierarchical execution of a test executive subsequence

A method for creating a computer program to be executed by a plurality of threads, in which the method utilizes a technique for execution synchronization referred to herein as a batch synchronization section. According to this technique, a plurality of threads may be associated with one another as a &ldquo;batch&rdquo; of threads. Each thread in the plurality (batch) of threads may execute the computer program simultaneously. The batch synchronization section may specify a portion of the computer program for which the execution of the portion by the plurality of threads is to be synchronized. In one embodiment different types of batch synchronization sections may be specified, wherein each type of batch synchronization section performs a different type of execution synchronization. In one embodiment the method may enable execution synchronization behavior for multiple concurrent executions of a test executive test sequence to be specified. The test sequence may include one or more batch synchronization sections. Multiple threads may each execute an instance of the test sequence to concurrently to test a group of units under test, and the batch synchronization sections may coordinate the execution of the multiple threads where necessary.

System and method for performing batch synchronization for a test sequence

A test executive program which provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive program includes process models for improved flexibility, modularity and configurability. Process models provide a modular and configurable entity for encapsulating operations and functionality associated with a class of test sequences. The process model thus encapsulates a &ldquo;testing process&rdquo; for a plurality of test sequences. The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The test executive program also includes step types for improved configurability. A step type is a modular, identifiable unit configured by the user which defines common properties and/or operations associated with a plurality of steps. Instances of the step type incorporate this common functionality and/or properties from the step type, and thus the user is not required to hard code that functionality and/or properties in each instance or step. The test executive program also provides distributed type storage and conflict resolution. When the user loads a file which includes a stored type, the method performs conflict resolution, including determining if the loaded type conflicts with one or more previously loaded/registered types. If the types do conflict, then the test executive or user selects a type, preferably using the version data associated with each type. The test executive system also performs automatic result collection to automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.

Growing community of inventors

Cedar Park, TX, United States of America

James A Grey