Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Google Inc.

Carlos Alejandro Arguelles

Ramakrishnan Hariharan Chandrasekharapuram

Ryan Preston Gantt

Vijay Pratap Singh

Mihaela Petrescu

David Robert Griffin

Choi Yong Ngo

Jeremy Boynes

Mark Evans Brighton

Adam Lloyd Days

Gavin Richard Jewell

Erin Harding Kraemer

Brian Jaffery Tajuddin

Brian Thomas Kachmarck

Michael Leo Weiss

Tyson Christopher Trautmann

Phillip Segel

Siamak Irantash

Jeenandra Kumar Uttamchand

Srinath Badrinath

Andy Kohn

Justin Alexander Tisi

Danhao Guo

Kiran Chitluri

Aman Ahmed

Praveen Kambam Sugavanam

Kyle Thomas Studd

Manoj Srivastava

Thomas Lowell Keller

Simon Kurt Johnston

Blair Livingstone Hotchkies

Meng Li

Faizal S Kassamali

Adithya Venkatesh

Kevin Lester Quadros

Ethan Fitzgerald John

James Beavis

Xiaozheng Chen

Sichao Shi

Jason Scott Wodicka

Gustavo Gallegos

Everett Maus

Benson Margulies

Peter Collin Nix

Choi Young Ngo

Priyanka Agha

Fernando Ciciliati

Tim Griffith

Wes McDaniel

David Panko

Xin-Liu Yao

Denis L Ruckebusch

Mikhail Khasanov

Daniel Alexander Jeffery

Darrin Prutsman

Harpreet Singh

Kathleen Joanna Tsoukalas

Mahmoud Mahmoud

Ayham Tannous

A method of populating a test environment for integration testing of a production database. The production database includes a plurality of tables, and may be represented by a directed acyclic graph (DAG), in which each of the plurality of tables is represented by a respective node of the DAG and schema-level constraints of the plurality of tables are represented by edges of the DAG. The method may involve constructing the DAG, topologically ordering the plurality of tables of the production database according to the DAG, randomly generating test data in accordance with the schema-level constraints of the plurality of tables, and populating the randomly generated test data to the test environment according to the topological ordering of the plurality of tables.

Method for populating spanner databases in test environments

Embodiments presented herein provide techniques for evaluating an asynchronous application using a test framework. The test framework may perform a load test of an asynchronous application or service composed from a collection of applications or services. To do so, the test framework may submit transactions to a distributed application at a specified transaction rate and monitor how the distributed application operates at that transaction rate. An aggregate load test component may evaluate the remaining work pending at work accumulation points of the distributed application to determine whether the distributed application can sustain the specified transaction rate. A transaction tracking component may initiate transactions to generate load at the specified transaction rate without blocking while the transactions are processed by the distributed application.

Non-blocking concurrent work in a transaction generator

A method of copying a production database to a test environment using one or more processors. The production database includes a plurality of tables and one or more constraints indicating interleaving of the plurality of tables. The method involves constructing a directed acyclic graph (DAG) of the plurality of tables, whereby each table is represented by a respective node of the DAG and interleaving of the tables is represented by edges of the DAG, topologically ordering the tables according to the DAG, and copying data from the production database to the test environment according to the topological ordering of the tables.

Method for copying spanner databases from production to test environments

Systems and processes are described for establishing and using a secure channel. A shared secret may be used for authentication of session initiation messages as well as for generation of a private/public key pair for the session. A number of ways of agreeing on the shared secret are described and include pre-sharing the keys, reliance on a key management system, or via a token mechanism that uses a third entity such as a hub to manage authentication, for example. In some instances, the third party may also perform endpoint selection (e.g., load balancing) by providing a particular endpoint along with the token.

Decoupled load generation architecture

Methods, systems, and computer-readable media for analysis of code coverage differences across environments are disclosed. A code coverage profiling system determines a first code coverage profile associated with execution of program code in a first environment. The first code coverage profile indicates one or more portions of the program code that were executed in the first environment. The code coverage profiling system determines a second code coverage profile associated with execution of the program code in a second environment. The second code coverage profile indicates one or more portions of the program code that were executed in the second environment. The code coverage profiling system performs a comparison of the first code coverage profile and the second code coverage profile. The comparison determines a difference between the portions of the program code that were executed in the first and second environments.

Analysis of code coverage differences across environments

A network-based scalable production load test service may be implemented on a provider network including a plurality of computing devices in order to provide load testing for network-based production systems. In some embodiments, the plurality of computing devices is configured to receive a request to capture to a load test data repository items of transaction data for a network-based production service. In some embodiments, the plurality of computing devices is configured to capture to the load test data repository the items of transaction data. The transaction data include input to the network-based production service over a network. In some embodiments, in response to a load test specification received by the scalable production load test service, the plurality of computing devices is configured to dynamically allocate one or more resources to perform a load test of the network-based production service according to the load test specification.

Scalable production test service

A test system includes a controller that creates test jobs to simulate a load for testing a target system or service. The jobs specify test data from a data store. The controller adds the test jobs to a job queue at a rate. Workers access jobs from the job queue and request data from the data store as specified in each job and send the request data to the service. Approximations (e.g., metric values indicating performance of the worker host resources, or performance of the system under test or performance of resources that implement the system under test, other feedback, etc.) of actual system characteristics such as the actual number of workers are obtained. Based on the metric values, the controller modulates various characteristics of the load generation system, such as a rate at which jobs are added to the job queue or the number of workers processing jobs.

Control system for distributed load generation

Techniques for performing load testing and profiling of services in a provider network are described. A load testing and profiling service is disclosed that analyzes profile data generated by a service and generates profile results associated with the service when the service operates at varying and/or increasing load capacities. The profile results are indicative of the performance of one or more functions performed by a service when the service operates at different load capacities. In certain embodiments, the load testing and profiling service can be invoked as part of a Continuous Deployment/Continuous Integration (CD/CI) environment that executes a load test against a test stack (e.g., test requests) before, for example, promoting code to production. For instance, the load testing and profiling service may be invoked as a step in a code deployment pipeline, e.g., for deploying a software product to a test environment, or to a production environment.

Framework for performing load testing and profiling of services

Systems and methods for validation of log formats are described herein. Log data is stored via a logging service in a data store or other storage system. An example log or proposed log format is received by the logging service. The proposed log format is validated against validation rules provided by log consumers.

Validation of log formats

Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Automated tuning of a service configuration

A contributed test management system receives a first request from a consumer system, where the first request comprises a request for a contributed test to be added to a deployment pipeline of a producer system, and where the contributed test is associated with an application component in the deployment pipeline. The contributed test management system causes the contributed test to test a code update provided by the producer system for the application component in the deployment pipeline, detects whether the first test fails during execution, and, if so indicates to the consumer system that the first test has failed.

Contributed test management in deployment pipelines

Methods, systems, and computer-readable media for ordered test execution based on code coverage are disclosed. A suite of tests are executed on a first version of program code to generate data indicative of code coverage of respective tests with respect to the program code. A mapping of the tests to the program code is determined based at least in part on the data indicative of code coverage and is stored. The mapping comprises data indicative of one or more portions of the program code exercised by respective tests from the suite. Based at least in part on the mapping of the tests to the program code and on data indicative of one or more modified or new portions of a second version of the program code, a subset of the tests is determined that are likely to be exercised by the second version of the program code.

Ordered test execution based on test coverage

Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may utilize the computed measures of effectiveness and weights to compute a grand measure of merit (MOM) for the candidate sets of application parameter values. An optimized set of application parameter values may then be selected based upon the computed grand MOMs. The optimized set of application parameter values may be deployed to a production execution environment executing the distributed application. Production safe application parameters might also be identified and utilized to optimize execution of the distributed application in a production execution environment.

Optimization of application parameters

Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host.

Automated determination of maximum service throughput

Some embodiments include methods and systems for performing non-deterministic load testing of a network-based production service. Some embodiments perform, using a load testing system implemented on one or more computers, receiving a load testing specification. The load testing specification indicates one or more load testing conditions for performing non-deterministic load testing of a network-based production service executing transactions in a networked computing environment. Some embodiments perform requesting from the virtual computing service resources to perform the non-deterministic load testing of the network-based production service, perform the non-deterministic load testing of the network-based production service. The non-deterministic load testing comprises executing test transactions on the network-based production service at pseudo-randomly varying times to measure system performance of the network-based production service at a plurality of levels of transaction intensity.

Non-deterministic load-testing

Growing community of inventors

Shoreline, WA, United States of America

Carlos Alejandro Arguelles