Amazon Technologies, Inc.

Richard Jeffrey Cole

Peter Sirota

Jai Vasanth

Andrew J Hitchcock

Luis Felipe Cabrera

Ian Paul Nowland

Richendra Khanna

Vaibhav Aggarwal

Kathryn Marie Shih

Carl Louis Christofferson

James P Bartlett

Adam David Gray

Alan D Mock

Brian James Schuster

George Steven McPherson

Yevgeniy Mikhaylyuta

Eider Brantly Moore

Raj K Kizhakkekalathil

Richard Rex McKnight

Tobin Daniel Baker

Andrew Bruce Dickinson

Sudheer Tumuluru

Lorenzo Minore

Mark Jacobson

Llewllyn J Mason

Shashank Senapaty

Karen R Haberkorn

Mike Perkowitz

Nachi Govindasamy

A metadata catalog may store metadata about data sources. The metadata may define table, column, partition and/or data formats, as well as title information, textual descriptions, modification information, data schemas and access information, such as location and connection information. A metadata catalog service may provide a shared, persistent, central repository for the metadata. The metadata of the catalog may be published via multiple access mechanisms and programmatically obtained and processed by other programs via the metadata catalog service. Metadata consumers may utilize the metadata catalog service to perform various metadata tasks, such as metadata retrieval, creation, addition, modification and/or deletion. The metadata catalog service may be directly integrated into existing data analytic tools as a persistent and durable metadata store allowing users to utilize the broad set of data processing tools.

Persistent metadata catalog

Techniques are described for managing distributed execution of programs. In some situations, the techniques include dynamically modifying the distributed program execution in various manners, such as based on monitored status information. The dynamic modifying of the distributed program execution may include adding and/or removing computing nodes from a cluster that is executing the program, modifying the amount of computing resources that are available for the distributed program execution, terminating or temporarily suspending execution of the program (e.g., if an insufficient quantity of computing nodes of the cluster are available to perform execution), etc.

Dynamically modifying a cluster of computing nodes used for distributed execution of a program

Techniques are described for managing distributed execution of programs. In at least some situations, the techniques include decomposing or otherwise separating the execution of a program into multiple distinct execution jobs that may each be executed on a distinct computing node, such as in a parallel manner with each execution job using a distinct subset of input data for the program. In addition, the techniques may include temporarily terminating and later resuming execution of at least some execution jobs, such as by persistently storing an intermediate state of the partial execution of an execution job, and later retrieving and using the stored intermediate state to resume execution of the execution job from the intermediate state. Furthermore, the techniques may be used in conjunction with a distributed program execution service that executes multiple programs on behalf of multiple customers or other users of the service.

Saving program execution state

Techniques are described for providing clients with access to functionality for creating, configuring and executing defined workflows that manipulate source data in defined manners, such as under the control of a configurable workflow service that is available to multiple remote clients over one or more public networks. A defined workflow for a client may, for example, include multiple interconnected workflow components that are specified by the client and that each are configured to perform one or more types of data manipulation operations on a specified type of input data. The configurable workflow service may further execute the defined workflow at one or more times and in one or more manners, such as in some situations by provisioning multiple computing nodes provided by the configurable workflow service to each implement at least one of the workflow components for the defined workflow.

Providing configurable workflow capabilities

Methods and systems for cost-minimizing job scheduling are disclosed. A definition of a task is received. The definition comprises a need-by time. The need-by time comprises a deadline for completion of execution of the task. An estimated duration to complete the execution of the task is determined for each of a plurality of computing resources. One or more of the computing resources are selected based on an estimated cost of completing the execution using the computing resources. The execution of the task is initiated at a scheduled time using the selected one or more computing resources. The scheduled time is earlier than the need-by time by at least the estimated duration.

Cost-minimizing task scheduler

Methods and systems for optimization of task execution are disclosed. A definition of a task is received. A plurality of parameter values for execution of the task are selected based on an execution history for a plurality of prior tasks performed for a plurality of clients. The plurality of parameter values are selected to optimize one or more execution constraints for the execution of the task. The execution of the task is initiated using one or more computing resources configured with the selected parameter values.

Parameter selection for optimization of task execution based on execution history for prior tasks

Techniques are described for managing aggregation of data in a distributed manner, such as for a particular client based on specified configuration information. The described techniques may include storing aggregated data values for an OLAP cube or other data structure in a distributed manner, such as in some situations in a distributed hash table. The aggregated data values to be stored may be generated in various manners, such as by performing multi-stage data manipulation operations&#x2014;for example, a map-reduce architecture may be used, with a first stage involving the use of one or more specified map functions to be performed, and with at least a second stage involving the use of one or more specified reduce functions to be performed.

Distributed storage of aggregated data

Techniques are described for managing distributed execution of programs. In some situations, the techniques include determining configuration information to be used for executing a particular program in a distributed manner on multiple computing nodes and/or include providing information and associated controls to a user regarding ongoing distributed execution of one or more programs to enable the user to modify the ongoing distributed execution in various manners. Determined configuration information may include, for example, configuration parameters such as a quantity of computing nodes and/or other measures of computing resources to be used for the executing, and may be determined in various manners, including by interactively gathering values for at least some types of configuration information from an associated user (e.g., via a GUI that is displayed to the user) and/or by automatically determining values for at least some types of configuration information (e.g., for use as recommendations to a user).

Managing distributed execution of programs

Methods and systems for using a scheduler in a data pipeline are disclosed. A plurality of objects in a first layer are created, each representing a respective regularly scheduled task. A plurality of objects in a second layer are created, each representing a respective scheduled instance of a regularly scheduled task. It is determined whether each object in the second layer is ready to execute. For at least one object in the second layer, it is determined if the object has received notifications from any objects on which it depends. For each object that is ready to execute, the regularly scheduled task associated with the object is performed. For each object that is not ready to execute, the object is put to sleep.

Scheduler for data pipeline

Methods and systems for task timeouts as a function of input data size are disclosed. A definition of a task is received. The definition of the task indicates a set of input data for the task. A timeout duration for the task is determined based on the set of input data. The timeout duration varies with one or more characteristics of the set of input data. The execution of the task is initiated. The execution of the task is stopped if the execution of the task exceeds the timeout duration.

Task timeouts based on input data characteristics

Methods and systems for automatic table schema generation are disclosed. A description of a data source is received. A table definition is determined based on the description of the data source, wherein the table definition maps the data source to one or more columns of an output table. A parser for the data source is selected from a set of predefined parsers based on the description of the data source. A query against the data source is received. One or more elements of data responsive to the query are retrieved from the data source using the selected parser. The one or more elements are transformed into the output table based on the table definition.

Growing community of inventors

Seattle, WA, United States of America

Richard Jeffrey Cole