Amazon Technologies, Inc.

Kathryn Marie Shih

Richard Jeffrey Cole

Peter Sirota

Vaibhav Aggarwal

Carl Louis Christofferson

Brian Irl Pratt

Patrick James Ward

James P Bartlett

Eider Brantly Moore

Richard Rex McKnight

Lorenzo Minore

Sudheer Tumuluru

Nachi Govindasamy

Methods and systems for implementing custom access controls are disclosed. A first task is added to a first workflow. A first role is generated for the first workflow. The first role comprises a first set of one or more permissions for using one or more computing resources. The one or more permissions in the first role are selected based on the first task. The first task is performed using the one or more computing resources in accordance with the first role.

Custom access controls

Methods and systems for cost-minimizing job scheduling are disclosed. A definition of a task is received. The definition comprises a need-by time. The need-by time comprises a deadline for completion of execution of the task. An estimated duration to complete the execution of the task is determined for each of a plurality of computing resources. One or more of the computing resources are selected based on an estimated cost of completing the execution using the computing resources. The execution of the task is initiated at a scheduled time using the selected one or more computing resources. The scheduled time is earlier than the need-by time by at least the estimated duration.

Cost-minimizing task scheduler

Methods and systems for optimization of task execution are disclosed. A definition of a task is received. A plurality of parameter values for execution of the task are selected based on an execution history for a plurality of prior tasks performed for a plurality of clients. The plurality of parameter values are selected to optimize one or more execution constraints for the execution of the task. The execution of the task is initiated using one or more computing resources configured with the selected parameter values.

Parameter selection for optimization of task execution based on execution history for prior tasks

Methods and systems for implementing versioned access controls are disclosed. A first task is added to a first workflow with a first version of a default role. A second version of the default role is generated after the first task is added. A second task is added to a second workflow with the second version of the default role. The first version and the second version each comprise one or more permissions for using one or more computing resources. The first task is performed using the permissions in the first version of the default role. The second task is performed using the permissions in the second version of the default role.

Versioned access controls

Methods and systems for using a scheduler in a data pipeline are disclosed. A plurality of objects in a first layer are created, each representing a respective regularly scheduled task. A plurality of objects in a second layer are created, each representing a respective scheduled instance of a regularly scheduled task. It is determined whether each object in the second layer is ready to execute. For at least one object in the second layer, it is determined if the object has received notifications from any objects on which it depends. For each object that is ready to execute, the regularly scheduled task associated with the object is performed. For each object that is not ready to execute, the object is put to sleep.

Scheduler for data pipeline

Methods and systems for task timeouts as a function of input data size are disclosed. A definition of a task is received. The definition of the task indicates a set of input data for the task. A timeout duration for the task is determined based on the set of input data. The timeout duration varies with one or more characteristics of the set of input data. The execution of the task is initiated. The execution of the task is stopped if the execution of the task exceeds the timeout duration.

Task timeouts based on input data characteristics

Methods and systems for automatic table schema generation are disclosed. A description of a data source is received. A table definition is determined based on the description of the data source, wherein the table definition maps the data source to one or more columns of an output table. A parser for the data source is selected from a set of predefined parsers based on the description of the data source. A query against the data source is received. One or more elements of data responsive to the query are retrieved from the data source using the selected parser. The one or more elements are transformed into the output table based on the table definition.

Automatic table schema generation

Methods and systems for a connector interface in a data pipeline are disclosed. A pipeline comprising two data source nodes and an activity node is configured. Each data source node represents data from a different data source, and the activity node represents a workflow activity that uses the data as input. Two connectors which implement the same connector interface are triggered. In response, data is acquired at each connector from the corresponding data source through the connector interface. The data is sent from the connectors to the activity node through the connector interface. The workflow activity is performed using the acquired data.

Growing community of inventors

Seattle, WA, United States of America

Kathryn Marie Shih