Amazon Technologies, Inc.

Andrew J Hitchcock

Peter Sirota

Richard Jeffrey Cole

Jai Vasanth

Luis Felipe Cabrera

Ian Paul Nowland

Richendra Khanna

Vaibhav Aggarwal

Techniques are described for managing distributed execution of programs. In some situations, the techniques include dynamically modifying the distributed program execution in various manners, such as based on monitored status information. The dynamic modifying of the distributed program execution may include adding and/or removing computing nodes from a cluster that is executing the program, modifying the amount of computing resources that are available for the distributed program execution, terminating or temporarily suspending execution of the program (e.g., if an insufficient quantity of computing nodes of the cluster are available to perform execution), etc.

Dynamically modifying a cluster of computing nodes used for distributed execution of a program

Techniques are described for managing distributed execution of programs. In some situations, the techniques include determining configuration information to be used for executing a particular program in a distributed manner on multiple computing nodes and/or include providing information and associated controls to a user regarding ongoing distributed execution of one or more programs to enable the user to modify the ongoing distributed execution in various manners. Determined configuration information may include, for example, configuration parameters such as a quantity of computing nodes and/or other measures of computing resources to be used for the executing, and may be determined in various manners, including by interactively gathering values for at least some types of configuration information from an associated user (e.g., via a GUI that is displayed to the user) and/or by automatically determining values for at least some types of configuration information (e.g., for use as recommendations to a user).

Managing distributed execution of programs

Techniques are described for managing distributed execution of programs, including by dynamically scaling a cluster of multiple computing nodes performing ongoing distributed execution of a program, such as to increase and/or decrease computing node quantity. An architecture may be used that has core nodes that each participate in a distributed storage system for the distributed program execution, and that has one or more other auxiliary nodes that do not participate in the distributed storage system. Furthermore, as part of performing the dynamic scaling of a cluster, computing nodes that are only temporarily available may be selected and used, such as computing nodes that might be removed from the cluster during the ongoing program execution to be put to other uses and that may also be available for a different fee (e.g., a lower fee) than other computing nodes that are available throughout the ongoing use of the cluster.

Dynamic scaling of a cluster of computing nodes

Use of temporarily available computing nodes for dynamic scaling of a cluster

Techniques are described for managing distributed execution of programs. In some situations, the techniques include dynamically monitoring the ongoing distributed execution of a program on a cluster of multiple computing nodes, and may include automatically determining the status of execution of the program on each of the multiple computing nodes and/or automatically determining the aggregate usage of one or more types of computing resources across the cluster of multiple computing nodes by the distributed program execution. The information obtained from the dynamic monitoring may be used in various manners, including to facilitate dynamically modifying the ongoing distributed program execution in various manners, such as to temporarily throttle usage of computing resources by the distributed program execution (e.g., to remove or reduce one or more bottlenecks).

Dynamically monitoring and modifying distributed execution of programs

Techniques are described for managing distributed execution of programs, including by dynamically scaling a cluster of multiple computing nodes used to perform ongoing distributed execution of a program, such as to increase and/or decrease the quantity of computing nodes in the cluster at various times and for various reasons. An architecture may be used that facilitates the dynamic scaling of a cluster, including by having at least some of the computing nodes act as core nodes that each participate in a distributed storage system for the distributed program execution, and having one or more other computing nodes that act as auxiliary nodes that do not participate in the distributed storage system. If computing nodes are selected to be removed from the cluster during ongoing distributed execution of a program, one or more nodes of the auxiliary computing node type may be selected for the removal.

Growing community of inventors

Seattle, WA, United States of America

Andrew J Hitchcock