Vmware, Inc.

Advanced Micro Devices Corporation

Banyanops, Inc.

Jayanth Gummaraju

Gabriel Tarasuk-Levin

Hui Henry Li

Michael Nelson

Yoshio Frank Turner

Anne Holler

Tariq Magdon-Ismail

Yunshan Luke Lu

Richard McDougall

Kinshuk Govil

Li Zheng

Razvan Cheveresan

Tarun Desikan

Nuwan S Jayasena

Daniel James Beveridge

Benedict R Gaster

Benjamin J Corrie

Rean Griffith

George Gregory Hicken

Junping Du

Laurent Morichetti

Systems and techniques are described for using virtual machines to write parallel and distributed applications. One of the techniques includes receiving a job request, wherein the job request specifies a first job to be performed by a plurality of a special purpose virtual machines, wherein the first job includes a plurality of tasks; selecting a parent special purpose virtual machine from a plurality of parent special purpose virtual machines to perform the first job; instantiating a plurality of child special purpose virtual machines from the selected parent special purpose virtual machine; partitioning the plurality of tasks among the plurality of child special purpose virtual machines by assigning one or more of the plurality of tasks to each of the child special purpose virtual machines; and performing the first job by causing each of the child special purpose virtual machines to execute the tasks assigned to the child special purpose virtual machine.

Parallel and distributed computing using multiple virtual machines

A control system authorizes access to a networked resource. The control system includes a client agent associated with a client resource running at a user device, and a destination agent associated the networked resource. The client agent transparently injects one or more identity tokens associated with the client resource and one or more access tokens associated with the networked resource into a network request issued by the client resource and directed to the networked resource. The destination agent intercepts the network request and uses the access tokens to selectively route the network request in accordance with one or more security policies associated with the access tokens.

Methods and systems for use in authorizing access to a networked resource

Embodiments include an infrastructure shared among cloud services that supports fast provisioning of virtual machines (VMs). A set of powered-on parent VM templates and a set of powered-off child VMs are maintained by the infrastructure in a hierarchy. The child VMs are instantiated from the parent VM templates, and pre-registered to a cloud operating system in some embodiments. In response to requests from the cloud services for the child VMs, where the requests specify child VM configurations, child VMs from the set of powered-off child VMs are selected and customized based on the child VM configurations, and then deployed for use by the cloud services. In some embodiments, the fast provisioning of VMs is supported by forking operations in the infrastructure.

Elastic compute fabric using virtual machine templates

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.

Multi-tenant distributed computing and database

A control system facilitates communication between a plurality of networked services. The control system includes a client agent associated with a first service of the networked services, and a destination agent associated with a second service of the networked services. The client agent includes an injection mechanism that intercepts a network request issued by the first service, transparently injects a token into the network request while the network request is in transit, and automatically transmits the network request to the second service in accordance with one or more security policies associated with the second service. The destination agent includes an interception mechanism that intercepts the network request, extracts the tokens from the network request, and determines whether to forward the network request to the second service.

Transparently enhanced authentication and authorization between networked services

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Production, test, and development deployments of a Hadoop application may be executed using multiple compute clusters and a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Data nodes executing as virtual machines (VMs) for test and development deployments can be linked clones of data nodes executing as VMs for a production deployment to reduce duplicated data and provide a shared storage space.

Multi-tenant production and test deployments of Hadoop

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

Container virtual machines for hadoop

Techniques for handling inheritance of disk state when forking virtual machines (VMs) are provided. In one embodiment, a computer system can receive a request to fork a child VM from a parent VM. In response, the computer system can take a disk snapshot of the parent VM, where the disk snapshot results in a child disk for the child VM, where the child disk is a delta disk that points to a parent disk of the parent VM, and where the parent disk serves as the parent VM's current running point. The computer system can then determine whether the parent disk is a delta disk. If so, the computer system can copy the content of the parent disk to the child disk, traverse a disk hierarchy associated with the parent disk to identify a base disk above the parent disk in the hierarchy, and cause the child disk to point directly to the base disk.

Handling disk state inheritance for forked virtual machines

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Multiple instances of a distributed computing framework, such as Hadoop, may be executed concurrently. A centralized manager detects when contention for computing resources, such as memory and CPU, causes tasks to run slower on VMs executing on a given host, and scales up or scales down a cluster based on the detected resource contention.

Virtual Hadoop manager

Embodiments support instant forking of virtual machines (VMs) and state customization. A computing device initiates execution of a first group of services (e.g., identity-independent) in a first VM. A second VM is instantiated from the first VM. The second VM shares memory and storage with the first VM. The computing device customizes the second VM based on configuration data associated with the second VM. A second group of services (e.g., identity-dependent) starts executing on the second VM after configuring the identity of the second VM. Customizing the second VM includes configuring one or more identities of the second VM. In some embodiments, a domain identity is selected from a pool of previously-created identities and applied to the second VM, before bootup completes on the second VM.

State customization of forked virtual machines

A method for executing a plurality of data-parallel threads of a processing logic on a processor core includes grouping the plurality of data-parallel threads into one or more workgroups, associating a first workgroup from the one or more workgroups with an operating system thread on the processor core, and configuring threads from the first workgroup as user-level threads within the operating system thread. In an example, a method enables the execution of GPU-kernels that has been previously configured for a GPU, to execute on a CPU such as a multi-core CPU. The mapping of the numerous data-parallel threads to the CPU is done in such a manner as to reduce the number of costly operating system threads instantiated on the CPU, and to enable efficient debugging.

Mapping processing logic having data-parallel threads across processors

Application resource scheduler module is provided to achieve cooperative application workload scheduling for a consolidated virtual environment. The application resource scheduler aids an application workload scheduler that is part of a distributed computing application, such as Hadoop, to achieve a specified relative priority of the application workload virtual machines to other virtual machines in the virtual environment. The application resource scheduler assists in achieving cooperative workload scheduling by revising the amount of resources that the application workload scheduler sees as available and by setting resource controls for the virtual machines of the distributed computing application to influence the resources the virtual machines receive from the underlying consolidated virtual environment.

Cooperative application workload scheduling for a consolidated virtual environment

A system and method embodiments for optimally allocating compute kernels to different types of processors, such as CPUs and GPUs, in a heterogeneous computer system are disclosed. These include comparing a kernel profile of a compute kernel to respective processor profiles of a plurality of processors in a heterogeneous computer system, selecting at least one processor from the plurality of processors based upon the comparing, and scheduling the compute kernel for execution in the selected at least one processor.

Growing community of inventors

San Francisco, CA, United States of America

Jayanth Gummaraju