Vmware, Inc.

Oracle America, Inc.

Sun Microsystems, Inc.

Richard McDougall

Brian L Wong

Spencer Shepler

David G Robinson

Michael Nelson

Benjamin J Corrie

Tariq Magdon-Ismail

Jayanth Gummaraju

Wei Lun Huang

Razvan Cheveresan

Daniel J Scales

Anne Holler

Michal Ostrowski

Kinshuk Govil

Rean Griffith

Junping Du

Glenn A Colaco

Denis J Sheahan

Ben Corrie

Methods, systems, and computer programs for providing an application server appliance utilizing one or more virtual machines are described. The application server appliance may be a virtual machine having a reduced guest operating system, a runtime environment, and a management agent installed therein. An appliance controller automatically determines one or more configurations and/or settings for the runtime environment based on a variety of factors, including the set up of the virtual machine appliance. The appliance controller generates an application package having the determined settings and transmits the package to the application server appliance, wherein the application package is configured to be executed by the runtime environment.

Execution of an application in a runtime environment installed in a virtual appliance

An elastic filesystem for temporary data provides storage space for virtual machines (VMs) in a distributed computing system. The filesystem redirects accesses to virtual disks in VMs to a common pool file. The system provides performance and storage efficiency at least on par with local, direct attached virtual disks, while providing a single pool of shared storage that is provisioned and managed independently of the VMs. The system provides storage isolation between VMs storing temporary data in that shared pool. Also, storage space for temporary data may be allocated on demand and reclaimed when no longer needed, thereby supporting a wide variety of temporary space requirements for different Hadoop jobs.

Elastic temporary filesystem

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

Container virtual machines for hadoop

Methods, systems, and computer programs for managing memory in a host where virtual machines (VMs) execute are presented. In one example, a method includes an operation for determining which amount of heap memory has been reserved in a Java virtual machine (JVM) that is in excess of the heap memory needed by the JVM. If there is excess heap memory, a Java balloon agent reserves a Java object in the heap memory. Typically, the Java object will be the size of one or more memory pages in the host. Further, the Java balloon agent loads the Java object with a certain value, for example, by zeroing out the page of memory. When a virtual machine monitor (VMM) in the host detects that a machine physical memory page associated with the Java object has the first value, then the VMM frees the machine physical memory page to make the memory available to other VMs or to other processes executing in the host.

Cooperative memory resource management via application-level balloon

Application resource scheduler module is provided to achieve cooperative application workload scheduling for a consolidated virtual environment. The application resource scheduler aids an application workload scheduler that is part of a distributed computing application, such as Hadoop, to achieve a specified relative priority of the application workload virtual machines to other virtual machines in the virtual environment. The application resource scheduler assists in achieving cooperative workload scheduling by revising the amount of resources that the application workload scheduler sees as available and by setting resource controls for the virtual machines of the distributed computing application to influence the resources the virtual machines receive from the underlying consolidated virtual environment.

Cooperative application workload scheduling for a consolidated virtual environment

Methods, systems, and computer programs for managing memory in a host where virtual machines (VMs) execute are presented. In one embodiment, a method includes an operation for determining which amount of heap memory has been reserved in a Java virtual machine (JVM) that is in excess of the heap memory needed by the JVM. If there is excess heap memory, a Java balloon agent reserves a Java object in the heap memory. Typically, the Java object will be the size of one or more memory pages in the host. Further, the Java balloon agent loads the Java object with a certain value, for example, by zeroing out the page of memory. When a virtual machine monitor (VMM) in the host detects that a machine physical memory page associated with the Java object has the first value, then the VMM frees the machine physical memory page to make the memory available to other VMs or to other processes executing in the host.

A system and method for emulating the input/output performance of an application. A workload description language is used to produce a small but accurate model of the application, which is flexible enough to emulate the application's performance with varying underlying system configurations or operating parameters. The model describes I/O operations performed by the application, and reflects any dependencies that exist between different application threads or processes. The model is then executed or interpreted with a particular system configuration, and various parameters of the I/O operations may be set at the model's run-time. During execution, the input/output operations described in the model are generated according to the specified parameters, and are performed. The system configuration and/or I/O operation parameters may be altered and the model may be re-run.

System and method for emulating input/output performance of an application

A system and method for ensuring or verifying the integrity of data transmitted between protection domains. When the data is transmitted, it may be received in a different logical configuration (e.g., as a different number of 'chunks'). The receiving domain computes its data integrity metadata (e.g., checksum, CRC, parity) on its form of the data using its protection scheme (e.g., checksum algorithm), and also applies the sending domain's protection scheme to the data as it was received from the sending domain. Similarly, the sending domain applies the receiving domain's protection scheme to compute data integrity metadata on the transmitted data as it appears in the receiving domain. The metadata may be compared to determine whether the data was corrupted during the transfer. Either domain may forward its data integrity metadata to the other, which may store and/or forward it as needed.

System and method for protecting data across protection domain boundaries

A system and method for decreasing the protocol processing incurred by an NFS (Network File System) client, and the network bandwidth consumed, when multiple NFS requests are issued close in time to each other. Requests received close in time to each other within one NFS client are aggregated into a single communication packet, even if they are from different NFS users, processes or applications. The per-request overhead of the NFS requests is thus avoided, and multiple streams of requests may be merged into one. When a first request is received by the client, it may be delayed for a short time to determine whether any other requests can be sent at the same time. NFS requests may be aggregated on an NFS client from multiple applications, processes, users or other entities. An NFS client may include an aggregator or other mechanism for performing the aggregation.

System and method for aggregating NFS requests

A system and method for facilitating a consistent point-in-time copy (PITC). A data provider offers access to data by one or more clients. A client issues a PITC request for a set of data to the data provider, or the provider initiates its own request. The data provider instructs one or more clients, particularly clients accessing the set of data, to flush any pending updates involving the files. The clients acknowledge the flush request, flush their data, and notify the data provider when their flushes are complete. Post-consistency updates are deferred (e.g., using a write-aside buffer). If a client fails to acknowledge the flush request or notify of flush completion, the data provider may abort the PITC or continue because the data may not be consistent or consistency may not be assured. If not aborted, the PITC may be marked to indicate that it is incomplete.

System and method for facilitating a consistent point-in time copy

A method for dirty region logging of a file that includes receiving a request to open the file, determining each of a plurality of component files associated with the file, opening each of the plurality of component files, writing to a region of the at least one of the plurality component files, and updating a dirty region log (DRL) associated with the one of the plurality of component files to reflect the write to the region.

Per file dirty region logging

A file is striped across multiple filers, file servers or other devices, to create a sparsely striped multi-component file. Each filer stores one sparse component. In particular, each component physically stores only those stripes allocated to that component. The other stripes are represented as holes. Thus, instead of contiguously packing each component's stripes at the block level, each component is a file having the same logical structure. A component of a sparsely striped multi-component file can be easily converted to a mirror by filling in its holes. Similarly, a mirror can be easily converted to one component of a sparsely striped multi-component file by removing or ignoring it unallocated stripes. In either case, the layout or logical of the component does not need to be reconfigured.

Sparse multi-component files

One embodiment of the present invention provides a system that facilitates sampling a cache in a computer system, wherein the computer system has multiple central processing units (CPUs), including a measured CPU containing the cache to be sampled, and a sampling CPU that gathers the sample. During operation, the measured CPU receives an interrupt generated by the sampling CPU, wherein the interrupt identifies a portion of the cache to be sampled. In response to receiving this interrupt, the measured CPU copies data from the identified portion of the cache into a shared memory buffer that is accessible by both the measured CPU and the sampling CPU. Next, the measured CPU notifies the sampling CPU that the shared memory buffer contains the data, thereby allowing the sampling CPU to gather and process the data.

Growing community of inventors

Menlo Park, CA, United States of America

Richard McDougall