Vmware, Inc.

Hewlett Packard Enterprise Development LP

Bluedata Software, Inc.

Wmware, Inc.

Thomas A Phelan

Joel Baxter

Michael J Moretti

Deng Liu

Gunaseelan Lakshminarayanan

Kiran Prakash Madnani

Mayank Rawat

Sambasiva Rao Bandarupalli

Erik C Cota-Robles

Wei Zhang

Swami Viswanathan

Ramaswami Kishore

Kumar Sreekanti

Li Zhou

Sandeep Madhav Uttamchandani

Ramkumar Vadivelu

Xiongbing Ou

Dragan Stancevic

David E Lee

David B Lee

Olivier F Lecomte

Adam Back

David W Barry

Lakshman Chinnakotla

Lakshminarayanan Gunaseelan

Sambasiva Bandarupalli

Secured data access in virtual data processing is described. An example includes instructions to receive a request from an application in a compute node of a compute cluster in a virtual data processing environment to access a secured data source for a user, the virtual data processing environment including a multiple secured data sources that are accessible by compute nodes of the virtual compute cluster; fetch a credential in a current application context and forward the credential for validation; validate the credential with a credential authority; and, upon successfully validating the credential, authenticate the user at the secured data source and establish a connection with the secured data source.

Secured data access in virtual data processing

Embodiments described herein are generally directed to caching and data access improvements in a large scale data processing environment. According to an example, an agent running on a first worker node of a cluster receives a read request from a task. The worker node of the cluster to which the data at issue is mapped is identified. When the first worker node is the identified worker node, it is determined whether its cache contains the data; if so, the data is fetched from a remote data lake and the agent locally caches the data; otherwise, when the identified worker node is another worker node of the compute cluster, the data is fetched from a remote agent of that worker node. The agent responds to the read request with cached data, data returned by the remote data lake, or data returned by the remote data agent as the case may be.

Efficient caching and data access to a remote data lake in a large scale data processing environment

Techniques for using a cache to accelerate virtual machine (VM) I/O are provided. In one embodiment, a host system can intercept an I/O request from a VM running on the host system, where the I/O request is directed to a virtual disk residing on a shared storage device. The host system can then process the I/O request by accessing a cache that resides on one or more cache devices directly attached to the host system, where the accessing of the cache is transparent to the VM.

Transparent host-side caching of virtual disks located on shared storage

Systems, methods, and software described herein to provide data to large-scale processing framework (LSPF) nodes in LSPF clusters. In one example, a method to provide data includes receiving an access request from a LSPF node to access data in accordance with a version of a distributed file system. The method further includes, responsive to the access request, accessing the data for the LSPF node in accordance with a different version of the distributed file system, and presenting the data to the LSPF node in accordance with the version of the distributed file system used by the LSPF node.

Inter-version mapping of distributed file systems

A cache is maintained with write order numbers that indicate orders of writes into the cache, so that periodic partial flushes of the cache can be executed while maintaining write order consistency. A method of storing data into the cache includes receiving a request to write data into the cache, identifying lines in the cache for storing the data, writing the data into the lines of the cache, storing a write order number, and associating the write order number with the lines of the cache. A method of flushing a cache having cache lines associated with write order numbers includes the steps of identifying lines in the cache that are associated with either a selected write order number or a write order number that is less than the selected write order number, and flushing data stored in the identified lines to a persistent storage.

Process for maintaining data write ordering through a cache

Systems, methods, and software described herein facilitate interfacing between processing nodes and a plurality of data repositories. In one example, a method of interfacing between a processing node and a plurality of data repositories includes identifying, for the processing node, a data access request using a first data access format, wherein the data access request includes a data connector identifier. The method further includes translating the access request to a second data access format based on the data connector identifier, and identifying a data repository in the plurality of data repositories to service the data access request based on the data connector identifier. The method also provides accessing data for the data access request in the data repository via the second data access format.

Data connectors in large scale processing clusters

Systems, methods, and software described herein facilitate servicing of data requests based on quality of service assigned to processing jobs. In one example, a method of prioritizing data requests in a computing system based on quality of service includes identifying a plurality of data requests from a plurality of processing jobs. The method further includes prioritizing the plurality of data requests based on a quality of service assessed to each of the plurality of processing jobs, and assigning cache memory in the computing system to each of the plurality of data requests based on the prioritization.

Prioritizing data requests based on quality of service

Embodiments disclosed herein provide systems, methods, and computer readable media for managing data consumption rate in a virtual data processing environment. In a particular embodiment, a method provides, in a cache node of a host system, identifying read completions for one or more virtual machines instantiated in the host system, with the one or more virtual machines processing one or more processing jobs. The method further provides allocating the read completions to individual processing jobs of the one or more processing jobs and accumulating the read completions on a per-job basis, with the cache node determining a data consumption rate for each processing job of the one or more processing jobs.

Host system and method for managing data consumption rate in a virtual data processing environment

Described herein are systems, methods, and software to enhance the allocation of cache resources to virtual nodes. In one implementation, a configuration system in large-scale processing environment is configured to identify a request to initiate a large-scale processing framework (LSPF) cluster, wherein the LSPF cluster comprises a plurality of virtual nodes, and identify host computing resources of a host computing system allocated to each virtual node of the LSPF cluster. The configuration system further allocates cache memory of a cache service to each of the virtual nodes based on the host computing resources, and initiate the LSPF cluster in the computing environment.

Enhanced cache memory allocation based on virtual node resources

Systems, methods, and software described herein facilitate accelerated input and output operations with respect to virtualized environments. In an implementation, a computing system passes a process identifier to a kernel driver for a host environment, wherein the process identifier identifies a guest process spawned in a virtual machine and wherein the kernel driver uses the process identifier to determine an allocation of host memory corresponding to guest memory for the guest process and returns the allocation of host memory. Additionally, the computing system performs a mapping of the allocation of host memory to an allocation of guest memory for the guest element.

Accelerated data operations in virtual environments

A computer system that employs a solid-state memory device as a physical storage resource includes a hypervisor that is capable of supporting TRIM commands issued by virtual machines running in the computer system. When a virtual machine issues a TRIM command to its corresponding virtual storage device to invalidate data stored therein, the TRIM command is received at an interface layer in the hypervisor that translates the TRIM command to a SCSI command known as UMMAP. A SCSI virtualization layer converts the UNMAP command to a file system command to delete portions of the virtual storage device that is maintained as a file in the hypervisor's file system. Upon receiving the delete commands, the hypervisor's file system driver generates a TRIM command to invalidate the data stored in the solid-state memory device at locations corresponding to the portions of the file that are to be deleted.

Trim support for a solid-state drive in a virtualized environment

Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Employing application containers in a large scale processing environment

Systems, methods, and software described herein allocate cache memory to job processes executing on a processing node. In one example, a method of allocating cache memory to a plurality of job process includes identifying the plurality of job processes executing on a processing node, and identifying a data object to be accessed by the plurality of job processes. The method further provides allocating a portion of the cache memory to each job process in the plurality of job processes and, for each job process in the plurality of job processes, identifying a segment of data from the data object, wherein the segment of data comprises a requested portion of data and a predicted portion of data. The method also includes providing the segments of data to the allocated portions of the cache memory.

Allocating cache memory on a per data object basis

Systems, methods, and software described herein facilitate an enhanced service architecture for large-scale data processing. In one implementation, a method of providing data to a large-scale data processing architecture includes identifying a data request from a container in a plurality of containers executing on a host system, wherein the plurality of containers each run an instance of a large-scale processing framework. The method further provides identifying a storage repository for the data request, and accessing data associated with the data request from the storage repository. The method also includes caching the data in a portion of a cache memory on the host system allocated to the container, wherein the cache memory comprises a plurality of portions each allocated to one of the plurality of containers.

Growing community of inventors

San Francisco, CA, United States of America

Thomas A Phelan