Hewlett Packard Enterprise Development LP

Bluedata Software, Inc.

Applied Micro Circuits Corporation

Other

Qualcomm Incorporated

Samsung Electronics Co., Ltd.

Michael J Moretti

Thomas A Phelan

Joel Baxter

Gunaseelan Lakshminarayanan

Salil Suri

Scott Furey

David Geddes

Thomas Wu

Swami Viswanathan

Kumar Sreekanti

Dragan Stancevic

Ramaswami Kishore

Lakshminarayanan Gunaseelan

Lakshman Chinnakotla

Dennis McCann

Systems, methods, and software described herein to provide data to large-scale processing framework (LSPF) nodes in LSPF clusters. In one example, a method to provide data includes receiving an access request from a LSPF node to access data in accordance with a version of a distributed file system. The method further includes, responsive to the access request, accessing the data for the LSPF node in accordance with a different version of the distributed file system, and presenting the data to the LSPF node in accordance with the version of the distributed file system used by the LSPF node.

Inter-version mapping of distributed file systems

Systems, methods, and software described herein facilitate interfacing between processing nodes and a plurality of data repositories. In one example, a method of interfacing between a processing node and a plurality of data repositories includes identifying, for the processing node, a data access request using a first data access format, wherein the data access request includes a data connector identifier. The method further includes translating the access request to a second data access format based on the data connector identifier, and identifying a data repository in the plurality of data repositories to service the data access request based on the data connector identifier. The method also provides accessing data for the data access request in the data repository via the second data access format.

Data connectors in large scale processing clusters

Systems, methods, and software described herein facilitate servicing of data requests based on quality of service assigned to processing jobs. In one example, a method of prioritizing data requests in a computing system based on quality of service includes identifying a plurality of data requests from a plurality of processing jobs. The method further includes prioritizing the plurality of data requests based on a quality of service assessed to each of the plurality of processing jobs, and assigning cache memory in the computing system to each of the plurality of data requests based on the prioritization.

Prioritizing data requests based on quality of service

Systems, methods, and software described herein facilitate accelerated input and output operations with respect to virtualized environments. In an implementation, a computing system passes a process identifier to a kernel driver for a host environment, wherein the process identifier identifies a guest process spawned in a virtual machine and wherein the kernel driver uses the process identifier to determine an allocation of host memory corresponding to guest memory for the guest process and returns the allocation of host memory. Additionally, the computing system performs a mapping of the allocation of host memory to an allocation of guest memory for the guest element.

Accelerated data operations in virtual environments

Systems, methods, and software described herein provide accelerated input and output of data in a work process. In one example, a method of operating a support process within a computing system for providing accelerated input and output for a work process includes monitoring for a file mapping attempt initiated by the work process. The method further includes, in response to the file mapping attempt, identifying a first region in memory already allocated to a cache service, and associating the first region in memory with the work process.

Associating cache memory with a work process

Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.

Employing application containers in a large scale processing environment

Systems, methods, and software described herein allocate cache memory to job processes executing on a processing node. In one example, a method of allocating cache memory to a plurality of job process includes identifying the plurality of job processes executing on a processing node, and identifying a data object to be accessed by the plurality of job processes. The method further provides allocating a portion of the cache memory to each job process in the plurality of job processes and, for each job process in the plurality of job processes, identifying a segment of data from the data object, wherein the segment of data comprises a requested portion of data and a predicted portion of data. The method also includes providing the segments of data to the allocated portions of the cache memory.

Allocating cache memory on a per data object basis

Systems, methods, and software described herein facilitate an enhanced service architecture for large-scale data processing. In one implementation, a method of providing data to a large-scale data processing architecture includes identifying a data request from a container in a plurality of containers executing on a host system, wherein the plurality of containers each run an instance of a large-scale processing framework. The method further provides identifying a storage repository for the data request, and accessing data associated with the data request from the storage repository. The method also includes caching the data in a portion of a cache memory on the host system allocated to the container, wherein the cache memory comprises a plurality of portions each allocated to one of the plurality of containers.

Data caching in a large-scale processing environment

Systems, methods, and software described herein facilitate the allocation of large scale processing jobs to host computing systems. In one example, a method of operating an administration node to allocate processes to a plurality of host computing systems includes identifying a job process for a large scale processing environment (LSPE), and identifying a data repository associated with the job process. The method further includes obtaining data retrieval performance information related to the data repository and the host systems in the LSPE. The method also provides identifying a host system in the host systems for the job process based on the data retrieval performance information, and initiating a virtual node for the job process on the identified host system.

Allocation of large scale processing job processes to host computing systems

Systems, methods, and software described herein facilitate the allocation of large scale processing jobs to host computing systems. In one example, a method of allocating job processes to a plurality of host computing systems in a large scale processing environment includes identifying a job process for the large scale processing environment, and obtaining accommodation data for a plurality of host computing systems in the large scale processing environment. The method further provides identifying a host computing system in the plurality of host computing systems for the job process based on the accommodation data, and initiating a virtual node on the host computing system for the job process.

Allocation of job processes to host computing systems based on accommodation data

Systems, methods, and software described herein facilitate accelerated input and output operations with respect to virtualized environments. In an implementation, upon being notified of a guest read process initiated by a guest element running in a virtual machine to read data into a location in guest memory associated with the guest element, a computing system identifies a location in host memory associated with the location in the guest memory and initiates a host read process to read the data into the location in the host memory that corresponds to the location in the guest memory.

Growing community of inventors

Saratoga, CA, United States of America

Michael J Moretti