Google Inc.

David Konerding

Jordan M Breckenridge

Daniel Belov

Jeffrey Adgate Dean

Sanjay Ghemawat

Jonathan Bingham

A global-level manager access a work order from a client and parameters associated with the work order. A service level agreement to meet the work order parameters is determined. The service level agreement includes a price. An indication is received from the client that the service level agreement is accepted. The one or more input files are partitioned into multiple shards, and the work order into multiple jobs. The jobs are distributed among a plurality of clusters to be processed using underutilized computing resources in the clusters. The job outputs are combined to form the work order output. The jobs are monitored to insure that the deadline for completion of the work order will be met.

Opportunistic job processing using worker processes comprising instances of executable processes created by work order binary code

Opportunistic job processing of input data divided into partitions and distributed amongst task level managers via a peer-to-peer mechanism supplied by a cluster cache

A method includes receiving, by a processing device, a plurality of genome files. Each genome file corresponds to a different sample and defining a genetic sequence. The method also includes generating, by the processing device, a two-dimensional alignment file based on the genome files and a reference sequence. A first dimension of the alignment file corresponds to individual genetic sequences and each of the genetic sequences is aligned with respect to the reference sequence along a second dimension of the alignment file. The method includes separating, by the processing device, the alignment file into a plurality of groups and storing the groups in a non-transitory genome data store. Each group contains segments of the genetic sequences of two or more of the genomic files.

Storing genetic data in a storage system

Opportunistic job processing of input data divided into partitions of different sizes

Opportunistic job Processing of input data divided into partitions of different sizes

A global-level manager accesses a work order referencing at least one executable file and one or more input files. The one or more input files are partitioned into multiple shards. The work order is partitioned into multiple jobs. The jobs are distributed among a plurality of to be processed by a task level manager at each of the plurality of clusters. The executable file is loaded into the native client environment through a validator. The validator is configured to insure that the executable file does not include one or more of a defined set of instructions, does not call instructions outside of the executable file, and does not read or write data outside of a data region associated with the executable file.

Opportunistic job processing in distributed computing resources with an instantiated native client environment with limited read/write access

A global-level manager receives a work order referencing at least one executable file and one or more input files. The one or more input files include multiple input resources. A first type of input resource is identified in the one or more input files and a second type of input resource in the one or more input files. The first type of input resource is split into a plurality of first-type input shards. The second type of input resource is split into a plurality of second-type input shards. The plurality of second-type input shards are associated with each of the first-type input shards. For each of the first-type input shards, the global-level manager distributes the first-type input shards, the associated second-type input shards, and the executable file to a single.

Opportunistic job processing

A central storage configured to store one or more input files and an executable file. A work order frontend is configured to receive, from the client, a representational state transfer (RESTful) message that contains a reference to the one or more input files in the central storage. The work order frontend is further configured to transmit, to a global-level manager, a work order. The global-level manager is configured to access the work order. The global-level manager is further configured to partition the one or more input files into multiple shards, the work order into multiple jobs, each job being associated with one or more of the multiple shards and the executable file. The global-level manager is further configured to distribute the jobs among a plurality of clusters.

Growing community of inventors

San Mateo, CA, United States of America

David Konerding