Amazon Technologies, Inc.

Fletcher Liverance

Chance Ackley

Dominic Corona

Long Xuan Nguyen

Ryan Paul Thomas

Stephanie Moyerman

Daniel Opincariu

Yangbae Park

Donald Erik Schneider

Alex Guazzelli

Ramesh Sekaran

John Sitar

Laxmi Siva Prasad Balaramaraju Jalumari

Kevin Yavno

David Said Martinez

Zhuonan Song

Systems and methods are disclosed to implement an indexing engine that maintains an index in an index store for a storage object in a data store. In embodiments, the index store may be implemented using an in-memory storage cluster separate from the data store. The storage object may have multiple indexes, which may have different filtering or sorting criteria for the data. In embodiments, updates to the storage object are received as an update stream by the indexing engine. Based on configurable indexing rules, the indexing engine applies the updates to the appropriate indexes. To service a query to the data store, a query engine first retrieves a set of keys satisfying the query from the index store, and then data corresponding to the keys from the data store or another index. In embodiments, the index may be refreshed via touch updates of selected data in the storage object.

Data store indexing engine with automated refresh

Methods, systems, and computer-readable media for indexing partitions using distributed Bloom filters are disclosed. A data indexing system generates a plurality of indices for a plurality of partitions in a distributed object store. The indices comprise a plurality of Bloom filters. An individual one of the Bloom filters corresponds to one or more fields of an individual one of the partitions. Using the Bloom filters, the data indexing system determines a first portion of the partitions that possibly comprise a value and a second portion of the partitions that do not comprise the value. Based (at least in part) on a scan of the first portion of the partitions and not the second portion of the partitions, the data indexing system determines one or more partitions of the first portion of the partitions that comprise the value.

Indexing partitions using distributed bloom filters

Methods, systems, and computer-readable media for a transformation specification format for multiple execution engines are disclosed. A transformation specification is expressed according to a transformation specification format. The transformation specification represents a polytree or graph linking one or more data producer nodes, one or more data transformation nodes, and one or more data consumer nodes. An execution engine is selected from among a plurality of available execution engines for execution of the transformation specification. The execution engine is used to acquire data from one or more data producers corresponding to the one or more data producer nodes, perform one or more transformations of the data corresponding to the one or more data transformation nodes, and output one or more results of the one or more transformations to one or more data consumers corresponding to the one or more data consumer nodes.

Transformation specification format for multiple execution engines

Techniques are generally described for adding transactional support to a distributed storage environment. In various examples, first data may be written to a first set of locations in a distributed computer-readable non-transitory storage system through a non-transactional file system interface. In various further examples, metadata associated with the first data may be generated during the writing of the first data. In some examples, the metadata may be stored associated with the first data in at least a second location in a second computer-readable non-transitory memory. In some examples, a manifest may be generated defining a transactional commit of at least a portion of the first data. In some examples, the manifest may be generated by processing the metadata using first committer logic. In some further examples, the manifest may be stored in a third computer-readable non-transitory memory.

Non-native transactional support for distributed computing environments

Methods, systems, and computer-readable media for lottery-based resource allocation with capacity guarantees are disclosed. A job request is received from a first client. The job request is submitted to a capacity management system that schedules jobs in a pool of compute resources. The pool comprises a first quantity of one or more slots and a second quantity of one or more slots. The second quantity is associated with a capacity guarantee for a second client. It is determined that the first quantity of one or more slots are in use by one or more jobs initiated prior to receiving the job request. It is determined that the second quantity of one or more slots comprises an available slot. The available slot is allocated to the job request.

Lottery-based resource allocation with capacity guarantees

Methods, systems, and computer-readable media for unified code and data management for machine learning models are disclosed. A plurality of dependency graphs, including a first dependency graph and a second dependency graph, are generated. The graphs comprise nodes associated with a software development model or machine learning model, and the nodes represent transforms. One or more transforms of the first dependency graph are used to generate first output corresponding to a node in the second dependency graph. One or more transforms of the second dependency graph are used to generate second output based at least in part on the first output.

Growing community of inventors

Oakland, CA, United States of America

Fletcher Liverance