Amazon Technologies, Inc.

Greg Burgess

Marvin Michael Theimer

John David Dunagan

Gaurav D Ghare

Ying Xiong

A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.

Data stream ingestion and persistence techniques

A control node of a multi-tenant stream management service receives a request to initialize a data stream to be comprised of a plurality of data records. The control node determines, based on a partitioning policy, parameters to be used to configure subsystems for ingestion, storage and retrieval of the records. The control node identifies resources to be used for a node of retrieval subsystem The retrieval node is configured to implement programmatic record retrieval interfaces, including respective interfaces to implement non-sequential and sequential access patterns. The control node configures the retrieval node using the selected resources.

Managed service for acquisition, storage and consumption of large-scale data streams

A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.

Dynamic partitioning techniques for data streams

A configuration request comprising a security option selected for a particular data stream is received. Nodes of a plurality of functional categories, such as a data ingestion category and a data retrieval category are to be configured for the stream. The security option indicates a security profile of a resource to be used for nodes of at least one functional category. In accordance with the configuration request, a node of a first functional category is configured at a resource with a first security profile, and configuration of a node of a second functional category is initiated at a different resource with a different security profile.

Growing community of inventors

Seattle, WA, United States of America

Greg Burgess