Amazon Technologies, Inc.

Sayantan Chakravorty

Benjamin Warren Mercier

Yasemin Avcular

Charlie Paucard

Allan Henry Vermeulen

Timothy Daniel Cole

Christopher Richard Jacques De Kadt

Carlos Vara Callau

Aaron Gifford Freshwater

Gaurav D Ghare

Vijay Gopal Joshi

Mushegh Malkhasyan

Nabanita Maji

Vijaya Rama Reddy Kistampalli

Vasudeva Gade

Sahil Dorwat

Sankalp Bhatia

Nagarjuna Koduru

Babu Prasad Dhandapani

Dennis Tighe

Damian Gorman Wylie

Artem Kryvokrysenko

Rohit Unnikrishnan Nair

Duc Viet Nguyen

Mitchell Ross Loeppky

Carlos Alessandro Chiconato

Gokul Ramanan Subramanian

Sumeetkumar Bagde

In response to a first programmatic request, metadata indicating that a first isolated read channel of a real-time category has been associated with a first target stream is stored at a stream management service. In response to another request, metadata indicating that a second isolated read channel of a non-real-time category has been associated with a second target stream is stored. In response to a read request indicating the first channel or the second channel, one or more data records of the corresponding target streams are provided.

Isolated read channel categories at streaming data service

Various embodiments of systems and methods for providing virtualized (e.g., serverless) broker clusters for a data streaming service are disclosed. A data streaming service uses a front-end proxy layer and a back-end broker layer to provide virtualized broker clusters, for example in a Kafka-based streaming service. Resources included in a virtualized broker cluster are monitored and automatically scaled-up, scaled-down, or re-balanced in a way that is transparent to data producing and/or data consuming clients of the data streaming service.

Data streaming service with virtualized broker clusters

A connection-based service impersonates request-based security for requests from clients that do not include credentials for the requests (e.g., data plane requests made via a connection-oriented security). A connection between a client and a connection-based service is established based on connection credentials that are based on security credentials from a request-based security service. The credentials are sent by a security component of the service to a local agent of the remote security service to be authenticated by the security service. An impersonation token is returned by the security service and cached by the local agent. Requests from the client to perform operations do not include credentials. For each request, the service passes an identifier for the client and the operation to a local authorization component that calls the agent for authorization of the requested operation. The agent uses the impersonation token to obtain authorization for the requested operation.

Impersonating request-based security in connection-based security environment

Configuration information indicating that one or more stream consumers are granted read-only access to contents of a shared-access data stream is stored at a stream management service. A virtual stream associated with the shared-access stream may be established. In response to a read request directed to the virtual stream, contents of a particular record of the shared-access data stream are provided.

Managed distribution of data stream contents

At a data stream management service, a first set of metadata indicating that a first isolated read channel has been associated with a first data stream is stored. The first isolated read channel has an associated read performance limit setting. A second set of metadata indicating that a second isolated read channel, with its own performance limit setting, has been associated with a data stream is also stored. Based on determining that the difference between a metric of read operations associated with the first channel and the read performance limit setting of the first channel meets a first criterion, the service initiates a throttling operation for reads associated with the first channel. The throttling decision is made independently of read metrics of the second channel.

Streaming data service with isolated read channels

From a stream management service, indications of transmissions of data records of a first data stream via a persistent network connection associated with a subscription are provided to a stream processing application. In response to detecting that a transfer throttling condition associated with the subscription has been met, a delay is introduced before additional data records are transmitted via the same persistent network connection. After a subscription expiration period has elapsed, an indication of the expiration is provided to the application via the persistent connection.

Stream data record reads using push-mode persistent connections

At a platform at which stream data retrieval requests directed to a stream management service are processed, respective metrics of reads being performed using a first set of one or more persistent network connections established between the platform and one or more stream processing applications are collected. A request to read additional stream data requests is directed to the platform based at least in part on a decision made by a load balancer. The request is rejected based on a determination that the collected metrics satisfy a threshold.

Workload management at streaming data service supporting persistent connections for reads

At a stream management service, one or more storage servers of a first repository are assigned to store at least a first partition of a first data stream. At the first repository, records of different partitions are interleaved. In response to determining that a set of data records of the first partition meet a first criterion, a rearranged version of the set in which the data records of the first partition are contiguous is stored at a second repository. Using the rearranged version, one or more records of the first partition are provided to an application which meets a lag criterion.

Optimization techniques to support lagging readers at streaming data service

Disclosed are various embodiments for consuming fragments of time-associated data streams. A streaming gateway receives a request from a consumer via a network to obtain a stream of time-associated data. The gateway determines that a first fragment of the stream of time-associated data is stored in a data store. The first fragment is obtained from the data store and sent to the consumer via the network. The gateway determines that the stream of time-associated data is currently being received by an endpoint from a producer. A second segment of the stream is obtained from the endpoint and sent to the consumer via the network.

Consuming fragments of time-associated data streams

In response to a first programmatic request, metadata indicating that a first isolated read channel associated with a data stream has been established is stored at a stream management service. A read request that indicates the first isolated read channel and the stream is obtained. In response to the read request, contents of data records of the stream are provided.

Isolated read channel management interfaces at streaming data service

Disclosed are various embodiments for fragmentation of time-associated streaming data. A producer application receives a stream of time-associated data. The producer application then generates a first fragment of the stream of time-associated data and assigns a first unique fragment identifier and a first timestamp to the first fragment. The producer application sends the first fragment via a network using an application-layer protocol to an endpoint. The producer application receives at least one acknowledgement for the first fragment from the endpoint via the network using the application-layer protocol.

Fragmentation of time-associated data streams

Various embodiments replicate and index fragments of a stream of time-associated data. In one embodiment, a stream endpoint application receives a first fragment of a stream of time-associated data from a producer via a network using an application-layer protocol. The stream endpoint application determines a producer timestamp from the first fragment. The stream endpoint application sends at least one acknowledgment to the producer via the network using the application-layer protocol. The stream endpoint application indexes the first fragment and sends the first fragment to a destination.

Growing community of inventors

Sammamish, WA, United States of America

Sayantan Chakravorty