Emc Corporation

Amazon Technologies, Inc.

Wayne William Duso

John Forecast

Sorin Faibish

Peter C Bixby

Seyfullah H Oguz

Daniel Gardere

Uday K Gupta

Uresh K Vahalia

Matti Juhani Oikarinen

Matteo Frigo

Dinesh Venkatesh

Michel Noury

Ugur Sezer

Danny Wei

Dennis P Ting

Michael Anthony Nuss

Vaibhav Tyagi

Alexander Sirotin

Ryan Michael Eccles

Siddhartha Roy

Zhicong Wang

James Christopher Sorenson

Pradeep Vincent

Percy Tzelnic

Brian William Gallagher

Peter Croft Jones

Daniel Nussbaum

Dennis P J Ting

Jacob A Strauss

Lev Vaitzblit

Boo Boon Khoo

Michael Robert Frasca

Neal John Charbonneau

Jacob David Luszcz

Xiaobin Wu

John McClain

Sebastian Keller

Raymond Mantchala

Jean Louis Rochette

Leslie E Kuczynski

Mark Allen Fogleman

William J Leary, Jr

Edward William Naim

Kannan M Alagappan

Steven Robert DeVos

Christopher John Schilling

Ivan Velickovic

William O Hultin

Iii James Christopher Sorenson

Systems and methods are described herein for routing data by transferring a physical storage device for at least part of a route between source and destination locations. In one example, a computing resource service provider, may receive a request to transfer data from a customer center to a data center. The service provider may determine a route, which includes one or more of a physical path or a network path, for the data loaded onto a physical storage device to reach the data center from the customer center. Determining the route may include associating respective cost values to individual physical and network paths between physical stations between the customer and end data centers, and selecting one or more of the paths to reduce a total cost of the route. Route information may then be associated with the physical storage device based on the route.

Dynamic physical data transfer routing

A cross-account data management (CAM) service of a provider network may assign, to a primary account of an organization of a client, permission to manage resource management plans for other accounts of the organization. The CAM service may specify, using the primary account (e.g., by an administrator using the primary account), a resource management plan (e.g., data backup plan) The CAM service may indicate, using the primary account, multiple accounts of the organization that the resource management plan is to be implemented for. The CAM service may cause, based on the permission assigned to the primary account, the resource management plan to be implemented for the different accounts of the organization (e.g., by causing execution of jobs to implement a backup plan).

Using a primary account to implement a resource management plan across accounts of an organization

Aspects described herein relate to securely performing cross-boundary backup operations. A service of a computing resource service provider may enable backup operations between a source account and a destination account of an organization based at least in part on a security policy allowing such operations.

Cross-boundary data backup background

A client request, formatted in accordance with a file system interface, is received at an access subsystem of a distributed multi-tenant storage service. After the request is authenticated at the access subsystem, an atomic metadata operation comprising a group of file system metadata modifications is initiated, including a first metadata modification at a first node of a metadata subsystem of the storage service and a second metadata modification at a second node of the metadata subsystem. A plurality of replicas of at least one data modification corresponding to the request are saved at respective storage nodes of the service.

Scalable file storage service

A low latency metadata subsystem for file systems in low latency (LL) mode in a distributed file storage service (DFSS). An LL server (LLS) may receive metadata requests from an access node of the DFSS. For read operations, the LLS may check a local cache of metadata and, for cache hits, retrieve the metadata from the cache. For cache misses, the metadata may be fetched from the storage subsystem of the DFSS and cached. For write operations, the LLS may write entries into a journal for the file system and notify the access node after the journal entries are committed to the journal; the journal entries are asynchronously committed in the storage subsystem. The access node may communicate with the storage subsystem to perform data reads and writes for the LL file system.

Low latency distributed storage service

An I/O request directed to a portion of a storage object managed at a distributed storage service is received. A congestion control parameter value to be used to schedule a storage operation corresponding to the I/O request is determined. The congestion control parameter is based at least in part on an offset within the storage object to which the I/O request is directed. The storage operation is scheduled in accordance with the congestion control parameter at a selected physical storage device to which the portion of the storage object is mapped.

Congestion control in storage systems

Offset-based congestion control in storage systems

Access to movies ranging from very popular movies to unpopular movies is managed by configuring sets of data movers for associated movie rankings, reserving data mover local cache resources for the most popular movies, reserving a certain number of streams for popular movies, negotiating with a client for selection of available movie titles during peak demand when resources are not available to start any freely-selected movie in disk storage, and managing disk bandwidth and primary and local cache memory and bandwidth resources for popular and unpopular movies. The assignment of resources to movie rankings may remain the same while the rankings of the movies are adjusted, for example, during off-peak hours. A movie locked in primary cache and providing a source for servicing a number of video streams may be demoted from primary cache to disk in favor of servicing one or more streams of a higher-ranking movie.

Video file server cache management using movie ratings for reservation of memory and bandwidth resources

A system for producing multiple concurrent real-time video streams from stored MPEG video clips includes a video server and at least one MPEG decoder array. The decoder array has multiple decoder pairs, each pair having a video switch for switching from one decoder in the pair to the other at a specified time. Switching may occur from a specified Out-point frame to a specified In-point frame, and the specified frames can be any frame type at any location in the group of pictures (GOP) structure. In a preferred construction, the video server has a controller server linked to a series of data mover computers, each controlling one or more respective decoder arrays. The data mover computers use a control protocol to control the decoder arrays, and each decoder uses a data protocol to request data from a respective data mover computer.

MPEG dual-channel decoder data and control protocols for real-time video streaming

Metadata for splicing of an encoded digital motion video stream (such as an MPEG Transport Stream) is prepared in real time while recording at the encoding bit rate and faster than encoded bit rate for off line encoding independent of the bit rate and mechanisms for ingestion of the data stream into data storage. Preprocessing is performed during a metered file transfer protocol (FTP) and includes pseudo real-time encoding. The preprocessing includes Group of Pictures (GOP) level pre-processing of splicing In Points and results in an intimate linkage between metadata and the file system in which the video data is stored. The preferred file system enables access to metadata in parallel to writing the data on disk. The pre-processing is performed simultaneous to writing the data to the disk using a carousel type buffer mechanism.

Preparation of metadata for splicing of encoded MPEG video and audio

The (run, level) pairs in an original series are inspected to determine whether or not modification of the (run, level) pairs would produce a desirable decrease in a number of bits required for variable-length encoding of the series of (run, level) pairs, despite introduction of noise. If so, the (run, level) pairs are modified prior to variable-length encoding. For example, a (run, level) pair of (M, N) is modified by substitution of a first (run, level) pair of (M&#x2212;1, 1) immediately followed by a second (run, level) pair of (0, N). A lookup table or testing of predetermined ranges of run length and level magnitude provides a fast determination of whether or not to modify a (run, level) pair. The decoder can be programmed to reduce the noise introduced by this process by recognizing and rejecting (run, level) pairs that are likely to have been inserted during the encoding process.

Insertion of noise for reduction in the number of bits for variable-length coding of (run, level) pairs

Transform coefficients for blocks of pixels in an original picture are quantized to produce respective sets of quantization indices for the blocks of pixels. The quantization indices for at least some of the blocks are produced by using a quantization step size that is not uniform within each block. Largest magnitude quantization indices are selected from the respective sets of quantization indices for (run, level) encoding to produce the (run, level) encoded picture. For example, MPEG-2 coded video includes a set of non-zero AC discrete cosine transform (DCT) coefficients for 8&#xd7;8 blocks of pixels. For scaling the MPEG-2 coded video, non-zero AC DCT coefficients are removed from the MPEG-2 coded video to produce reduced-quality MPEG-2 coded video that includes no more than a selected number of largest magnitude quantization indices for the non-zero AC DCT coefficients for each 8&#xd7;8 block.

Largest magnitude indices selection for (run, level) encoding of a block coded picture

Original-quality MPEG coded video is processed to produce reduced-quality MPEG coded video at a reduced bit rate. The processing is based on a scale factor between average frame size of the original-quality MPEG coded video and a desired average frame size of the reduced-quality MPEG coded video. For each Discrete Cosine Transform (DCT) block of each frame, the processing calculates a size of the block of the reduced frame by scaling the original block size by the scale factor, and removes a sufficient number of bits from the original block to obtain substantially the calculated size. In addition, the processing accumulates excess bits when the block size reduction eliminates more bits from a block than are necessary for the desired reduction of the size of the block, and any excess bits are used for processing a number of following blocks.

Adaptive bit rate control for rate reduction of MPEG coded video

Coded video from an on-line MPEG video encoder is stored as a clip in a video server or is otherwise received in the video server and prepared or used for splicing. In order to reduce apparent frame inaccuracy that may result from the splicing process, the on-line MPEG video encoder and the server are coordinated so that the group-of-picture (GOP) structure in the encoder provides specified In-points and Out-points that are valid and desirable for splicing. An encoder control protocol is also provided for remote control of the on-line MPEG video encoder in order to coordinate the on-line MPEG video encoder with the video server.

MPEG encoder control protocol for on-line encoding and MPEG data storage

Growing community of inventors

Shrewsbury, MA, United States of America

Wayne William Duso