Yahoo! Inc.

Dell Products Inc.

Emc IP Holding Company LLC

Excalibur Ip, LLC

Nec Europe Ltd.

Yahoo Holdings, Inc.

Flavio Junqueira

Benjamin Clay Reed

Vassilis Plachouras

Shivesh Ranjan

Tom Kaitchuck

Ricardo Alberto Baeza-Yates

Aristides Gionis

Thomas Kaitchuck

Raúl Gracia-Tinedo

Vincent Leroy

Berkant Barla Cambazoglu

Maysam Yabandeh

Marco Serafini

Andrei Paduroiu

Lu Lei

Yi Liu

Saverio Niccolini

Adam Silberstein

Jan Seedorf

Qi Bao

Ingmar Weber

Luciano Barbosa

Martin Stiemerling

Ivan Kelly

Xiao Bai

Yurun Wu

Matthieu Morel

Shirish Tatikonda

Raúl Gracia Tinedo

Techniques described herein relate to a method for managing metadata for unstructured data. The method includes identifying a first storage event associated with a first unstructured data object; streaming, from a storage resource, first system metadata associated with the first unstructured data object; identifying, using a first system metadata processor, a first object type associated with the first unstructured data object based on the first system metadata; storing the first system metadata in a first entry of a first metadata table; streaming second system metadata associated with a second unstructured data object; identifying, using the first system metadata processor, a second object type associated with the second unstructured data object based on the second system metadata; and storing the second system metadata in a second entry of a second metadata table, wherein the second metadata is queried to obtain the first unstructured data object.

Method and system for extracting metadata from unstructured data

A method for managing a data stream (DS) includes: receiving a first request from an administrator; obtaining the DS from a streaming storage system (SSS); performing a first analysis of the DS to extract relevant data; making a first determination that the relevant data is worth indexing; generating a set of indexes corresponding at least a portion of the DS; receiving a query about a data item in the DS from a user; sending a second request to a database engine comprising the query; performing a second analysis of the query; making a second determination that a re-indexing of the portion of the DS is not required; identifying an index from the set of indexes that is associated with the query; sending position information of the data item to an API service; retrieving the data item from the SSS; and initiating displaying of the data item to the user.

Method and system for content-based indexing of data streams in streaming storage systems

A first user-initiated data write request including stream data is received from an edge device executing in a first zone. The stream data is partitioned into at least one segment and an event included in the stream data is written into the at least one segment based on an event's routing key. A segment container of a segment store hosts the at least one segment. The stream data is transmitted to a tier-1 storage of the first zone in order to provide a low-latency access to the stream data for the first user. Thereafter, the stream data is moved to a tier-2 storage of a second zone for permanent storage of the stream data. When requested by a second user operating in the second zone, a read-only version of the segment container retrieves the stream data from the tier-2 storage.

Method and system for addressing zone-awareness in a heterogeneous environment

A system of managing data, comprising a streaming data storage system operating on a processor that causes the processor to receive streaming data and to process the streaming data in response to an append command, a long-term storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a chunk of data, an object storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate an object of data and a tables system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a table of data.

Formatted long-term storage of streaming system to allow data access through both stream and object API

Event loading is individually tracked for each of the segments of a stream. Event loading trends of each of the segments of the stream based on event loading changes are identified and reported. Selected segments are automatically scaled-up or scaled-down based on the event loading trends. Individual segment event loading is measured by computing an exponential weighted moving average of data, measured either in events or bytes, received per fixed interval of time. Exponential functions are used to assign exponentially decreasing weights over time. Multiple different running averages are computed for different time intervals, each with a different weight that represents a decay factor.

Automatically scaling streams in distributed stream storage

Processing of continuously generated data using a rolling transaction procedure is described. For instance, a system can process a data stream comprising a first segment and a second segment. A transaction associated with the data stream can be initiated and in response to the transaction being initiated, a first transaction segment for the first segment and a second transaction segment for the second segment are generated. Further, a scaling event that modifies the second segment into a third segment and a fourth segment can be detected, and a data stream transaction procedure is executed to end the transaction.

Processing continuously generated data using a rolling transaction procedure

Reporting time progress for written events on a stream is disclosed. The system comprises operations for obtaining time report information comprising respective timestamp information and respective positional offset information for respective writers with respect to events written to a stream by the respective writers. The operations further comprise aggregating the time report information and determining a lowerbound, e.g., a minimum timestamp before which new events to be written by the stream storage system will no longer arrive, and determining an upperbound, e.g., a stream offset representative of an overall offset of the events written thus far in the stream. The operations further comprise storing the minimum timestamp and the stream offset in a data structure for the events written thus far.

Reporting time progress on events written to a stream storage system

Software for an online social service receives a request for a newsfeed from a user. The newsfeed depends at least in part on a memory-cached list of social friends of the user and on a memory-cached list of events associated with each social friend. The software refreshes a memory-cached copy of the list of social friends from a corresponding list in a persistent data store, based upon a determination of a utility associated with the user. And the software refreshes a memory-cached copy of a list of events associated with a social friend from a corresponding list of events in a persistent data store, based upon a determination of a utility associated with the social friend. Then the software causes the newsfeed to be displayed to the user.

Cache refreshing for online social newsfeeds

Described is a reliable and efficient design for a transactional status oracle ('ReTSO'), which supports large-scale storage systems. ReTSO uses a centralized scheme and implements snapshot isolation, a property that guarantees that read operations read a consistent snapshot of the data store. The centralized scheme of ReTSO enables a lock-free commit algorithm that prevents unreleased locks of a failed transaction from blocking other transactions. Each transaction submits the identifiers of modified rows to the ReTSO, the transaction is committed only if none of the modified rows is committed by a concurrent transaction, and a log of the committed transaction is cached. To recover from failures, before a client is notified of a change, the change cached is first persisted into a write-ahead log, such that the memory state can be fully reconstructed by reading from the write-ahead log. Furthermore, ReTSO implements a lazy, auto-garbage collector hash map that reduces the number of memory accesses per request.

Lock-free transactional support for large-scale storage systems

A search engine for finding objects that correspond to a search request, including an input module for receiving a keyword query from a user, and a search module being configured to map the keyword query to the identifiers of objects that semantically match the keyword or the plurality of keywords contained in the keyword query, and to generate a search result that contains a listing of matching object identifiers, is characterized in that the search module is further configured to generate the search result by considering network layer information about the user within the process of mapping the keyword query to identifiers of matching objects, wherein the network layer information include sophisticated information the search module receives from a dedicated entity.

Search engine and method for performing a search for objects that correspond to a search request

Briefly, embodiments of methods or systems to replicate indexes are described. According to an embodiment, a method may include executing instructions by one or more processors to bring about generating a first replication threshold of documents to be replicated at a local computing site and a second replication threshold of document entries to be stored in a posting list at the local computing site.

Index replication in distributed search engines

Disclosed herein is parallel processing of a query, which uses inter-query parallelism in posting list intersections. A plurality of tasks, e.g., posting list intersection tasks, are identified for processing in parallel by a plurality of processing units, e.g., a plurality of processing cores of a multi-core system.

Posting list intersection parallelism in query processing

In a system for storing and retrieving a plurality of records, the plurality of records associated with a ledger, a client issues read and write requests associated with one of the plurality of records, a plurality of record servers responds to the requests received from the client, and a management server maintains and coordinates, between the client and the record servers, information associated with the ledger, records, and record servers.

System and method for logging operations

Disclosed herein are a system, method and architecture for optimizing a manner in which queries and updates are issued to data stores servicing an application such as a social networking application. A social graph modeling a social network may be used to generate a data store request schedule that optimizes, e.g., minimizes, data store accesses, such as without limitation queries and updates to the data store.

Growing community of inventors

Barcelona, Spain

Flavio Junqueira