Yahoo! Inc.

Trifacta Inc.

Excalibur Ip, LLC

Adam Silberstein

Sean Philip Kandel

Brian Frank Cooper

Edward Eli Marschner

Alon Bartur

Ramana Yerneni

Philip John Vander Broek

Raghunath Ramakrishnan

Wei Zheng

Chris Beavers

Jeffrey Terrace

Aaron J Elmore

Joseph M Hellerstein

Utkarsh Srivastava

Masood Mortazavi

Jianjun Chen

Erik Nathan Vee

Flavio Junqueira

Raghu Ramakrishnan

Sahaya Andrews Albert

Xiao Bai

Ashwin Machanavajjhala

Adwait N Tumbde

Suryanarayan Perinkulam

Russell Sears

A data preprocessing system builds transformation scripts for preprocessing datasets for processing by a data analysis system. The data preprocessing system presents various representations of data of a dataset including visual representations, textual representations, or structural representations. The data preprocessing system receives selections of attributes or values based on these representations. The data preprocessing system generates recommendations of transformations based on the attributes or values selected. The data preprocessing system builds a transformation script based on the recommendations of the transformations. The transformation script can be used for preprocessing the dataset for analysis by a data analysis system.

Visual interactions for transforming datasets with nested data structures

Transforming datasets with nested data structures

A system determines samples of datasets that are typically processed by big data analysis systems. The samples are for use for development and testing of transformations for preprocessing the datasets in preparation for analysis by big data systems. The system receives one or more transform operations input datasets for the transform operations. The system determines samples associated with the transform operations. According to a sampling strategy, the system determines samples that return at least a threshold number of records in the result set obtained by applying a transformation. According to another sampling strategy, the system receives criteria describing the result of the transform operations and determines sample sets that generate result sets satisfying the criteria as a result of applying the transform operations.

Sampling for preprocessing big data based on features of transformation results

Transformation based sampling for preprocessing big data

A system and method identifies rows in a file as uniform rows or outlier rows based on statistics from the file, and displays a sampling of uniform and outlier rows.

System and method for displaying a sample of uniform and outlier rows from a file

Software for an online social service receives a request for a newsfeed from a user. The newsfeed depends at least in part on a memory-cached list of social friends of the user and on a memory-cached list of events associated with each social friend. The software refreshes a memory-cached copy of the list of social friends from a corresponding list in a persistent data store, based upon a determination of a utility associated with the user. And the software refreshes a memory-cached copy of a list of events associated with a social friend from a corresponding list of events in a persistent data store, based upon a determination of a utility associated with the social friend. Then the software causes the newsfeed to be displayed to the user.

Cache refreshing for online social newsfeeds

A system and method automatically identifies any or all of potential row, column and string delimiters in a file in which such delimiters are unknown to the program making such identification.

System and method for identifying delimiters in a computer file

Methods and system for providing social feeds from a plurality of third party sites to a user at a host site includes retrieving one or more access logs capturing online behavior of the user. The access logs are analyzed to determine the user's interactive behavioral pattern related to social feeds from each of the plurality of third party sites. A refresh schedule for the user is computed to refresh cache entries of social feeds at the host site based on the analysis of the user's online behavior at the social feeds. Cache entries of social feeds for the user are refreshed at the host site from the one or more of the plurality of third party sites at an allotted time specified by the refresh schedule.

User behavior-driven background cache refreshing

In a large-scale transaction such as the bulk loading of new records into an ordered, distributed database, a transaction limit such as an insert limit may be chosen, partitions on overfull storage servers may be designated to be moved to underfull storage servers, and the move assignments may be based, at least in part on the degree to which a storage server is underfull and the move and insertion costs of the partitions to be moved.

System and/or method for bulk loading of records into an ordered distributed database

Method, system, and programs for balancing work load in a distributed system. A plurality of multi-dimensional load metrics are received from a plurality of resource units in the distributed system. Based on the received plurality of multi-dimensional load metrics and a global statistical load model, a load deviance for each resource unit is computed. The plurality of resource units in the distributed system are then ranked based on the load deviance of each resource unit. At least one load balancing action is further determined based on the ranked resource units and at least one load balancing policy.

Method and system for work load balancing

In one embodiment, for each one of one or more content producers, access a content-producing rate at which the content producer produces one or more items of content; and for each one of one or more content consumers, access a content-consuming rate at which the content consumer consumes one or more items of content. For each one of a plurality of consumer-producer pairs, wherein the consumer-producer pair comprises one of the content consumers and one of the content producers, and the content consumer follows the content producer, select between a push strategy and a pull strategy for delivering one or more items of content from the content producer to the content consumer based on the content-consuming rate of the content consumer and the content-producing rate of the content producer.

Mechanism for supporting user content feeds

Disclosed are methods and apparatus for error handling within jobs that utilize a plurality of tasks for data transfer of individual data records to a storage destination. For each task, one or more failed records may be logged to a file for later insertion. If a high percentage of a task's output (e.g., writes to another data storage system) is determined to be failing, the task short-circuits itself. Each task is also configured to perform checkpoint logging as the task completes work. If the entire job later short-circuits and is to be restarted, the restarted job only repeats a minimal amount of previously completed work for the tasks which have not already completed their data insertions. Together, these techniques can ensure that in the face of periodic failures, the job completes long-running job in a minimal time with minimal effects.

Efficient failure detection for long running data transfer jobs

An improved system and method for loading records into a partitioned database table is provided. A translation of records may be generated from a set of source partitions to a set of target partitions by generating a bipartite graph, determining a maximal matching using dynamic programming for a chain of nodes remaining in the bipartite graph after removing singleton edges, and generating a maximal matching after adding back the singleton edges for translation of records from the set of source partitions to the set of target partitions. The partition translation may be executed by traversing from top to bottom the set of source partitions and the set of target partitions in record key order to generate an optimal sequence of operations to transfer the records from the set of source partitions to the set of target partitions.

Growing community of inventors

Sunnyvale, CA, United States of America

Adam Silberstein