Databricks Inc.

Andreas Neumann

Michael Paul Armbrust

Mukul Murthy

Jonathan Mio

Tathagata Das

Li Zhang

Kiavash Kianfar

The present application discloses a method, system, and computer system for automatically scaling task-processing capacity. The method includes obtaining, at a data layer, a current measure of queued tasks and/or a task-processing capacity, obtaining, by one or more processors, a cost-prioritized criterion or a latency-prioritized criterion, determining a set of tasks to process using the task-processing capacity based at least in part on a set of an input data to process, and automatically scaling the task-processing capacity based at least in part on the current measure of queued tasks and/or the task-processing capacity and either (i) the cost-prioritized criterion or (ii) the latency-prioritized criterion.

Priority for autoscaling of streaming workloads

A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.

Dataflow graph processing

A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.

Growing community of inventors

Oakland, CA, United States of America

Andreas Neumann