Microsoft Technology Licensing, LLC

Pivotal Software, Inc.

Ashvin Agrawal

Avrilia Floratou

Jagannathan Ramnarayanan

Daniel Allen Smith

Anthony M Baker

Hemant Bhanawat

Swapnil Prakash Bawaskar

Srikanth Kandula

Ke Wang

Ishai Menache

Sriram S Rao

Joseph Naor

Christopher William Douglas

Virajith Jalaparti

Sumedh Wale

Neeraj Kumar

Daniel E Musgrave

Mainak Ghosh

Subramaniam Venkatraman Krishnan

Fotios Psallidas

MohammadHossein Namaki

Yinghui Wu

A system is described herein for mitigating slow process instances in a streaming application. The system includes a slow process instance candidate identifier configured to identify, based on a relative watermark latency, a set of slow process instance candidates from among a plurality of process instances that comprise the streaming application. The system further includes a set of filters configured to remove false positives from the set of slow process instance candidates. The filters account for window operations performed by the process instances as well as stabilization time needed for downstream process instances to stabilize after a slow upstream process instance is mitigated by a mitigation implementer, which may also be included in the system.

Mitigating slow instances in large-scale streaming pipelines

A system enables tracking machine learning ('ML') model data provenance. In particular, a computing device is configured to accept ML model code that, when executed, instantiates and trains an ML model, to parse the ML model code into a workflow intermediate representation (WIR), to semantically annotate the WIR to provide an annotated WIR, and to identify, based on the annotated WIR and ML API corresponding to the ML model code, data from at least one data source that is relied upon by the ML model code when training the ML model. A WIR may be generated from an abstract syntax tree (AST) based on the ML model code, generating provenance relationships (PRs) based at least in part on relationships between nodes of the AST, wherein a PR comprises one or more input variables, an operation, a caller, and one or more output variables.

Tracking provenance in data science scripts

Methods, systems, apparatuses, and computer program products are provided for prefetching data. A workload analyzer may identify job characteristics for a plurality of previously executed jobs in a workload executing on a cluster of one or more compute resources. For each job, identified job characteristics may include identification of an input dataset and an input bandwidth characteristic for the input dataset. A future workload predictor may identify future jobs expected to execute on the cluster based at least on the identified job characteristics. A cache assignment determiner may determine a cache assignment that identifies a prefetch dataset for at least one of the future jobs. A network bandwidth allocator may determine a network bandwidth assignment for the prefetch dataset. A plan instructor may instruct a compute resource of the cluster to load data to a cache local to the cluster according to the cache assignment and the network bandwidth assignment.

Cache and I/O management for analytics over disaggregated stores

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel processing database system that persists table data in memory to a distributed file system. A distributed parallel processing database system persists table data in memory to a distributed file system. A parameter of creating a database table specifies that data records in the database table and history of changes to the data records can be stored in memory as well as in the distributed file system. When the database table is populated or otherwise modified, data records and the history in memory are evicted to the distributed file system as log files and removed from memory. The log files can be designated as write-only, where the data records, once written, cannot be read by structured query language (SQL) queries, or as read-write, where the data records, once written, can be read by SQL queries.

Table data persistence

Methods, systems, and apparatus for obtaining one or more metadata files, determining, by one or more computers and in accordance with a minor compaction setting, to perform a minor compaction of the one or more metadata files, creating one or more intermediate metadata files that each include at least compacted contacts of one or more of the metadata files, according to the determination to perform minor compaction of the one or more metadata files, determining, in accordance with a major compaction setting, to perform a major compaction of one or more of the intermediate metadata files, and creating one or more snapshot metadata files that each include at least compacted contents of one or more of the intermediate metadata files, according to the determination to perform the major compaction of one or more of the intermediate metadata files.

Compacting data history files

Compacting data file histories

Growing community of inventors

Cupertino, CA, United States of America

Ashvin Agrawal