Microsoft Technology Licensing, LLC

Rahul Potharaju

Navendu Jain

Wentao Wu

Terry Yumin Kim

Vitaly Voloshin

Luhui Hu

Liyuan Zhang

Mingshi Wang

Joseph K W Chan

Lev Novik

Apoorve Dave

Joseph Kw Chan

Laura M Cruz

Joseph KW Chan

Methods, systems and computer program products are described herein that provide a serverless, multi-engine, multi-user data lake indexing subsystem and application programming interface. Indexes are defined as derived datasets and stored on the data lake in a universal format that enables disparate engines to create and/or discover indexes for workload optimization. Embodiment of indexes enable stateful control and management of an index via metadata included in the index and stored on the data lake.

Serverless data lake indexing subsystem and application programming interface

A hierarchical network analytics system operated by a computing device or system is described. In some example techniques, the analytics system may determine results of a plurality of first level analyses each based at least in part on results of a respective plurality of data queries that return respective subsets of a plurality of types of network data. The analytics system may determine a result of a second level analysis based at least in part on results of the plurality of first level analyses.

Hierarchical network analysis service

Methods, systems, apparatuses, and computer program products are described herein for the generation and traversal of a hierarchical index structure. The structure indexes search keys from data ingested from different data sources and enables efficient retrieval of the keys. As data is ingested, index nodes are generated at the lowest level of the structure. The nodes are analyzed to determine whether such nodes comprise duplicate keys. Responsive to doing so, a new index node is generated located at a higher level of the structure. This process results in a DAG comprising orphan nodes including different search keys. When processing a query for search keys, the orphan index nodes are initially analyzed for the keys. Upon finding a search key, its child nodes are recursively searched until location information specifying the location of ingested data in which the search key is located is found.

Generation and traversal of a hierarchical index structure for efficient data retrieval

Methods, systems, apparatuses, and computer program products are directed to the generation of a global index structure. Agents executing on different data sources locally pre-process (e.g., format, filter, compress, encode, serialize etc.) data generated thereby and index such data. The agents also manage the resources thereof to perform the pre-processing and indexing operations. Each index generated by an agent is formatted as a plurality of index nodes. The index nodes and pre-processed data are provided to backend server(s) that maintain the global index structure and store the data in a globally distributed file system, which aid in unexpected disaster recovery. The backend server(s) generate the global index structure based on the index nodes. As new index nodes are received by the backend servers, the backend servers merge the newly-received index nodes with the global index structure. Global index structure traversal techniques for retrieving search keys are also described herein.

Agent-based data pre-processing and data indexing for efficient data retrieval

Methods, systems and computer program products are described herein that enable data workload optimization through 'what-if' modeling of indexes and index recommendation. In an example aspect, a system is configured to accept a workload comprising a plurality of queries directed at data having a first physical data layout, generate a set of candidate indexes based on the plurality of queries, enumerate index configurations based of the set of candidate indexes, each index configuration comprising a subset on the set of candidate indexes, generate a hierarchical graph of the index configurations, search the hierarchical graph for a recommended index configuration comprising an index configuration with the lowest estimated cost while pruning index configurations not considered from the graph of index configurations to generate a pruned graph, execute a graph query against the pruned graph generating a graph query result and perform an optimization operation based on the graph query result.

Data lake workload optimization through explaining and optimizing index recommendations

Methods, systems and computer program products are described herein that enable data workload optimization through 'what-if' modeling of indexes and index recommendation. In an example aspect, a system is configured to accept a workload in the form of a plurality of data queries, extract indexable columns from the queries, generate candidate indexes from the indexable columns, select and then build the best candidate indexes. Furthermore, the system may receive a query, generate a query plan for the query configured to use one or more of the built candidate indexes, and execute the query plan to generate a query result.

Data lake workload optimization through index modeling and recommendation

Described herein are various technologies pertaining to providing assistance to an operator in a data center with respect to failures in the data center. An alarm is received, and a failing device is identified based upon content of the alarm. Failure conditions of the alarm are mapped to a failure symptom that may be exhibited by the failing device, and troubleshooting options previously employed to mitigate the failure symptom are retrieved from historical data. Labels are respectively assigned to the troubleshooting options, where a label is indicative of a probability that a troubleshooting option to which the label has been assigned will mitigate the failure symptom.

Identifying troubleshooting options for resolving network failures

A dataflow controller technology. A dataflow execution graph (such as a directed acyclic graph) receives or is to receive one or more data streams for processing. The control of the dataflow execution graph is performed by a set of one or more dataflow controllers that are dedicated to that dataflow execution graph. Each dataflow execution graph is instantiated and assigned to the dataflow execution graph such that the assignment results in the dataflow controller executing dataflow control functions for that dataflow execution graph. By dedicating controller(s) to only one dataflow execution graph, the risk of failure of the controller is bound to a single dataflow execution graph.

Dataflow controller technology for dataflow execution graph

At least some embodiments described herein relate to the automatic tuning of a dataflow execution graph. Such dataflow execution graphs are often used to execute some processing against a stream of data messages. A performance parameter of the dataflow execution graph is monitored, and compared against a service level objective. Based on the comparison, it is automatically decided whether a configuration of the dataflow execution graph should be changed. If a change is decided to be made, the configuration of the dataflow execution graph is altered. Thus, rather than require explicit instructions to change the configuration of a dataflow execution graph, the configuration of a dataflow execution graph is changed (or tuned) depending on compliance of performance with a service level objective.

Automatic tuning of a dataflow execution graph

Disclosed herein is a system and method for generating an enhanced index for documents contained in a knowledge base. Documents or configuration snapshots are tokenized and then passed through a number of filters. The filters modify the token stream to generate an enhanced token stream for the document by removing words from the stream and adding domain level knowledge to the tokens in the stream. The token stream is then added to the index for the document so that searches can be completed against the knowledge base when searches are generated from configuration snapshots or from inputted search queries.

Building features and indexing for knowledge-based matching

When a computing device has an issue, a detector receives (or retrieves) data associated with the computing device. The data may include parameter key-value pairs. The detector creates queries based on the data and distributes the queries to one or more matching engines, such as an exact matching engine or a proximity matching engine. The one or more matching engines look for matches in an index of database documents. The results from the one or more matching engines are ranked based on relevancy scores. In some cases, users may provide feedback regarding the relevancy of the results and the feedback may be used to recalibrate how the relevancy scores are determined.

Growing community of inventors

West Lafayette, IN, United States of America

Rahul Potharaju