International Business Machines Corporation

Jsonar Inc.

Imperva, Inc.

Guardium, Inc.

Ron Ben-Natan

Ury Segal

Leonid Rodniansky

Luigi Labigalini

Uri Hershenhorn

Ariel Farkash

Abigail Goldsteen

Boris Rozenberg

Tamar Domany

Derek Difilippo

Roman Krashanitsa

Igor Gokhman

Ksenya Kveler

Gabriel Beyo

Igor Gorkhman

Yuval Hager

Izar Tarandach

Rosa Miroshnikov

Joey Andres

Dan Nguyen

Ishai Kones

Embodiments of the present disclosure relate to utilizing an existing login process of a data repository to enable the data repository to delegate MFA functionality to an external MFA system. When a purported user attempts to log in to the data repository, a delegation module within the login process may insert a record into a table associated with the login process. A program executing on a security device external to the data repository may periodically poll the table for new records and upon detecting the new record, may call the external MFA system to verify the login attempt. The external MFA system may indicate to the program whether the login attempt was verified and the program may update the table with the indication. Upon detecting the indication, the delegation module may complete or terminate the login attempt based on the indication.

Delegating multi-factor authentication in legacy databases

Embodiments of the present disclosure provide a data gathering and query method for collecting ongoing updates to large, unstructured or semi-structured databases. The method comprises gathering a plurality of events defined in a database syntax that is not structured and aggregating the plurality of events into one or more part files. Each of the one or more part files store a subset of the plurality of events in a columnar format, and each of the one or more part files comprises a header file that includes metadata corresponding to a subset of the plurality of events stored in the part file and is separate from the subset of events stored in the part file. The method further comprises uploading the one or more part files to a cloud storage repository configured to store the one or more part files so that they can be queried by a query server based on the header files.

Direct cloud storage intake and upload architecture

Embodiments of the present disclosure relate to generating a high level security policy for a data repository without knowledge of the access control, entitlement, and other models of the data repository. A set of abstractions that define a security policy language may be generated based on data in a data repository collection. The set of abstractions may define a security policy language, which may be provided to a security administrator who can define a security policy with the security policy language. The security policy may be translated into a common physical language to generate a common physical policy. The processing device may then translate the common physical policy into a set of commands for each of one or more data repositories that the data repository collection is comprised of.

System and method for policy control in databases

A log message classifier employs machine learning for identifying a corresponding parser for interpreting the incoming log message and for retraining a classification logic model processing the incoming log messages. Voluminous log messages generate a large amount of data, typically in a text form. Data fields are parseable from the message by a parser that knows a format of the message. The classification logic is trained by a set of messages having a known format for defining groups of messages recognizable by a corresponding parser. The classification logic is defined by a random forest that outputs a corresponding group and confidence value for each incoming message. Groups may be split to define new groups based on a recurring matching tail (latter portion) of the incoming messages. A trend of decreased confidence scores triggers a periodic retraining of the random forest, and may also generate an alert to operators.

High volume message classification and distribution

Classification for data intake operations in an enterprise ensures that sensitive data is not disseminated inappropriately, but incurs substantial time, effort and expense. A method of classifying data in a large set of data repositories captures a set of raw rules resulting from inputs indicative of evaluations and conclusions of data classification operations, typically by logging data classification operations, and identifies patterns in the set of raw rules by consolidating duplicative conditions and eliminating inconsequential conditions. External conditions and observations may be referenced for applying a context to the rules based on a usage or domain of the data, and data sets of disparate entities may be examined for anonymizing the data and combining with other sets of anonymized data.

Data item classification and organization in large data sets

A data gathering and query method for collecting ongoing updates to large, unstructured databases performing data collection from multiple sites. A large repository of unstructured or semi-structured data according to a JSON syntax receives periodic updates from an enterprise site for gathered event data. A bigdata cloud store receives the additions as columnar parts that arrange the data in a columnar form for storing similarly named fields consecutively. The enterprise site generating the event data arranges the block files containing the columnar data, and header files containing metadata. Incremental time and/or size triggers the periodic part upload, and a query server in network communication with the cloud store integrates the incoming additions by receiving the header files and updating a catalog of collections in the cloud store, without downloading the larger block files containing the actual columnar data. Query requests operate on the cloud store without moving the block files.

Data storage for unstructured data such as JSON data stored as collections of documents transforms the JSON data into a columnar form of storing unstructured data by grouping similar fields together for facilitating retrieval of the individual fields from a range of documents. Groups of fields are stored in individual files for each field. Compound data such as arrays and subdocuments are also broken down into files for each atomic field. In other words, a compound document structure that defines a hierarchy or 'tree' of fields is flattened such that each 'leaf' of the tree is stored in a separate file.

Columnar storage and processing of unstructured data

Mechanisms are provided for propagating source identification information from an application front-end system in an application layer to a data layer inspection system associated with a back-end system. An incoming user request is received, at the data layer inspection system, from a gateway system associated with the application front-end system. One or more outgoing statements targeting a back-end system are received at the data layer inspection system. The data layer inspection system accesses a mapping data structure based on the one or more outgoing statements to thereby correlate the one or more outgoing statements with the incoming user request. The data layer inspection system retrieves source identification information associated with the incoming user request based on the correlation of the one or more outgoing statements with the incoming user request. The data layer inspection system performs a data layer inspection operation based on the source identification information.

Identity propagation through application layers using contextual mapping and planted values

A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage-full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest.

Query usage based organization for very large databases

A query server performs method of generating a query result using an aggregation pipeline by identifying, based on a query, a sequence of operations to be applied to documents from an unstructured database, in which a portion of the operations are dependent on other operations in the sequence of operations. The pipeline determines, from the operations, lightweight and heavyweight operations, in which the heavyweight operations generate a materialized result have a substantial impact on processing resources. The pipeline defers the lightweight operations until a materialized result is needed, for performing with a corresponding heavyweight operation, in which the materialized result includes either creation of a new document or movement of substantial data from a document. Lightweight operations are grouped with heavyweight operations such that multiple operations can be collapsed into a single operation that act upon the data together thus avoiding the number of materializations.

Query processing pipeline for semi-structured and unstructured data

A system and method of storing data in an unstructured or semi-structured database, such as a JSON database, includes defining a columnar store, or hybrid column portion having a set of values for each field, such that each set stores a sequence of values from commonly named fields of each document together, and defining a hybrid store, or hybrid row portion having, for each document in the columnar store, a hybrid entry storing fields from the same document together, such that each entry in the hybrid store includes fields of a corresponding document represented in the columnar store. The hybrid arrangement provides that both the columnar store and the hybrid store are derived from the same set of documents, and each of the columnar store and the hybrid store are configured to fulfill a query request for determining whether to access the columnar store or the hybrid store.

Hybrid storage and processing of very large databases

A query engine for an unstructured database satisfies window based queries and analytics by defining a window of documents, and performing analytics on the window using a default value for omitted field. A tabular index containing only values needed for analytics and document ordering defines each window. The tabular index includes all fields from each document that are required to satisfy the query, retrieved on a single pass by the query engine so that multiple fetches to the same document are avoided. Since each document in the window need not contain all the same fields as the other documents, an adapter includes logic for defining a default or placeholder value for a field called for in an analytic computation but nonexistent in a particular document. By retrieving only the computationally relevant fields, and by performing the retrieval only once on each document, the I/O overhead is greatly reduced.

Window queries for large unstructured data sets

An analytics processing system generates analytics from a collection of unstructured data by identifying trends in the data and deriving associations or correlations between series of values. Each series is generated from a set of field labeled values in the set, and compared to other series in the collection. Identified relationships in the series are scored based on depiction of an illustrative, predictive, or non-random association, and ranked by a scoring metric for analytical value. A visualization of the relationships are ranked and rendered such that the visualization highlights the association in a manner not achievable by simple inspection of the field values. Relationships are graphed by lines, circles, bars (histogram) on labeled axes based on the series. In this manner, a user may generate analytic results from a large data set, and pinpoint significant associations by paging through renderings scored as the most illustrative of notable trends.

Growing community of inventors

Lexington, MA, United States of America

Ron Ben-Natan