International Business Machines Corporation

Masaki Komedani

Hiroaki Kikuchi

Takuma Murakami

Fumihiko Terui

Shunsuke Ishikawa

Ken Kumagai

Akihiro Nakayama

Tohru Hasegawa

Takashi Fukuda

Keisuke Nitta

Yutaka Moriya

Yasumasa Kajinaga

Daiki Tsuzuku

Shimpei Yotsukura

Hirofumi Nishikawa

Osamu Furusawa

A system and method iteratively update a determined structure of a digital document file to remediate disconnected text in blocks of the determined structure. In embodiments, a method includes determining a structure of a digital document file using a document understanding analysis, the structure including blocks of elements having text information; determining for each of the blocks of the digital document file whether text information in the block is disconnected; determining an order of the blocks in the digital document file; pairing two blocks from a list of blocks with disconnected text information to form a block pair, wherein the two blocks are ordered based on the determined order of the blocks; determining that the text information of the block pair forms a complete sentence using natural language processing; and consolidating the block pair to form a new block.

Iteratively updating a document structure to resolve disconnected text in element blocks

Facet-based search processing is provided which includes receiving a query search context for querying documents of a document set, and retrieving, by similar document search processing, a document subset from the document set. The document subset includes documents of the set most similar to a search document of the query search context. Facet analysis processing is used to generate M candidate facets most-related to the query search context, and facets of the M candidate facets associated with documents of the subset are identified, and classified into a positive facet set and a negative facet set based, at least in part, on extent of facet commonality across the documents. A listing is of the documents in the document subset is provided, with the listing highlighting facets of the positive facet set.

Positive/negative facet identification in similar documents to search context

An approach is provided in which an information handling system matches a set of documents, from a corpus of documents, to a query received from a client. The information handling system determines an amount of facets corresponding to the matching set of documents by dynamically dividing the corpus of documents into multiple sub-corpuses and performing complement facet counting on at least one of the sub-corpuses. In turn, the information handling system sends a set of results to the client that includes the determined amount of facets corresponding to the matching set of documents.

Large scale facet counting on sliced counting lists

A method for evaluating annotation quality is provided. The method may include obtaining annotation information associated with a plurality of annotators and a plurality of data elements including a plurality of annotation entries corresponding to at least one data element and entered based on an annotation guideline, determining a quality rating for the annotation guideline based on a comparison between a first value associated with the plurality of annotators and the plurality of data elements and a second value associated with any disparity among the plurality of annotation entries, determining a proficiency rating for an annotator from the plurality of annotators based on a comparison between a third value associated with annotation entries by the annotator and the second value, and generating a report based on the quality rating and the proficiency rating.

Evaluating quality of annotation

A mechanism is provided in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement a document analysis device for performing a statistical analysis of documents with respect to a facet. An acceptance module accepts a natural language sentence. An extraction module extracts a first facet from the natural language sentence. A statistical analysis module performs a first statistical analysis of a set of documents with respect to the first facet and determines a value of the first facet based on a result of the first statistical analysis responsive to information being extracted from the natural language sentence, the information requesting for a second statistical analysis. The statistical analysis module performs the second statistical analysis of the set of documents using the value of the first facet. A user interface presents a second facet determined based on a result of the second statistical analysis.

Supporting interactive text mining process with natural language and dialog

An analysis device for analyzing a text document is provided. The analysis device includes a context storage unit configured to store context information that shows a position of a character set of a predetermined context in the text document. The analysis device also includes an index storage unit configured to store index information that shows a position of a word in the text document, for each word of a plurality of words contained in the text document. An input unit is configured to input a target word. A position detection unit is configured to detect from the index information a position of the target word contained in the text document. A frequency detection unit is configured to detect an appearance frequency of the target word per each type of context in the text document based on the position of the target word and on the context information.

Context-aware text document analysis

An aspect includes phrase searching using exclusion tokens. A token division unit is configured to divide an input character string to be searched into a plurality of tokens. A token position definition unit is configured to set each token to be excluded in an occurrence position calculation as an exclusion token and to set each token to be included in the occurrence position calculation as a headword token, and define an occurrence position for each headword token. A position offset information assigning unit is configured to assign, to each of the exclusion tokens, position information obtained with the headword token followed by the exclusion tokens and to assign the headword token followed by the exclusion tokens as a starting point. An indexing processing unit is configured to perform indexing on the plurality tokens such that whether or not the exclusion tokens follow one of the plurality of tokens is identifiable.

Searching with exclusion tokens

Retrieval is completed in a short time for presenting a retrieval result of a document file, which satisfies a retrieval condition, to a user having the authority to perform predetermined processing. A retrieval device includes a correspondence information generating unit for generating correspondence information indicating membership of a user to a group, an index information generating unit for generating index information by extracting user identification information associated with group identification information included in authority information provided to a document file from the correspondence information, the index information including information in which the extracted user identification information is associated with the document file, and a retrieving unit for retrieving a document file, which satisfies a retrieval condition whose input is received by an input receiving unit and in which the user identification information included in the index information coincides with user information whose input is received.

Retrieval device, retrieval system, retrieval method, and computer program for retrieving a document file stored in a storage device

Methods, apparatus and systems, including computer program products for creating an index for search in a search system by collecting and analyzing documents to be searched. A first storage section stores system configuration information of the search system. A second storage section stores attribute information of a collected document and the system configuration information at the time of document collection for each document. A comparison section compares the attribute information of the documents to be collected and the system configuration information stored in the first storage section with the attribute information and the system configuration information stored in the second storage section, respectively, if forced re-collection of the documents is performed. A document collection section collects the documents according to a predetermined schedule under normal conditions and re-collects only documents for which a mismatch is detected by the comparison section in the case of forced re-collection.

Growing community of inventors

Kanagawa, Japan

Masaki Komedani