International Business Machines Corporation

Anthony N Hylick

Fadi Hikmat Gebara

Rouwaida N Kanj

Anne Elizabeth Gattiker

Wesley Michael Felter

John Bruce Carter

Jian Li

Malcolm Scott Ware

Freeman Leigh Rawson, Iii

Eric Van Hensbergen

Techniques for managing big data include tagging of documents and subsequent retrieval using per-subject dictionaries having entries with some entries specially designated as entities. An entity indicates that the term in the entry has special meaning, e.g., brands (trademarks/service marks), trade names, geographic identifiers or other classes of terms. A dictionary may include a non-entity entry for a term and one or more entity entries, for different entity types. The entries may also include subject-determining-power scores. The subject-determining-power scores provide an indication of the descriptive power of the term with respect to the subject of the dictionary containing the term. The same term may have entries in multiple dictionaries with different subject-determining-power scores in each of the dictionaries. The entity distinctions for a term can then be used in tagging documents and processing retrieval requests.

Document tagging and retrieval using entity specifiers

Techniques for managing big data include tagging of documents and subsequent retrieval using per-subject dictionaries having entries with subject-determining-power scores. The subject-determining-power scores provide an indication of the descriptive power of the term with respect to the subject of the dictionary containing the term. The same term may have entries in multiple dictionaries with different subject-determining-power scores in each of the dictionaries. A retrieval request for one or more documents containing search terms descriptive of the one or more documents can be processed identifying a set of candidate documents tagged with subjects and optional terms, and then applying subject-determining-power scores from the multiple dictionaries for the search term to determine a subject for the search term. The method then selects the one or more documents from the candidate documents according to the subject.

Document tagging and retrieval using per-subject dictionaries including subject-determining-power scores for entries

Techniques for managing big data include retrieval using per-subject dictionaries having multiple levels of sub-classification hierarchy within the subject. Entries may include subject-determining-power (SDP) scores that provide an indication of the descriptive power of the entry term with respect to the subject of the dictionary containing the term. The same term may have entries in multiple dictionaries with different SDP scores in each of the dictionaries. A retrieval request for one or more documents containing search terms descriptive of the one or more documents can be processed by identifying a set of candidate documents tagged with subjects, i.e., identifiers of per-subject dictionaries having entries corresponding to a search term, then using affinity values to adjust the aggregate score for the terms in the dictionaries. Documents are then selected for best match to the subject based on the adjusted scores. Alternatively, the adjustment may be performed after selecting the documents by re-ordering them according to adjusted scores.

Document retrieval using internal dictionary-hierarchies to adjust per-subject match results

A mechanism is provided in a data processing system for optimization of a distributed file system by log data analysis. A name node in the distributed file system initiates a log analysis map/reduce job on one or more connected data nodes via a map/reduce processing framework in the distributed file system and receives result data resulting from the log analysis map/reduce job from the one or more connected data nodes via the map/reduce processing framework in the distributed file system. The name node performs analysis on the received result data and generates an optimization plan for the one or more connected data nodes based on results of the analysis. The name node initiates the optimization plan on the one or more connected data nodes via the map/reduce processing framework in the distributed file system.

File system optimization by log/metadata analysis

A token value is maintained based on an allowable number of low power transitions of a hard disk drive without adversely affecting reliability, compared to an actual number of low power transitions of said hard disk drive. The allowable number of low power transitions increases over the hard disk drive's lifetime. Before the hard disk drive performs a low power transition, the token is evaluated to determine if the hard disk drive is allowed to perform a low power transition. Low power transitions discussed include parking the head and spinning-down the hard disk drive.

Reliability-aware disk power management

A technique for performing storage power management on storage subsystems includes measuring, using a power measurement device, power consumption of a storage subsystem. A first average power and a second average power for the storage subsystem are calculated based on the measured power consumption. In this case, the first average power is calculated over a shorter time period than the second average power. One or more first actuators are incremented in response to the first average power of the storage subsystem being greater than a first power level to reduce the first average power of the storage subsystem below the first power level within a first time period. One or more second actuators are incremented in response to the second average power of the storage subsystem being greater than a second power level and less than the first power level to reduce the second average power of the storage subsystem below the second power level within a second time period that is greater than the first time period. Finally, one or more of the first actuators and the second actuators are decremented in response to the second average power of the storage subsystem being more than an offset below the second power level to increase the power consumption of the storage subsystem.

Growing community of inventors

Austin, TX, United States of America

Anthony N Hylick