Huawei Technologies Co., Limited

Weiguo Tan

Fangshan Wang

Chen Li

Gang Shao

A data processing method, a related device and a system are disclosed. In an embodiment, a data processing method includes generating, by a client, a data digest for a stored data set, and sending the data digest to a server, wherein the data digest is used to trigger the server to determine, according to the data digest, feedback information for the client, receiving, by the client, the feedback information sent by the server, when the feedback information comprises a target data selection policy and a target policy parameter, determining, by the client, a data subset from the data set according to the target data selection policy and the target policy parameter, wherein the target policy parameter is used to limit a quantity of pieces of data in the data subset, and sending, by the client, the data subset to the server.

Data processing method and related device, and system

A data mining method and apparatus where the method includes determining multiple execution steps of a data mining process, acquiring a correspondence between a physical resource required by each execution step in a running process and a physical resource occupied by input data of the data mining process, determining a node for executing each execution step, determining, according to a maximum amount of data of input data that can be processed by the node for executing each step, a maximum amount of data of input data that can be processed by the distributed system, and processing to-be-mined data in accordance with the data mining process according to the maximum amount of data of the input data that can be processed by the distributed system. The input data is accurately and effectively limited such that normal running of the system can be ensured.

Data mining method and apparatus

A data processing method and apparatus are disclosed. The method is determining candidate computing frameworks for each sub-task in a sub-task set; predicating operation time and resource consumption that correspond to each candidate computing framework when the candidate computing framework executes the sub-task; and selecting, in the candidate computing frameworks according to the predicated operation time and resource consumption that correspond to each candidate computing framework when the candidate computing framework executes the sub-task, a target computing framework executing the sub-task (), and executing the sub-task (). In this way, a resource management system selects a target computing framework from multiple computing frameworks according to operation time and resource consumption, to execute each sub-task, so as to improve the data processing efficiency and working performance of the system.

Data processing method and apparatus

A method and an apparatus for data processing in data modeling, where the method includes performing, according to a data transformation function corresponding to a preset data processing category identifier, data transformation on a data column corresponding to each characteristic in original data in order to generate a corresponding extended characteristic column, combining extended characteristic columns corresponding to all the characteristics in the original data in order to generate an extended characteristic set, determining a correlation coefficient of each characteristic in the extended characteristic set, selecting a characteristic whose correlation coefficient satisfies a specified condition as an important characteristic, and obtaining, by screening from the extended characteristic set, a data column corresponding to the important characteristic. Therefore, problems such as a long consumed time and a large calculation amount caused because data modeling is performed by exhaustively listing data preprocessing methods are avoided, thereby improving calculation efficiency.

Growing community of inventors

Shenzhen, China

Weiguo Tan