Beijing Bytedance Network Technology Co., Ltd.

Beijing Youzhuju Network Technology Co., Ltd.

Xinsong Zhang

Hang Li

Yan Zeng

Pengshuai Li

Disclosed are a language representation model system, a language representation model pre-training method, a natural language processing method, an electronic device, and a storage medium. The language representation model system includes: a word granularity language representation sub-model based on segmentation in units of words, and a phrase granularity language representation sub-model based on segmentation in units of words. The word granularity language representation sub-model is configured to output, based on a sentence segmented in units of words, a first semantic vector corresponding to a semantic expressed by each segmented word in the sentence. The phrase granularity language representation sub-model is configured to output, based on the sentence segmented in units of phrases, a second semantic vector corresponding to a semantic expressed by each segmented phrase in the sentence.

Language representation model system, pre-training method and apparatus, device, and medium

Provided in the present application are a method and apparatus for training a visual language pre-training model, and a device and a medium. The method includes: acquiring pairing groups respectively corresponding to N images, wherein the pairing group of a first image includes: a first pairing group which is composed of the first image and description text of the first image, and a second pairing group which is composed of a local image of the first image and description text of the local image, N is an integer greater than 1, and the first image is any one of the N images; and training a visual language pre-training model according to the pairing groups respectively corresponding to the N images.

Growing community of inventors

Beijing, China

Xinsong Zhang