Baidu USA LLC

Baidu.com Times Technology (Beijing) Co. Ltd.

Beijing Baidu Netcom Science Technology Co., Ltd

Hao Tian

Daming Lu

Jeff Chienyu Wang

Xi Chelsea Chen

Xing Li

Le Kang

Xin Zhou

Zhiyu Cheng

Xi Chen

Darning Lu

Yichen Hu

Bo He

Jingyu Xin

Dapeng Li

Jingya Xun

Presented herein are systems, methods, and datasets for automatically and precisely generating highlight or summary videos of content. In one or more embodiments, the inputs comprise a text (e.g., an article) of the key event(s) (e.g., a goal, a player action, etc.) in an activity (e.g., a game, a concert, etc.) and a video or videos of the activity. In one or more embodiments, the output is a short video of an event or events in the text, in which the video may include commentary about the highlighted events and/or other audio (e.g., music), which may also be automatically synthesized.

Generating highlight video from video and text inputs

A method for aligning a text with a media material, an apparatus, and a storage medium are provided. The method includes: determining a set of anchor points in the text; performing following operations i) to v) repeatedly until all anchor points are removed from the set of anchor points or all media materials are removed from a set of media materials: i) ranking the anchor points in the set of anchor points, ii) selecting a target anchor point from the set of anchor points based on the ranked anchor points in the set, iii) determining, from the set of media materials, a media material matching a text segment starting from the target anchor point, iv) removing the target anchor point, and v) removing the media material matching the text segment starting from the target anchor point; and aligning the text segments with respective media matching materials.

Method for aligning text with media material, apparatus and storage medium

With rapidly evolving technologies and emerging tools, sports-related videos generated online are rapidly increasing. To automate the sports video editing/highlight generation process, a key task is to precisely recognize and locate events-of-interest in videos. Embodiments herein comprise a two-stage paradigm to detect categories of events and when these events happen in videos. In one or more embodiments, multiple action recognition models extract high-level semantic features, and a transformer-based temporal detection module locates target events. These novel approaches achieved state-of-the-art performance in both action spotting and replay grounding. While presented in the context of sports, it shall be noted that the systems and methods herein may be used for videos comprising other content and events.

Transformer-based temporal detection in video

The present disclosure provides examples of a method and apparatus for processing a video, a device and a storage medium. The method may include: acquiring a target video and a target comment of the target video; recognizing a picture in the target video to obtain text information of the picture; determining a target comment matching a content of the text information; and inserting, in response to displaying the picture in the target video, the target comment matching the content in a form of a bullet screen.

Method for processing video, device and storage medium

Presented herein are systems, methods, and datasets for automatically and precisely generating highlight or summary videos of content. For example, in one or more embodiments, videos of sporting events may be digested or condensed into highlights, which will dramatically benefit sports media, broadcasters, video creators or commentators, or other short video creators, in terms of cost reduction, fast, and mass production, and saving tedious engineering hours. Embodiment of the framework may also be used or adapted for use to better promote sports teams, players, and/or games, and produce stories to glorify the spirit of sports or its players. While presented in the context of sports, it shall be noted that the methodologies may be used for videos comprising other content and events.

Automatically and precisely generating highlight videos with artificial intelligence

Embodiments of the present disclosure provide a method and apparatus for aligning a paragraph and a video. The method may include: acquiring a commentary and a candidate material resource set corresponding to the commentary, a candidate material resource being a video or an image; acquiring a matching degree between each paragraph in the commentary and each candidate material resource in the candidate material resource set; and determining a candidate material resource sequence corresponding to the each paragraph in the commentary based on the matching degrees between the paragraphs in the commentary and the candidate material resources, playing durations of the candidate material resources and text lengths of the paragraphs in the commentary, an image playing duration being a preset image playing duration.

Method and apparatus for aligning paragraph and video

Embodiments of the present disclosure provide a method and apparatus for generating a commentary. The method may include: acquiring at least one news cluster composed of pieces of news generated within a first preset time length, the pieces of news in the news cluster direct to a given news event; determining a target news cluster based on the at least one news cluster; determining, for each piece of news in the target news cluster, a score of being suitable for generating a commentary for the piece of news; and generating, based on a piece of target news, a commentary for the target news cluster, where the piece of target news is a piece of news having a highest score of being suitable for generating a commentary in the target news cluster.

Method and apparatus for generating commentary

The present disclosure discloses a method and apparatus for determining video material of news. The method for determining video material of news comprises: recognizing a person name in a news text; searching a video based on the person name, to obtain a to-be-selected video; extracting a key frame in the to-be-selected video; recognizing a person in the key frame to obtain identity information of the person; and determining the to-be-selected video as video material of news, in response to the identity information of the person conforming to the person name. The present disclosure improves the consistency between the video material of the news and the news text.

Method and apparatus for determining (raw) video materials for news

The present disclosure discloses a method and apparatus for determining video material of news. The method for determining video material of news comprises: acquiring a weighted score value of a score of a keyword of a news text in a plurality of dimensions; filtering a keyword set of news based on the weighted score value of the score of the keyword; searching a pre-selected video using the keyword set of the news; and determining video material of the news based on the pre-selected video. The present disclosure improves the consistency between the video material of the news and the news text.

Embodiments of the present disclosure provide a method and apparatus for generating a video. The method may include: receiving a query text inputted by a user; querying a material resource set related to the query text, material resources being images, videos, or audios; presenting the material resource set; determining a material resource sequence, in response to receiving a selecting operation and a ranking operation of the user on the material resources in the presented material resource set; and generating the video based on the material resource sequence.

Method and apparatus for generating video

Embodiments of the present disclosure provide a method and apparatus for generating a video. The method may include: determining a commentary of a target news cluster, each piece of news in the target news cluster being specific to a given news event; generating a voice corresponding to each paragraph in the commentary using a speech synthesis technology; determining a candidate material resource set corresponding to the commentary based on a video and an image included in the target news cluster, the candidate material resource being a video or image; determining a candidate material resource sequence corresponding to the each paragraph in the commentary; and generating a video corresponding to the commentary based on the voice corresponding to the each paragraph in the commentary and the candidate material resource sequence.

Embodiments of the present disclosure provide a method and apparatus for generating information. The method may include: determining at least one video segment obtained by semantically segmenting videos included in a target news cluster as a target video set, where respective pieces of news in the target news cluster directs to a given news event; determining a commentary for the target news cluster; determining, based on the target video set and a target image set, a candidate material resource set corresponding to the commentary, where the target image set is composed of respective images included in the target news cluster; and for each paragraph in the commentary, determining degrees of matching between the paragraph and candidate material resources in the candidate material resource set.

Method and apparatus for generating information

A method and apparatus for adjusting a parameter are provided. The method may include: acquiring a current value of at least one parameter which is in a process of generating a video corresponding to a commentary of the news cluster based on a news cluster; determining a video evaluation score of the video which is generated based on the news cluster and according to the current value of the at least one parameter; performing feature extraction on the current value of the at least one parameter to obtain a feature representation; inputting the feature representation and the determined video evaluation score into a pre-trained evaluation network to obtain a predicted video evaluation score; inputting the feature representation and the predicted video evaluation score into a pre-trained operation network, to obtain current operation information; and adjusting the current value of the at least one parameter based on the current operation information.

Growing community of inventors

Sunnyvale, CA, United States of America

Hao Tian