Google Inc.

Sarah Fay Smith

Clement Wright

Golden Gopal Krishna

Carsten Hinz

Zhipeng Pan

Stacy Lou Chiou

Simon Edward Roberts

Rosemary Margaret La Prairie

Shadia Walsh

Jonathan Matthew Malmaud

Runmin Zhao

Nicolas Paul-Stringall Higuera

Jeff Hsu

Gabriel Rubow Culbertson

Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Visual search interface in an operating system

Application prediction based on a visual search determination

A server system obtains a transcription of content from a video and applies a machine learning resource to identify entities which are most likely to be searched for by a user viewing the video, based on the transcription of the content. The server system generates entity cards for each of the entities, each of the entity cards including descriptive content relating to a respective entity among the entities. The server system provides a user interface, to be displayed on a respective display of one or more user computing devices, for playing the video on a first portion of the user interface. When the video is played and a first entity among the entities is mentioned in the video, the user interface displays a first entity card on a second portion of the user interface, the first entity card including descriptive content relating to the first entity.

Entity cards including descriptive content relating to entities from a video

Growing community of inventors

San Francisco, CA, United States of America

Sarah Fay Smith