Google Inc.

Gokhan Hasan Bakir

Behshad Behzadi

Marcin M Nowak-Przygodzki

Ibrahim Badr

Roland Peter Kehl

Nils Grimsmo

Mugurel Ionut Andreica

Aayush Kumar

Michal Jastrzebski

Tak Keung Cheng

Karoly Csalogany

Kavin Karthik Ilangovan

Malte Nuhn

Marcin Bortnik

Oliver Heckmann

Alexander Taboriskiy

Matsvei Zhdanovich

Aurelien Boffy

Emmanuel Mogenet

Kamil Anikiej

Viacheslav Kuznetsov

Matthias Heiler

Adrian Zumbrunnen

Aneto Okonkwo

Thomas Jonathan Stone

Vikram Aggarwal

Aparna Chennapragada

Paige Alexis Dunn-Rankin

Suleyman Yurekli

Cristiano De Carvalho

Kaan Yucer

Methods, systems, and apparatus, for associating data items with objects depicted in an image and search the associated data items based on object detected in images that are subsequently processed and presenting the data items associated with an object when the object is identified in the subsequently processed image.

Association of data items and objects

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for analyzing images for generating query responses. One of the methods includes determining, using a textual query, an image category for images responsive to the textual query, and an output type that identifies a type of requested content; selecting, using data that associates a plurality of images with a corresponding category, a subset of the images that each belong to the image category, each image in the plurality of images belonging to one of the two or more categories; analyzing, using the textual query, data for the images in the subset of the images to determine images responsive to the textual query; determining a response to the textual query using the images responsive to the textual query; and providing, using the output type, the response to the textual query for presentation.

Image query analysis

A computer-implemented method is described. The method includes a computing system receiving a first voice input from a user for generating a bookmark. Responsive to receiving the voice input, the computing system obtains a resource identifier associated with a resource page. The method includes the computing system storing the resource identifier as a bookmark in a bookmark index that is accessible in response to a request from a user device. The method further includes the computing system using the resource identifier, stored as the bookmark, to access the resource page. The computing system uses the resource identifier to access the resource page based on the request from the user device. The request is generated by the user device in response to the user device receiving a second voice input from the user.

Contextual deep bookmarking

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving a query provided by a user and comprising one or more terms. Obtaining context data based on at least a portion of a first resource displayed to the user at a time that the query is received. Obtaining a revised query that is based on the query and the context data. Receiving a plurality of search results responsive to the revised query. Automatically, selecting a search result that represents a second resource from the plurality of search results, and providing the second resource for display to the user.

Assistive browsing using context

Methods, systems, and media for processing queries relating to presented media content are provided. In some implementations, a method comprises: receiving a request to associate with a media playback device that is presenting media content to a user of the mobile device, wherein a mobile application executing on the mobile device and a media application executing on the media playback device exchange media playback information; activating a microphone associated with the mobile device to receive ambient sounds in response to associating with the media playback device; converting the received ambient sounds to one or more text inputs; determining whether the text inputs include a trigger term that corresponds to a request to initiate a query relating to the presented media content and the query; in response to determining that the trigger term has been included in the text inputs, determining the media playback information from the media application that includes timing information corresponding to when during the presentation of the media content the query was received and media content identification information; causing a search to be performed that includes the query, the timing information, and the media content identification information; obtaining a search result that is responsive to the query; and presenting at least a portion of the search result to the query on a mobile display associated with the mobile device.

Methods, systems, and media for processing queries relating to presented media content

Methods, systems, and apparatus for recognizing objects and providing content related to the recognized objects are described. In one aspect, a method includes detecting presence of one or more objects depicted in a viewfinder of a camera of the mobile device. In response to detecting the presence of the one or more objects, image data representing the one or more objects is sent to a content distribution system that selects content related to objects depicted in images. A location of each of the one or more objects in the viewfinder of the camera is tracked while waiting to receive content from the content distribution system. Content related to the one or more objects is received from the content distribution system. A current location of each object in the viewfinder is determined and the content related to the object is presented within the viewfinder at the current location of the object.

Real time object detection and tracking

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, modifying queries based on non-textual content. In one aspect, a method includes receiving, from a user device, a query including a plurality of terms; determining active non-textual data displayed in an application environment on the user device; determining, from the non-textual textual data, modification data for the query; generating a set of modified queries based on the query and the modification parameters; scoring the modified queries according to one or more scoring criteria; selecting one of the modified queries based on the scoring; and providing, to the user device, search results responsive to the selected modified query.

Query modification based on non-textual resource context

Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.

Identifying and controlling smart devices

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.

Contextually disambiguating queries

Image analysis for results of textual image queries

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing contextual information to a user. In one aspect, a method includes receiving, from a user device, a query-independent request for contextual information relevant to an active resource displayed in an application environment on the user device, generating multiple queries from displayed content from the resource, determining a quality score for each of the multiple queries, selecting one or more of the multiple queries based on their respective quality scores, and providing, to the user device for each of the selected one or more queries, a respective user interface element for display with the active resource, wherein each user interface element includes contextual information regarding the respective query and includes the respective query.

Query recommendations for a displayed resource

Growing community of inventors

Zurich, Switzerland

Gokhan Hasan Bakir