Google Inc.

Apple Inc.

Thomas Deselaers

Victor Carbune

Daniel M Keysers

Sandro Feuz

Pedro Gonnet Anders

Henry Allan Rowley

Shumin Zhai

Franz Josef Och

Abraham Murray

Li-Lun Wang

Matthew Nirvan Sharifi

Ryan S Dixon

Nima Ferdosi

Ashok Chhabedia Popat

Mayank Garg

Dmitriy Genzel

Peder Blekken

Marcos Calvo

Stephan Robert Gammeter

Tianchang Gu

Francisco Alvaro Munoz

Daniel Martin Keysers

Techniques are disclosed that enable automating user interface input by generating a sequence of actions to perform a task utilizing a multi-agent reinforcement learning framework. Various implementations process an intent associated with received user interface input using a holistic reinforcement policy network to select a software reinforcement learning policy network. The sequence of actions can be generated by processing the intent, as well as a sequence of software client state data, using the selected software reinforcement learning policy network. The sequence of actions are utilized to control the software client corresponding to the selected software reinforcement learning policy network.

Reinforcement learning techniques for selecting a software policy network and autonomously controlling a corresponding software client based on selected policy network

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing actionable suggestions are disclosed. In one aspect, a method includes receiving (i) an indication that an event detection module has determined that a shared event of a particular type is presently occurring or has occurred, and (ii) data referencing an attribute associated with the shared event. The method includes selecting, from among multiple output templates that are each associated with a different type of shared event, a particular output template associated with the particular type of shared event detected by the module. The method generates a notification for output using at least (i) the selected particular output template, and (ii) the data referencing the attribute associated with the shared event. The method then provides, for output to a user device, the notification that is generated.

Actionable suggestions for activities

Systems and methods enable a computing system to recognize a sequence of repeated actions and offer to automatically repeat any such recognized actions. An example method includes determining a current sequence of user actions is similar to previous sequence of user actions, determining whether the previous sequence is reproducible and, when reproducible, initiating display of a prompt that requests approval for completing the current sequence based on the previous sequence and, responsive to receiving an indication of approval, completing the previous sequence. Another example method included determining that a first current sequence of user interactions is complete and is not similar to any saved sequence of user interactions, saving the first current sequence as a previous sequence, identifying a second sequence as satisfying a similarity threshold with the previous sequence, and initiating display of a prompt that requests approval for saving the previous sequence as a shortcut.

Smart assist for repeated actions

A method is provided that can include determining, based on a plurality of media streams of a plurality of client devices participating in a video conference, a subject being discussed during the video conference at a point in time, and providing one or more inputs to a machine learning model, the one or more inputs comprising a plurality of transcriptions each corresponding to one of the plurality of media streams, and an indication of a request to identify one or more media streams that are relevant the subject. The method can further include determining, based on one or more outputs of the machine learning model, a media stream that is most relevant to the subject, and providing the determined media stream for presentation on one or more of the plurality of client devices while the subject is being discussed during the video conference.

Video conference content auto-retrieval and focus based on learned relevance

Implementations set forth herein relate to generating a pre-call analysis for one or more users that are receiving and/or initializing a call with one or more other users, and/or prioritizing pre-call content according to whether security-related value was gleaned from provisioning certain pre-call content. One or more machine learning models can be employed for determining the pre-call content to be cached and/or presented prior to a user accepting a call from another user. Feedback provided before, during, and/or after the call can be used as a basis from which to prioritize certain content and/or sources of content when generating pre-call content for a subsequent call. Other information, such as contextual data (e.g., calendar entries, available peripheral devices, location, etc.) corresponding to the previous call and/or the subsequent call, can also be used as a basis from which to provide a pre-call analysis.

Generating and/or prioritizing pre-call content for rendering when awaiting acceptance of an incoming call

In some examples, an electronic device including a touch screen can sense a stylus proximate to the touch screen. For example, the user can draw or write with the stylus to create a simulated drawing or simulated handwriting displayed on the touch screen. In some examples, the electronic device can sense a series of 'points' corresponding to the location of the stylus over time while the drawing/writing input was provided. In some situations, noise in the stylus data can cause the electronic device not to sense one or more noisy points included in the drawing or handwriting input. In some examples, the electronic device can interpolate the noisy points based on sensed points before and/or after the time of the skipped points.

System and method for stylus detection

Implementations relate to an automated assistant that is capable of interacting with non-assistant applications that do not have functionality explicitly provided for interfacing with certain automated assistants. Application data, such as annotation data and/or GUI data, associated with a non-assistant application, can be processed to map such data into an embedding space. An assistant input command can then be processed and mapped to the same embedding space, and a distance from the assistant input command embedding and the non-assistant application data embedding can be determined. When the distance between the assistant input command embedding and the non-assistant application data embedding satisfies threshold(s), the automated assistant can generate instruction(s), for the non-assistant application, that correspond to the non-assistant application data. For instance, the instruction(s) can simulate user input(s) that cause the non-assistant application to perform one or more operations characterized by, or otherwise associated with, the non-assistant application data.

Automated assistant control of external applications lacking automated assistant application programming interface functionality

To select a lane in a multi-lane road segment for a vehicle travelling on the road segment, a system identifies, in multiple lanes and in a region ahead of the vehicle, another vehicle defining a target; the system applies an optical flow technique to track the target during a period of time, to generate an estimate of how fast traffic moves; and the system applies the estimate to machine learning (ML) model to generate a recommendation which one of the plurality of lanes the vehicle is to choose.

Lane selection using machine learning

Systems and methods for video conference content auto-retrieval and focus based on learned relevance is provided. A method may include determining collaborative documents each associated with at least one user participating in a video conference, and providing one or more first inputs to a machine learning model, the one or more first inputs identifying the plurality of collaborative documents, and comprising an indication of a request to identify a collaborative document that is most relevant to a first subject discussed during the video conference at a first point in time. The method may further include determining, based on one or more first outputs of the machine learning model, the collaborative document that is the most relevant to the first subject discussed during the video conference at the first point in time, and providing the determined collaborative document for presentation on one or more client devices while the first subject is being discussed during the video conference.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that use generative adversarial models to increase the quality of sensor data generated by a first environmental sensor to resemble the quality of sensor data generated by another sensor having a higher quality than the first environmental sensor. A set of first and second training data generated by a first environmental sensor having a first quality and a second sensor having a target quality, respectively, is received. A generative adversarial mode is trained, using the set of first training data and the set of second training data, to modify sensor data from the first environmental sensor by reducing a difference in quality between the sensor data generated by the first environmental sensor and sensor data generated by the target environmental sensor.

Modifying sensor data using generative adversarial models

A navigation service determines that a first user intends to navigate to a shared destination from a first location, at a first time, and that a second user intends to navigate to the shared destination from a second location, at a second time within a threshold interval of the first time. The navigation service notifies the first user using an electronic notification that the second user intends to navigate to the shared destination, receives from the first user an electronic request to coordinate navigation to the shared destination with the second user, and in response to receiving the electronic request, provides navigation directions to the shared destination to a device associated with the first user in view of a progress of the second user toward the shared destination.

Providing navigation instructions to one device in view of another device

Implementations described herein determine, for a given document generated by a given source, one or more portions of content (e.g., phrase(s), image(s), paragraph(s), etc.) of the given document that may be influenced by a source perspective of the given source. Further, implementations determine one or more additional resources that are related to the given source and that are related to the portion(s) of content of the given document. Yet further, implementations utilize the additional resource(s) to determine additional content that provides context for the portion(s) that may be influenced by a source perspective. A relationship, between the additional resource(s) and the portions of the given document, can be defined. Based on the relationship being defined, the additional content can be caused to be rendered at a client device in response to the client device accessing the given document.

Generating and provisioning of additional content for source perspective(s) of a document

This document describes techniques and systems that enable privacy controls for sharing embeddings for searching and indexing media content. A set of images of a user's face are obtained and a machine-learned model is applied to the set of images to generate a user-specific dataset of face embeddings for the user. Media content stored in a media storage is indexed by applying the machine-learned model to the media content to provide indexed media information identifying one or more faces shown in the media content. Access to the indexed media information by another user querying the media content for images or videos depicting the user is controlled based on a digital key shared by the user with the other user, where the digital key is associated with the user-specific dataset and the user-specific dataset is usable to identify the images or videos depicting the user.

Privacy controls for sharing embeddings for searching and indexing media content

Systems and methods for video conference content auto-retrieval and focus based on learned relevance is provided. In accordance with the systems and methods, audio streams and video streams from client devices participating in a video conference are received. Based on the audio streams, a subject being discussed during the video conference at a point in time is determined. A video stream that is most relevant to the subject being discussed during the video conference at the point in time is determined from the video streams. The determined video stream is provided to the client devices for presentation on the client devices while the subject is being discussed during the video conference.

Methods, apparatus, and computer readable media related to receiving textual input of a user during a dialog between the user and an automated assistant (and optionally one or more additional users), and generating responsive reply content based on the textual input and based on user state information. The reply content is provided for inclusion in the dialog. In some implementations, the reply content is provided as a reply, by the automated assistant, to the user's textual input and may optionally be automatically incorporated in the dialog between the user and the automated assistant. In some implementations, the reply content is suggested by the automated assistant for inclusion in the dialog and is only included in the dialog in response to further user interface input.

Growing community of inventors

Zurich, Switzerland

Thomas Deselaers