Microsoft Technology Licensing, LLC

Anirudh Koul

Serge-Eric Tremblay

Sambodhi Chakrabarty

Baihan Huang

Meher Kasam

Elizabeth Halper

Swetha Machanavajhala

Michael Johnston

Ranjitha Gurunath Kulkarni

Venkata Naga Vijaya Swetha Machanavajhala

Stéphane Morichère-Matte

Sitthinon Jinaphant

Shery Sharonjit Sumal

Vinay Vyas Vemuri

David Dai Wei Zhang

Anass Al-Wohoush

Jan Ervin Mickel Takata Clarin

Sheng-Ting Lin

This disclosure describes how to identify objects in an augmented reality environment. More specifically, the various systems and methods described herein describe how an augmented reality device can recognize objects within a real world environment, determine where the object is located, and also identify the various surfaces of the object in real time or substantially real time.

Real time object surface identification for augmented reality environments

Aspects of the present disclosure relate to systems and methods that aid users with hearing and/or speech impediments to have a conversation with a remote phone number without human assistance. In one aspect, an application residing on a user's device, such as a smartphone, tablet computer, laptop, etc., may be used to initiate a phone call to a recipient. Upon initiating the phone call locally, a service residing on the server may receive a request to initiate a connection to the recipient. Once the recipient answers, the user may converse with the recipient by providing text input to their local app. The text input may be transmitted to the service. The service may use a text to speech converter to translate the received text to speech that can be delivered to the recipient.

Automated real time interpreter service

Architecture that enables the identification of entities such as people and content in live broadcasts (e.g., streaming content (e.g., video) of live events) and non-live presentations (e.g., movies), in realtime, using recognition processes. This can be accomplished by extracting live data related to a live event. With respect to people entities, filtering can be performed to identify the named (people) entities from the extracted live data, and trending topics discovered as relate to the named entities, as associated with the live event. Multiple images of the named entities that capture the named entities under different conditions are captured for the named entities. The images are then processed to extract and learn facial features (train one or more models), and facial recognition is then performed on faces in the video using the trained model(s).

Automatic recognition of entities in media-captured events

Users' interaction performance with an automated voice system is improved, as is users' efficiency, by visually displaying options audibly presented by the automated voice system, thereby enabling users to interact with the system more quickly and accurately. Options can be obtained from a centralized audible menu database with the communicational identifier utilized to establish a communication connection with the automated voice system. The database is populated from crowdsourced information, provided when users establish communicational connections with portions of automated voice systems whose options have not yet been stored in the database, and then transcribe the options that are audibly presented by the automated voice system. Transcription of audibly presented options likewise serves as a double check to verify options already displayed. User interaction generates a subsequent communicational connection, with a different communicational identifier, to a different portion of the automated voice system, re-triggering the mechanisms.

Automatic visual display of audibly presented options to increase user efficiency and interaction performance

Non-limiting examples of the present disclosure describe proactive speech detection on behalf of a user and alerting the user when a specific word, name, etc. is detected. Speech detection is actively executed through a computing device, where the speech detection analyzes spoken utterances in association with a dynamic grammar file stored locally on the computing device. An alert is generated that indicates when a spoken word of the utterances matches a word stored in the dynamic grammar file. The alert may be displayed, for example, through the computing device. The alert provides indication that the spoken word is identified in the spoken utterances. In further examples, a buffered window of the spoken utterances is captured which is associated with a detection of the specific word. A live transcription of the content in the buffered window is generated and provided to a computing device of the user.

Proactive speech detection and alerting

One or more systems and/or techniques are provided for automatic closed captioning for media content. In an example, real-time content, occurring within a threshold timespan of a broadcast of media content (e.g., social network posts occurring during and an hour before a live broadcast of an interview), may be accessed. A list of named entities, occurring within the social network data, may be generated (e.g., Interviewer Jon, Interviewee Kathy, Husband Dave, Son Jack, etc.). A ranked list of named entities may be created based upon trending named entities within the list of named entities (e.g., a named entity may be ranked higher based upon a more frequent occurrence within the social network posts). A dynamic grammar (e.g., library, etc.) may be built based upon the ranked list of named entities. Speech recognition may be performed upon the broadcast of media content utilizing the dynamic grammar to create closed caption text.

Automated closed captioning using temporal data

Growing community of inventors

Sunnyvale, CA, United States of America

Anirudh Koul