Microsoft Technology Licensing, LLC

Emad Barsoum

Roderick Michael Toll

Charles Claudius Marais

John Raymond Justice

Krassimir Emilov Karamfilov

David James Quinn

Shahram Izadi

Simon Stachniak

Michael Anthony Hall

Craig C Peeper

Tommer Leyvand

Cem Keskin

Vladimir Tankovich

Yichen Wei

Christoph Rhemann

Robert Mitchell Smith

Sean Ryan Francesco Fanello

Spencer Dean Perreault

Yao Shen

Ben Butler

Simon P Stachniak

Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way.

Client side processing of game controller input

An initial candidate foreground region is identified within an infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range. A depth of surfaces within the initial candidate foreground region is estimated based on infrared intensity values the pixels of the initial candidate foreground region. The initial candidate foreground region is expanded to an expanded candidate foreground region based on a body-model estimate. The body model estimate is seeded with one or more of the initial candidate foreground region, the depth of surfaces, and/or a face of a human subject identified by facial recognition. Each pixel of the infrared image is identified as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region. Pixels identified as background pixels may be modified within a corresponding visible light image.

Distinguishing foreground and background with inframed imaging

Techniques for automatically selecting from amongst multiple camera views available in a multi-camera system. In an aspect, one or more metrics are calculated for multiple camera views. The metrics may be used to identify one or more of the camera views as fulfilling a condition, in response to which a particular one of the camera views may be selected for display. For example, in a 'select-better-view' strategy, the camera view associated with a more optimal image (e.g., better lighting, or absence of occlusion) is selected for display. In a 'flag-diagnostic-view' strategy, the camera view associated with a less optimal image is displayed, e.g., to indicate that a particular camera may require user attention. Specific techniques for calculating metrics are further disclosed.

Multi-camera view selection based on calculated metric

Embodiments of the present invention split game processing and rendering between a client and a game server. A rendered video game image is received from a game server and combined with a rendered image generated by the game client to form a single video game image that is presented to a user. Game play may be controlled using a rich sensory input, such as three-dimensional image data and audio data. The three-dimensional image data describes the shape, size and orientation of objects present in a play space. The rich sensory input is communicated to a game server, potentially with some preprocessing, and is also consumed locally on the client, at least in part. In one embodiment, latency sensitive features are the only features processed on the client and rendered on the client.

Client rendering of latency sensitive game features

A signal encoding an infrared (IR) image including a plurality of IR pixels is received from an IR camera. Each IR pixel specifies one or more IR parameters of that IR pixel. IR-skin pixels that image a human hand are identified in the IR image. For each IR-skin pixel, a depth of a human hand portion imaged by that IR-skin pixel is estimated based on the IR parameters of that IR-skin pixel. A skeletal hand model including a plurality of hand joints is derived. Each hand joint is defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

Two-dimensional infrared depth sensing

A scene is illuminated with modulated illumination light that reflects from surfaces in the scene as modulated reflection light. Each of a plurality of pixels of a depth camera receive the modulated reflection light and observe a phase difference between the modulated illumination light and the modulated reflection light. For each of the plurality of pixels, an edginess of that pixel is recognized, and the phase difference of that pixel is smoothed as a function of the edginess of that pixel.

Edge preserving depth filtering

Growing community of inventors

Sammamish, WA, United States of America

Emad Barsoum