Sony Interactive Entertainment Inc.

Andrew Young

Jeffrey Roger Stafford

Chris Ho

Javier Fernandez Rico

Michael G Taylor

Glenn T Black

Todd Tokubo

Dennis Dale Castleman

Megumi Kikuchi

Brandon Sangston

Patrick Connor

Hadi Alamdar

Methods and systems for normalizing views of real world space and virtual environment of an interactive application as viewed through a pair of augmented reality glasses include receiving content of the interactive application for rendering on a display screen of the pair of augmented reality glasses. The content is streaming content provided in response to inputs provided by the user and includes virtual objects. The virtual objects are dynamically scaled to match a scale of the real world objects to a scale of the virtual objects. The content of the interactive application dynamically scaled is projected as an overlay over the real world objects of the real world space viewed through the pair of augmented reality glasses. The content that is dynamically scaled enables a normalized view between the real world space and the virtual environment.

Projector assisted augmented reality, adjusting AR size, shape and presentation based on real world space

A hand-gesture tracking engine includes a first artificial intelligence (AI) model component configured and trained to automatically identify a hand-gesture made by a source player within a video game. A hand-gesture intent engine includes a second AI model component configured and trained to automatically determine a message for communication to a target player within the video game as conveyed by the hand-gesture identified by the hand-gesture tracking engine. A hand-gesture translation engine having a third AI model component configured and trained to automatically generate a communication to the target player that conveys the message as determined by the hand-gesture intent engine. The third AI model component is configured and trained to automatically control sending of the communication to the target player.

Systems and methods for artificial intelligence (AI)-assisted temporal context processing in support of hand-gesture communication in video gaming

Methods and systems for determining a type of material of an object in a real-world environment are described. One of the methods includes receiving a plurality of sets of audio data based on sounds received from a plurality of objects within a plurality of environments. The method further includes receiving a plurality of sets of input data regarding a plurality of types of materials of the plurality of objects, training an artificial intelligence (AI) model based on the plurality of sets of audio data and the plurality of sets of input data, and applying the AI model to a set of audio data captured from the real-world environment to determine the type of material of the object within the real-world environment to enhance the realism of augmented reality (AR) or virtual reality (VR) video games and applications.

Systems and methods for determining a type of material of an object in a real-world environment

A method for determining an environment in which a user is located is described. The method includes receiving a plurality of sets of audio data based on sounds emitted in a plurality of environments. Each of the plurality of environments has a different combination of objects. The method further includes receiving input data regarding the plurality of environments, and training an artificial intelligence (AI) model based on the plurality of sets of audio data and the input data. The method includes applying the AI model to audio data captured from an environment surrounding the first user to determine a type of the environment.

Systems and methods for training a model to determine a type of environment surrounding a user

A method for updating information for a graphics pipeline including executing in the first frame period an application on a CPU to generate primitives of a scene for a first video frame. Gaze tracking information is received in a second frame period for an eye of a user. In the second frame period a landing point on an HMD display is predicted at the CPU based at least on the gaze tracking information. A late update of the predicted landing point to a buffer accessible by the GPU is performed in the second frame period. Shader operations are performed in the GPU in the second frame period to generate pixel data based on the primitives and based on the predicted landing point, wherein the pixel data is stored into a frame buffer. The pixel data is scanned out in the third frame period from the frame buffer to the HMD.

Generating and rendering primitives in a frame period with late update of eye tracking information for fast foveated rendering

Gaze tracking data representing a user's gaze is analyzed to determine one or more regions of interest. One or more gaze tracking parameters are determined from the gaze tracking data. Adjusted foveation data is determined representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters. The compression of the one or more transmitted images is adjusted so that fewer bits are needed to transmit data for portions of an image outside the one or more regions of interest than for portions of the image within the one or more regions of interest. Adjusting compression includes eliminating the region(s) of interest from images that are presented to the user during the saccade or blink.

Use of eye tracking to adjust region-of-interest (ROI) for compressing images for transmission

By using the user head and/or eye tracking capabilities of an extended reality (XR) system, a region of gaze (ROG) of the user towards the virtual screen can be determined and signaled to non-XR games and applications running on the host system connected to the XR system. The existing non-XR games/applications can be patched, upgraded, or remastered to support the application of the ROG and ROG based enhancements of the visuals and user interfaces, using the provided by the system software on the host system that is rendering the game for the XR system.

Foveated enhancement of non-XR games within a HMD system

A method for predicting eye movement in a head mounted display (HMD). The method including tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points. The method including determining velocity of the movement based on the movement of the eye. The method including determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity. The method including predicting a landing point on the display of the HMD corresponding to a direction of the eye for the saccade.

Building saccade models for predicting a landing point of a saccade experienced by a player

Gaze tracking data representing a user's gaze is analyzed to determine one or more regions of interest. One or more gaze tracking error parameters are determined from the gaze tracking data including a rate of rotation of a user's eye with respect to one or more axes. An error in a fixation of the user's eye on a particular region of interest is determined. Adjusted foveation data is determined representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters. The compression of the one or more transmitted images is adjusted so that fewer bits are needed to transmit data for portions of an image outside the one or more regions of interest than for portions of the image within the one or more regions of interest.

Late update of eye tracking information to GPU for fast foveated rendering

Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

Temporal supersampling for foveated rendering systems

Dynamic graphics rendering based on predicted saccade landing point

Gaze tracking data may representing a user's gaze with respect to one or more images transmitted to a user are analyzed to determine one or more regions of interest. The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

Eye tracking to adjust region-of-interest (ROI) for compressing images for transmission

Gaze tracking data representing a user's gaze is analyzed to determine one or more regions of interest. One or more gaze tracking error parameters are determined from the gaze tracking data including a rate of rotation of a user's eye with respect to one or more axes. An error in a fixation of the user's eye on a particular region of interest is determined. Adjusted foveation data is determined representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters. The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

Growing community of inventors

San Mateo, CA, United States of America

Andrew Young