Sportvision, Inc.

Sportsvision, Inc.

Walter Hsiao

James R Gloudemans

Richard H Cavallaro

Marvin S White

Stanley K Honey

Kevin R Meier

Felicia Yue

Jerry Neil Gepner

Terence J O'Brien

James Orville McGuffin

John LaCroix

Image data of an event is provided by updating a textured 3d model of the event. For example, in a sporting event, a model of a stadium can be periodically updated to reflect changes over time in lighting, advertisements, number of spectators in the stands and so forth. Different virtual viewpoints of the event can be depicted in an animation using the textured 3d model and image data from objects at the event such as participants in the sporting event. The same image from which object data is obtained can also be used to update the textured 3d model so that the model is current in the animation, resulting in greater realism. The updating can be based on an operator command or automatic detection of a specified event, such as change in lighting or passage of time. The animation can be provided in a broadcast television signal.

Updating background texture for virtual viewpoint animations

A video effect is created that provides an experience to a viewer of freezing time during an event that is the subject of a video presentation, investigating the event during that frozen moment in time, and (optionally) resuming the action of the event. During that frozen moment in time, the video can move around the scene of the event and/or zoom in (or out) to better highlight an aspect of the event. In one embodiment, there will be a transition from video captured by a broadcast camera (or another camera) to a high resolution still image, movement around the high resolution still image, and a transition from the high resolution still image back to video from the broadcast camera (or another camera).

Video effect using movement within an image

In one aspect, images of an event are obtained from a first video camera and a second camera, where the second camera captures images at a higher resolution than the first video camera. A particular image of interest is identified from the images obtained by the first video camera, e.g., based on an operator's command. A corresponding image which has been obtained by the second camera is then identified. The second image is used to depict virtual viewpoints which differ from the real viewpoints of the first and second camera, such as by combining data from a textured 3d model of the event with data from the second image. In another aspect, a presentation includes images from a first camera, followed by an animation of different virtual viewpoints, followed by images from a second camera which has a different real viewpoint of the event than the first camera.

Virtual viewpoint animation

A system and method for repairing an object in image data of an event. An image of the event is obtained from a camera, and an object is detected in the image. For example, the event may be a sporting event in which the object is a participant. Moreover, a portion of the object is occluded in a viewpoint of the camera. For instance, a limb of the participant may be occluded by another participant. The object is repaired by providing a substitute for the occluded portion. A user may perform the repair via a user interface by selecting part of an image from an image library and positioning the selected portion relative to the object. A textured 3d model of the event is combined with data from the repaired object, to depict a realistic virtual viewpoint of the event which differs from a viewpoint of the camera.

Image repair interface for providing virtual viewpoints

In one aspect, lines in image data of an event are automatically found and repaired. For example, the event may be a sporting event which is played on a field, and the line segment is a field line on the field which may be obscured by a player, game ball or other object. The line segment is automatically detected in a mask image, and a portion of the line segment which is occluded by the object is automatically determined, and the object is automatically removed. The line segment can also be repaired. Optionally, a virtual viewpoint of the event is provided from the image, with the line repaired and the object removed. In another aspect, an object in an image of an event is automatically located by detecting blobs in the image which meet at least one specified criterion, such as size, aspect ratio, density or color profile.

Line removal and object detection in an image

3d textured objects are provided for virtual viewpoint animations. In one aspect, an image of an event is obtained from a camera and an object in the image is automatically detected. For example, the event may be a sports event and the object may be a stationary object which is detected based on a known location, color and shape. A 3d model of the object is combined with a textured 3d model of the event to depict a virtual viewpoint which differs from a viewpoint of the camera. The textured 3d model of the event has texture applied from an image of the event, while the 3d model of the object does not have such texture applied, in one approach. In another aspect, an object in the image such as a participant in a sporting event is represented in the virtual viewpoint by a textured 3d kinematics model.

3D textured objects for virtual viewpoint animations

A telestrator system is disclosed that allows a broadcaster to annotate video during or after an event. For example, while televising a sporting event, an announcer (or other user) can use the present invention to draw over the video of the event to highlight one or more actions, features, etc. In one embodiment, when the announcer draws over the video, it appears that the announcer is drawing on the field or location of the event. Such an appearance can be performed by mapping the pixels location from the user's drawing to three dimensional locations at the event. Other embodiments include drawing on the video without obscuring persons and/or other specified objects, and/or smoothing the drawings in real time.

Telestrator system

A system is disclosed that can find an image of a foreground object in a still image or video image. Finding the image of the foreground object can be used to reduce errors and reduce the time needed when creating morphs of an image. One implementation uses the detection of the image of the foreground object to create virtual camera movement, which is the illusion that a camera is moving around a scene that is frozen in time.

Foreground detection

A system is disclosed for blending two image that makes use of a color map which indicates colors in a foreground can be mixed with the background and how much of each source to mix. One embodiment of the invention restricts the use of the color map to only pixels in the foreground that correspond to a graphic (or effect) in the background. Another embodiment makes use of a gray scale matte which stores blending values for each pixel in the foreground.

Video compositor

A three-dimensional model is created to represent an environment to be captured on video. A camera is fitted with pan, tilt and/or zoom sensors. An operator selects a location in the environment. The three-dimensional model is used to determine the three-dimensional coordinates of the location selected by the operator. Information from the pan, tilt and/or zoom sensors is used to transform the three-dimensional coordinates to a two-dimensional position in the video from the camera. Using the two-dimensional position of the video, a graphic is properly added to the video such that the graphic appears to be at the selected location in the environment.

Growing community of inventors

Mountain View, CA, United States of America

Walter Hsiao