Sportvision, Inc.

Felicia Yue

James R Gloudemans

Walter Hsiao

Image data of an event is provided by updating a textured 3d model of the event. For example, in a sporting event, a model of a stadium can be periodically updated to reflect changes over time in lighting, advertisements, number of spectators in the stands and so forth. Different virtual viewpoints of the event can be depicted in an animation using the textured 3d model and image data from objects at the event such as participants in the sporting event. The same image from which object data is obtained can also be used to update the textured 3d model so that the model is current in the animation, resulting in greater realism. The updating can be based on an operator command or automatic detection of a specified event, such as change in lighting or passage of time. The animation can be provided in a broadcast television signal.

Updating background texture for virtual viewpoint animations

A user such as a television viewer is provided with the capability to view an event from different virtual viewpoints which differ from the viewpoint of a camera. In one aspect, the user is informed of particular camera images from which a virtual viewpoint can be viewed. For example, a menu interface may provide thumbnail preview images of the one or more particular camera images. In another aspect, the use enters commands to manipulate a virtual viewpoint, such as by rotating around a point, moving a lookout point and zooming in or out. In response, a display is provided which includes a textured 3d model of the event combined with at least one textured object in the event. For instance, the event may be a sporting event and the at least one textured object may be a participant in the sporting event.

User interface for accessing virtual viewpoint animations

In one aspect, images of an event are obtained from a first video camera and a second camera, where the second camera captures images at a higher resolution than the first video camera. A particular image of interest is identified from the images obtained by the first video camera, e.g., based on an operator's command. A corresponding image which has been obtained by the second camera is then identified. The second image is used to depict virtual viewpoints which differ from the real viewpoints of the first and second camera, such as by combining data from a textured 3d model of the event with data from the second image. In another aspect, a presentation includes images from a first camera, followed by an animation of different virtual viewpoints, followed by images from a second camera which has a different real viewpoint of the event than the first camera.

Virtual viewpoint animation

A system and method for repairing an object in image data of an event. An image of the event is obtained from a camera, and an object is detected in the image. For example, the event may be a sporting event in which the object is a participant. Moreover, a portion of the object is occluded in a viewpoint of the camera. For instance, a limb of the participant may be occluded by another participant. The object is repaired by providing a substitute for the occluded portion. A user may perform the repair via a user interface by selecting part of an image from an image library and positioning the selected portion relative to the object. A textured 3d model of the event is combined with data from the repaired object, to depict a realistic virtual viewpoint of the event which differs from a viewpoint of the camera.

Image repair interface for providing virtual viewpoints

In one aspect, lines in image data of an event are automatically found and repaired. For example, the event may be a sporting event which is played on a field, and the line segment is a field line on the field which may be obscured by a player, game ball or other object. The line segment is automatically detected in a mask image, and a portion of the line segment which is occluded by the object is automatically determined, and the object is automatically removed. The line segment can also be repaired. Optionally, a virtual viewpoint of the event is provided from the image, with the line repaired and the object removed. In another aspect, an object in an image of an event is automatically located by detecting blobs in the image which meet at least one specified criterion, such as size, aspect ratio, density or color profile.

Line removal and object detection in an image

In one aspect, an image of an event is obtained from a camera, and an object is detected in the image. Data of a textured 3d model of the event and data from the object are combined to depict virtual viewpoints of the event which differ from a real viewpoint of the camera. The object is at least partially faded out at a virtual viewpoint which meets a specified criterion, such as a specified angular offset, or exceeding a specified difference from the viewpoint of the camera. The fading can occur over multiple virtual viewpoints, or over a time period in which the fading is otherwise apparent to a human viewer. In another aspect, an animation is provided in which one object is faded out at a specified virtual viewpoint so that it does not fully occlude another object. For instance, the objects can be participants in a sports event.

Growing community of inventors

Yorktown Heights, NY, United States of America

Felicia Yue