Other

Samuel Chenillo

Oren Steinfeld

Oran Gilad

Roy Jonathan Rosser

Systems and methods for producing high-quality image segmentation masks by efficiently augmenting low-quality segmentation masks initially obtained by lightly trained neural networks are disclosed. Sequences of low-quality segmentation masks are transformed into high-quality segmentation masks by first identifying groups of pixels that are consistently segmented in at least three successive video frames. The pixels surrounding the consistently segmented pixels are then examined and assigned gradient alpha values, i.e., degrees of transparency. In one embodiment the alpha values are made proportional to the RGB distance of the pixel from the mean RGB value of the consistently segmented group of pixels.

Enhanced AI image segmentation

A system and method of organically generating a camera-pose map is disclosed. A target image is obtained of a location deemed suitable for augmenting with a virtual augmentation or graphic. An initial camera-pose map is created having a limited number of calibrated camera-pose images having calculated camera-pose locations and homographies to the target image. Then, during the event, the system automatically obtains current images of the event venue and determines homographies to the nearest calibration camera-pose image in the camera-pose map. The separation in camera-pose space between the current images and the camera-pose images are calculated. If this separation is less than a predetermined threshold, that current image is fully calibrated and added to the camera-pose map, thereby growing the map organically.

Organic camera-pose mapping

A system and method of obtaining an occlusion key using a background pixel map is disclosed. A target image containing a target location suitable for displaying a virtual augmentation is obtained. A stream of current images are transformed into a stationary stream having the camera pose of the target image. These are segmented using a trained neural network. The background pixel map is then the color values of background pixels found at each position within the target location. An occlusion key for a new current image is obtained by first transforming it to conform to the target image and then comparing each pixel in the target location with the color values of background pixels in the background pixel map. The occlusion key is then transformed back to conform to the current image and used for virtual augmentation of the current image.

Occlusion key generation

Systems and methods for precision downstream synchronization of digital streaming content on an edge content processor in the absence of access to pixel-level data by a video player app operative on the edge content processor are disclosed. Encrypted video streams are synchronized to unencrypted video streams using acquired knowledge of the edge content processor's latency, i.e., the time elapsed between a command to render a video frame and that frame being displayed by the edge content processor. Once a predicted time of display of an encrypted video is obtained by a video player app operative on the edge content processor, a corresponding RGBA video frame is delayed by an amount of time equal to that predicted time minus the edge content processor's latency before a command to render it is issued by the video player app, thereby ensuring both frames are displayed simultaneously.

Systems and methods for precision downstream synchronization of content

A system and method for identifying media segments using audio augmented image cross-comparison is disclosed, in which a media segment identifying system analyses both audio and video content, producing a unique identifier to compare with previously identified media segments in a media segment database. The characteristic landmark-linked-image-comparisons are constructed by first identifying an audio landmark. The audio landmark is an audio peak that exceeds a predetermined threshold. Two digital images are then obtained, one associated directly with the audio landmark, and one obtained a predetermined landmark time removed from the first image. The two images are then used to provide a characteristic landmark-linked-image-comparison. The pair of images are reduced in pixel size and converted to gray scale. Corresponding pixels are compared to form a numeric comparison. One image is mirrored before comparison to reduce the possibility of null comparisons.

System and method for media segment identification

A system and method for remunerating a display of a cognate substitute image sequence is disclosed. Images sequences in received video content are examined for sequences that are cognate to known image sequences. When detected, such sequences are replaced with cognate substitute image sequences to create a modified video image stream that is displayed on a display screen. A reasonable remuneration for displaying the cognate substitute image is automatically calculated. This monetary charge reflects the nature of the image sequence, the time it was displayed for, the time at which was displayed and the number of electronic devices within viewing range of the display at the time of display. This remuneration is accumulated, and equivalent digital assets automatically transferred between client and provider accounts at an appropriate time.

System and method for remunerating a display of a cognate substitute video

A system and method of synchronizing auxiliary content to a video stream is disclosed that uses a block of bit-vectors linked to a target frame in a video steam. The block of bit-vectors consists of a multi-frame sequence of image bit-vectors. The video stream and block of bit-vectors are transmitted to an end user device that uses the bit-vector block to uniquely identify the target frame. The target frame is used to synchronize auxiliary content to the video stream.

System and method for synchronizing auxiliary content to a video stream

A system and method of identifying cognate image sequences is disclosed that examines significant frames of a stream of video images using an array of image indexes. The image index array includes image indexes obtained by at least two different image indexing methods. These are compared to a corresponding array of image indices of significant frames of known image sequences. An image quality indicator is used to determine which set of image index thresholds to use in making the comparison. These thresholds are more stringent for higher quality frames. Two image sequences are considered cognate when a string of sufficiently many sequential frame matches is established. In an alternate embodiment, image blurriness is also, or instead, used to determine the appropriate set of image index thresholds. The sets of image index thresholds are determined using machine learning on a curated set of representative images.

System and method for identifying cognate image sequences

A system and method for identifying media segments using audio augmented image cross-comparison is disclosed, in which a media segment identifying system analyses both audio and video content, producing a unique identifier to compare with previously identified media segments in a media segment database. The characteristic landmark-linked-image-comparisons are constructed by first identifying pairs of audio landmarks separated by a characteristic, or landmark, time. Digital images associated the audio landmarks are then compared, with the combination providing a characteristic landmark-linked-image-comparison. The audio landmarks are audio peaks that exceed predetermined thresholds. A landmark-time is the time between adjacent pairs of audio peaks. The pair of images associated with the audio peaks are reduced in pixel size and converted to gray scale. Corresponding pixels are compared to form a numeric comparison. One image may be mirrored before comparison to reduce the possibility of null comparisons.

A system and method for identifying media segments using audio augmented image cross-comparisoning is disclosed, in which a media segment identifying system analyzes both audio and video content, producing a unique identifier to compare with previously identified media segments in a media segment database. The characteristic landmark-linked-image-comparisones are constructed by first identifying pairs of audio landmarks separated by a characteristic, or landmark, time. Digital images associated the audio landmarks are then image comparisoned, with the combination providing a characteristic landmark-linked-image-comparison. The audio landmarks are audio peaks that exceed predetermined thresholds. A landmark-time is the time between adjacent pairs of audio peaks. The pair of images associated with the audio peaks are reduced in pixel size and converted to gray scale. Corresponding pixels are compared to form a numeric comparison. One image may be mirrored before comparison to reduce the possibility of null comparisones.

System and method for media segment indentification

A system for the augmented assessment of digital media streams for virtual less than insertion opportunities is disclosed. A digital media stream is automatically decomposed, using a programmed digital processor, into one or more candidate-clips have a predetermined minimum length, and no internal shot transitions. These candidate-clips are then examined, and the ones deemed suitable for virtual insertion use, are classified as viable-clips and stored in a digital database. The artificial intelligence techniques of machine learning and deep learning are then used to further classify the viable-clips according to their virtual insertion related attributes that may be attractive to advertisers, such as scene context, emotional tone and contained characters. A value is then assigned to the viable-clips, dependent on their insertion related attributes, and the overlap of those attributes with client requested requirements.

System for the augmented assessment of virtual insertion opportunities

A system and method of occlusionless virtual insertion is disclosed that uses an animation of the image. The animation has a full image frame, a null frame, and a number of intervening video frames. When the animation is run, the image appears to be physically moved behind an edge of the animation video frames. An event video-stream is duplicated, and the duplicate delayed to provide a delayed event video-stream. The animation, stopped at the full image frame, is mixed into the delayed event video-stream steam at a selected location using video match moving algorithms. The resultant video-stream is broadcast. The un-delayed, event video-stream is monitored in the region corresponding to where the virtual insertion is occurring in the delayed event video-stream. When this corresponding region begins to become occluded by a foreground object, the animation is triggered. As the time delay of the delayed event video-stream equals the time it takes the animation to run from the full image frame to the null frame, when the foreground object moves into the insertion region in the delayed event video-stream there is no inserted image to occlude it. To the viewer, it appears as if the image is on a movable sign, and just happened to be translated out of view before the object of interest moved into the region.

Occlusionless method for virtual image insertion

A method of automated content selection is disclosed, in which an end user views multimedia content provided via by a content aggregator that is also programmed to detect content delivery boundaries. This occurs, for instance, when the provider switches from showing an event to showing interstitial advertising. On detecting a delivery boundary, a substitute stream of multimedia content is then automatically sent. Detecting a content delivery boundary is accomplished in a twostep process. First, a candidate frame indicative of a deliver boundary is found. This is done, for instance, by finding a change in average sound volume of sufficient magnitude. The candidate frame is then compared to a database of representative frames of known interstitial video segments. If a sufficiently good match is found, the frame is determined to be a content boundary frame, and appropriate switching of the video being relayed is made.

Method of video content selection and display

A system and method for automatically acquiring a matrix of digital reference images containing landmarks relevant to recognizing and tracking objects in a digital video image is disclosed. In order to virtually insert a digital graphic having a correct pose in a stream of images, an indexed matrix of images of the landmarks in a venue is accumulated. The system and method allow for automated acquisition of new landmarks within a venue, and for the automated expansion of the reference matrix to accommodate reference images indicative of changed conditions within a venue and for reference images of new landmarks. When a landmark is recognized in an image, the graphic reference matrix is queried to obtain a preferred reference image to be used in making a virtual insertion.

Graphic reference matrix for virtual insertions

A system and method of synchronizing the processing and distribution of audio-visual assets in a distributed production studio are presented. The method addresses the processing of broadcast television, including match moving computer graphics technology, in such an environment. Audio-visual sources provide the assets while processing nodes interact with them to produce a composite output. Synchronization of the communication between, and the processing by, the sources and nodes is managed by a master controller and delay buffers located on the nodes. Assets are tagged with updatable reference-to-zero values indicative of a current transmission delay between their last source and the controller. These reference-to-zero values are used by processing nodes to determine where in their local buffer to place each asset so they emerge from the buffer to a local digital signal processor in synchronicity and that assets originally produced at a common time are executed locally at a common time.

Growing community of inventors

New York, NY, United States of America

Samuel Chenillo