Samsung Electronics Co., Ltd.

Michael Dinerstein

Doron Sabo

Roee Sfaradi

Evgeny Artyomov

Ishay Goldin

Roy Josef Jevnisek

Tomer Peleg

Ran Vitek

Sapir Kontente

Shachar Praisler

Yuval Shicht

Ohad Givaty

Eyal Rosin

Yarden Sharabi

Niv Zehngut

Amir Ben-dror

Pablo Szekely

Stas Dubinchik

Yonatan Dinai

A method for processing images acquired by a multi-spectral RGB-NIR (red/green/blue/near infra-red) sensor includes receiving a RGB-NIR digital image from a multi-spectral RGB-NIR sensor, interpolating an NIR contribution to each R, G and B pixel value, wherein an NIR image is obtained, subtracting the NIR contribution from each R, G and B pixel value in the RGB-NIR digital image wherein a decontaminated RGB-NIR image is obtained, constructing a red, green and blue (RGB) Bayer image from the decontaminated RGB-NIR image, and processing the Bayer image wherein a full color image is obtained. The RGB-NIR digital image includes red (R) pixels, green (G) pixels, blue (B) pixels, and NIR pixels, and every other row in the RGB-NIR digital image includes NIR pixels that alternate with green pixels, and every other row in the RGB-NIR digital image includes green pixels that alternate with red and blue pixels.

RGB-NIR processing and calibration

One or more aspects of the present disclosure enable high accuracy computer vision and image processing techniques with decreased system resource requirements (e.g., with decreased computational load, shallower neural network designs, etc.). As described in more detail herein, one or more aspects of the described techniques may leverage key layers (e.g., certain key layers of a neural network) and compressed tensor comparisons to efficiently exploit temporal redundancy in videos and other slow changing signals (e.g., to efficiently reduce neural network inference computational burden, with only minor increase in data transfer power consumption). For example, key layers of a neural network may be identified, and temporal/spatial redundancy across frames may be efficiently leveraged such that only a computation region in a subsequent frame n+1 is re-computed in layers between identified key layers, while remaining feature-map calculations may be disabled in the layers between the identified key layers.

Low-power change-based neural network inference for image processing

Methods, systems, and apparatus for combined or separate implementation of coarse-to-fine neural architecture search (NAS), two-phase block NAS, variable hardware prediction, and differential hardware design are provided and described. A variable predictor is trained, as described herein. Then, a controller or policy may be used to iteratively modify a neural network architecture along dimensions formed by neural network architecture parameters. The modification is applied to blocks (e.g., subnetworks) within the neural network architecture. In each iteration, the remainder of the neural network architecture parameters are modified and learned with a differential NAS method. The training process is performed with two-phase block NAS and incorporates a variable hardware predictor to predict power, performance, and area (PPA) parameters. The hardware parameters may be learned as well using the variable hardware predictor.

Method for co-design of hardware and neural network architectures using coarse-to-fine search, two-phased block distillation and neural hardware predictor

A method of warping data includes the steps of providing a set of target coordinates&#x2208;, calculating, by a warping engine, source coordinates'&#x2208;for the target coordinates&#x2208;, requesting, by the warping engine, data values for a plurality of source coordinates from a cache, and computing, by the warping engine, interpolated data values for eachin a neighborhood of&#x2032; from the data values of the source coordinates returned from the cache. Requesting data values from the cache includes notifying the cache that data values for a particular group of source points will be needed for computing interpolated data values for a particular target point, and fetching the data values for the particular group of source points when they are need for computing interpolated data values for the particular target point.

Warping data

A system of convolutional neural networks (CNNs) that synthesize middle non-existing frames from pairs of input frames includes a coarse CNN that receives a pair of images acquired at consecutive points of time, a registration module, a refinement CNN, an adder, and a motion-compensated frame interpolation (MC-FI) module. The coarse CNN outputs from the pair of images a previous feature map, a next feature map, a coarse interpolated motion vector field (IMVF) and an occlusion map, the registration module uses the coarse IMVF to warp the previous and next feature maps to be aligned with pixel locations of the IMVF frame, and outputs registered previous and next feature maps, the refinement CNN uses the registered previous and next feature maps to correct the coarse IMVF, and the adder sums the coarse IMVF with the correction and outputs a final IMVF.

Efficient CNN-based solution for video frame interpolation

At least one example embodiment provides an apparatus including a processor configured to execute computer-readable instructions to receive image data from a plurality of pixels, determine a first white point based on the image data and a threshold percentage of a histogram of the image data, determine a second white point based on the image data, determine a third white point based on groups of the image data corresponding to a same hue or desaturation, the processor configured to divide the image data into the groups and determine an image based on at least the first white point, the second white point and the third white point.

Methods and systems for white balance

Growing community of inventors

Rehovot, Israel

Michael Dinerstein