International Business Machines Corporation

Technion Research & Development Foundation Ltd

Joseph Shtok

Leonid Karlinsky

Asaf Tzadok

Sivan Harary

Mattias Marder

Yochay Tzur

Eliyahu Schwartz

Oded Dubovsky

Amit Alfassy

Udi Barzelay

Ophir Azulai

Daniel Nechemia Rotman

Adi Raz Goldfarb

Michael Elad

Amit Aides

Michael Zibulevsky

Assaf Arbelle

Yevgeny Yaroker

Sivan Doveh

A computing system, computer program product, and computer-implemented method for incremental learning without forgetting for a classification/detection model are provided. The method includes receiving, at a computing system, a classification/detection model including a base embedding space and corresponding base embedding vectors that are based on a base training dataset including base classes. The method also includes expanding the classification/detection model to account for a new training dataset including new classes by lifting the base embedding space to add an orthogonal subspace for the new classes, producing an expanded embedding space and corresponding expanded embedding vectors that are of a higher dimension than the base embedding vectors. In some embodiments, the method also includes further expanding the expanded classification/detection model to account for another new training dataset including additional new classes by lifting the expanded embedding space to add another orthogonal subspace for the additional new classes.

Incremental learning without forgetting for classification and detection models

An example system includes a processor to receive detected chart regions in a page of a document. The processor is to produce, via a graphical elements detector, predicted heatmaps and bounding boxes for graphical objects in the detected chart regions. The processor is also to apply chart type specific analysis algorithm to the predicted heatmaps and bounding boxes, to extract tabular chart data. The processor can then generate an output data file and a visualization based on the predicted heatmap and the extracted tabular chart data.

Detecting graphical elements in charts using predicted heatmaps

A system and method for generating hard training data from easy training data. Training data including visual data with synthetic semantic implants ('VSSI') having at least one cue is received. An annotator identifies at least one cue in the VSSI and annotates the VSSI to indicate the cue to create a modified training data set. A data scrambler removes at least one cue from the VSSI to create the tagged training data, which can then be used to train a classifier to identify transitions between segments when the cues are not present.

Automatic creation of difficult annotated data leveraging cues

An example system includes a processor to receive, a randomly generated alpha-map, a pair of training images, and a pair of training texts associated with the pair of training images. The processor is to generate a blended image based on the randomly generated alpha-map and the pair of training images. The processor is to train a visual language grounding model to separate the blended image into a pair of heatmaps identifying portions of the blended image corresponding to each of the training images using a separation loss.

Training visual language grounding models using separation loss

Embodiments may include novel techniques for Task-Adaptive Feature Sub-Space Learning (TAFSSL). For example, in an embodiment, a method may be implemented in a computer system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising: training a machine learning system to classify features in images by: generating a sample set comprising one or a few labeled training samples and one or a few additional samples containing instances of target classes, performing dimensionality reduction computed on the samples in the sample set to form a dimension reduced sub-space, generating class representatives in the dimension reduced sub-space using clustering, and classifying features in images using the trained machine learning system.

TAFSSL: task adaptive feature sub-space learning for few-shot learning

A method, a computer program product, and a computer system for transferring knowledge from an expert to a user using a mixed reality rendering. The method includes determining a user perspective of a user viewing an object on which a procedure is to be performed. The method includes determining an anchoring of the user perspective to an expert perspective, the expert perspective associated with an expert providing a demonstration of the procedure. The method includes generating a virtual rendering of the expert at the user perspective based on the anchoring at a scene viewed by the user, the virtual rendering corresponding to the demonstration of the procedure as performed by the expert. The method includes generating a mixed reality environment in which the virtual rendering of the expert is shown in the scene viewed by the user.

Expert knowledge transfer using egocentric video

There is provided a method of computing a model for classification of a data element, comprising: feeding a plurality of labeled auxiliary data elements and at least one labeled relevant data elements for each of a plurality of relevant classification categories, into a synthesizer component, for outputting at least one synthetic labeled relevant data element for each one of the plurality of relevant classification categories, feeding the synthetic labeled relevant data elements and a plurality of unlabelled training relevant data elements into a domain adaptation component, for outputting a respective relevant classification category for each of the plurality of unlabelled training relevant data elements, iteratively end-to-end training the synthesis component and the domain adaptation component, and providing the trained domain adaption component, for outputting a relevant classification category in response to an input of a query unlabelled data element.

Systems and methods for training a model using a few-shot classification process

Embodiments of the present systems and methods may provide techniques to discover features such as object categories that provide improved accuracy and performance. For example, in an embodiment, a method may comprise extracting, at the computer system, features from a dataset comprising a plurality of data samples using a backbone neural network to form a features vector for each data sample, training, at the computer system, using the features vectors for at least some of the plurality of data samples, an unsupervised generative probabilistic model to perform clustering of extracted features of the at least some of the plurality of data samples by minimizing a negative Log-Likelihood function, wherein clusters of extracted features form categories, and categorizing, at the computer system, at least some different data samples of the plurality of data samples, into the formed categories.

Unsupervised, semi-supervised, and supervised learning using deep learning based probabilistic generative models

A system for generating a set of digital image features, comprising at least one hardware processor adapted for: producing a plurality of input groups of features, each produced by extracting a plurality of features from one of a plurality of digital images; computing an output group of features by inputting the plurality of input groups of features into at least one prediction model trained to produce a model group of features in response to at least two groups of features, such that a model set of labels indicative of the model group of features is similar, according to at least one similarity test, to a target set of labels computed by applying at least one set operator to a plurality of input sets of labels each indicative of one of the at least two groups of features; and providing the output group of features to at least one other processor.

Method and system for producing digital image features

Embodiments provide 3D coordinates for points in a scene that are observed to be in the correct physical position in a series of images. A method may comprise obtaining a plurality of images including a base image having at least one annotated point corresponding to a point of an object shown in the base image, and a plurality of side images showing the object from different viewpoints than the base image, wherein the plurality of side images are given with the camera poses relative to the base image, extracting from at least some of the side images, image patches showing the annotated point, wherein a plurality of sets of image patches are extracted, wherein a set of image patches is extracted at a plurality of corresponding candidate depth values, classifying each set as having a corresponding candidate depth value that is correct or incorrect, and outputting a correct depth value.

Point depth estimation from a set of 3D-registered images

Embodiments may provide visual recognition techniques that provide improved recognition accuracy and reduced use of computing resources in cases where only a small set of examples is used to train an unlimited number of recognized categories. For example, in an embodiment, a computer-implemented method of visual recognition may comprise generating a plurality of personal embedding models, each personal embedding model including categories relating to a person, and object, or a subject, wherein at least some of the personal embedding models include at least some different categories, training the plurality of personal embedding models using image training data having a limited number of examples of each category, wherein the examples of each category are used to train more than one category in more than one of the personal embedding models, recognizing images from image data using the plurality of personal embedding models, and outputting information relating to the recognized images.

System and method for a visual recognition and/or detection of a potentially unbounded set of categories with limited examples per category and restricted query scope

A method can include learning a common embedding space and a set of parameters for each one of a plurality of sets of mixture models, wherein one mixture model is associated with one class of objects within a set of object categories. The method can also include adding new mixture models to the set of mixture models to support novel categories based on a set of example embedding vectors computed for each one of the novel categories. Additionally, the method includes detecting in images a plurality of boxes with associated labels and corresponding confidence scores, wherein the boxes correspond to image regions comprising objects of both known categories and the novel categories. Furthermore, the method includes, given a query image, executing an instruction based on the common embedding space and the set of mixture models, the instruction comprising identifying objects from both categories in the query image.

Representative-based metric learning for classification and few-shot object detection

Embodiments of the present disclosure include training a model using a plurality of pairs of feature vectors related to a first class. Embodiments include providing a sample feature vector related to a second class as an input to the model. Embodiments include receiving at least one synthesized feature vector as an output from the model. Embodiments include training a classifier to recognize the second class using a training data set comprising the sample feature vector related to the second class and the at least one synthesized feature vector. Embodiments include providing a query feature vector as an input to the classifier. Embodiments include receiving output from the classifier that identifies the query feature vector as being related to the second class, wherein the output is used to perform an action.

Out-of-sample generating few-shot classification networks

A method of constructing hierarchical representation models of compound 3D physical objects, comprising receiving a plurality of image sets depicting a compound 3D physical object constructed of a static component having a single representation state and dynamic component(s) having multiple representation states, each pair of the image sets corresponds depicts a respective dynamic component in one of its representation states, generating a pair of point cloud structures from each pair of image sets, analyzing the point cloud structures pair to identify common region(s) associated with the static component and region(s) of difference associated with the dynamic component, generating a hierarchical 3D representation model comprising a high level model of the static component generated from the common region(s) with a surface outline of the region(s) of difference associated with multiple lower level models constructed for the representation states of the dynamic component and outputting the hierarchical 3D representation model.

Construction of an efficient representation for a three-dimensional (3D) compound object from raw video data

There is provided a method of identifying objects in an image, comprising: extracting query descriptors from the image, comparing each query descriptor with training descriptors for identifying matching training descriptors, each training descriptor is associated with a reference object identifier and with relative location data (distance and direction from a center point of a reference object indicated by the reference object identifier), computing object-regions of the digital image by clustering the query descriptors having common center points defined by the matching training descriptors, each object-region approximately bounding one target object and associated with a center point and a scale relative to a reference object size, wherein the object-regions are computed independently of the identifier of the reference object associated with the object-regions, wherein members of each cluster point toward a common center point, and classifying the target object of each object-region according to the reference object identifier of the cluster.

Growing community of inventors

Binyamina, Israel

Joseph Shtok