The Penn State Research Foundation

The Penn State University

Jia Li

James Z Wang

Poonam Suryanarayan

Xin Lu

Reginald B Adams

Michelle Newman

Yu Zhang

Michael Alan Steinberg

Jianbo Ye

Lei Yao

Stephen Wistar

Ritendra Datta

Siqiong He

Neela Sawant

Zihan Zhou

Mu Qiao

A method of improving quality of crowdsourced affective data based on agreement relationship between a plurality of annotators include receiving, by a processor, a collection of stimuli previously given affective labels by the plurality of annotators, executing, by a processor, an algorithm operative to perform the steps including constructing an agreement multigraph as a probabilistic model including a pair-wise status of agreement between the affective labels given by different ones of the plurality of annotators, learning the probabilistic model computationally using the crowdsourced affective data, identifying a reliability of each of the plurality of annotators based on the learned model, and adjusting the crowdsourced affective data by calculating the affective labels of each stimuli based on the identified reliability of each of the plurality of annotators, thereby improving the quality of the crowdsourced affective data.

Probabilistic multigraph modeling for improving the quality of crowdsourced affective data

Systems, methods, and computer-readable media for electronically assessing a visual significance of pixels or regions in an electronic image are disclosed. A method includes receiving the electronic image, performing a composition analysis on the electronic image, the composition analysis includes partitioning the electronic image into a plurality of segments or a plurality of parts, constructing an attributed composition graph having a plurality of nodes, where each node corresponds to a segment or a part and where each node includes one or more attributes, modeling the visual significance of the electronic image based on the attributed composition graph using a statistical modeling process or a computational modeling process to obtain a plurality of values, and constructing a composition significance map having a significance score for each segment or each part according to the values obtained from the statistical modeling process or the computational modeling process.

Systems and methods for detection of significant and attractive components in digital images

Shape features in natural images influence emotions aroused in human beings. An in-depth statistical analysis helps to understand the relationship between shapes and emotions. Through experimental results on the International Affective Picture System (IAPS) dataset, evidence is presented as to the significance of roundness-angularity and simplicity-complexity on predicting emotional content in images. Shape features are combined with other state-of-the-art features to show a gain in prediction and classification accuracy. Emotions are modeled from a dimensional perspective in order to predict valence and arousal ratings, which have advantages over modeling the traditional discrete emotional categories. Images are distinguished vis-a-vis strong emotional content from emotionally neutral images with high accuracy. All of the methods and steps disclosed herein are implemented on a programmed digital computer, which may be a stand-alone machine or integrated into another piece of equipment such as a digital still or video camera including, in all embodiments, portable devices such as smart phones.

Automatically computing emotions aroused from images through shape modeling

Satellite images from vast historical archives are analyzed to predict severe storms. We extract and summarize important visual storm evidence from satellite image sequences in a way similar to how meteorologists interpret these images. The method extracts and fits local cloud motions from image sequences to model the storm-related cloud patches. Image data of an entire year are adopted to train the model. The historical storm reports since the year 2000 are used as the ground-truth and statistical priors in the modeling process. Experiments demonstrate the usefulness and potential of the algorithm for producing improved storm forecasts. A preferred method applies cloud motion estimation in image sequences. This aspect of the invention is important because it extracts and models certain patterns of cloud motion, in addition to capturing the cloud displacement.

Identifying visual storm signatures from satellite images

Computationally efficient accelerated D2-clustering algorithms are disclosed for clustering discrete distributions under the Wasserstein distance with improved scalability. Three first-order methods include subgradient descent method with re-parametrization, alternating direction method of multipliers (ADMM), and a modified version of Bregman ADMM. The effects of the hyper-parameters on robustness, convergence, and speed of optimization are thoroughly examined. A parallel algorithm for the modified Bregman ADMM method is tested in a multi-core environment with adequate scaling efficiency subject to hundreds of CPUs, demonstrating the effectiveness of AD2-clustering.

Accelerated discrete distribution clustering under wasserstein distance

The trend of analyzing big data in artificial intelligence requires more scalable machine learning algorithms, among which clustering is a fundamental and arguably the most widely applied method. To extend the applications of regular vector-based clustering algorithms, the Discrete Distribution (D2) clustering algorithm has been developed for clustering bags of weighted vectors which are well adopted in many emerging machine learning applications. The high computational complexity of D2-clustering limits its impact in solving massive learning problems. Here we present a parallel D2-clustering algorithm with substantially improved scalability. We develop a hierarchical structure for parallel computing in order to achieve a balance between the individual-node computation and the integration process of the algorithm. The parallel algorithm achieves significant speed-up with minor accuracy loss.

Massive clustering of discrete distributions

Automatic selection of training images is enhanced using an instance-weighted mixture modeling framework called ARTEMIS. An optimization algorithm is derived that in addition to mixture parameter estimation learns instance-weights, essentially adapting to the noise associated with each example. The mechanism of hypothetical local mapping is evoked so that data in diverse mathematical forms or modalities can be cohesively treated as the system maintains tractability in optimization. Training examples are selected from top-ranked images of a likelihood-based image ranking. Experiments indicate that ARTEMIS exhibits higher resilience to noise than several baselines for large training data collection. The performance of ARTEMIS-trained image annotation system is comparable to using manually curated datasets.

Instance-weighted mixture modeling to enhance training collections for image annotation

A composition model is developed based on the image segmentation and the vanishing point of the scene. By integrating both photometric and geometric cues, better segmentation is provided. These cues are directly used to detect the dominant vanishing point in an image without extracting any line segments. Based on the composition model, a novel image retrieval system is developed which can retrieve images with similar compositions as the query image from a collection of images and provide feedback to photographers.

Composition modeling for photo retrieval through geometric image segmentation

A comprehensive system to enhance the aesthetic quality of the photographs captured by mobile consumers provides on-site composition and aesthetics feedback through retrieved examples. Composition feedback is qualitative in nature and responds by retrieving highly aesthetic exemplar images from the corpus which are similar in content and composition to the snapshot. Color combination feedback provides confidence on the snapshot to contain good color combinations. Overall aesthetics feedback predicts the aesthetic ratings for both color and monochromatic images. An algorithm is used to provide ratings for color images, while new features and a new model are developed to treat monochromatic images. This system was designed keeping the next generation photography needs in mind and is the first of its kind. The feedback rendered is guiding and intuitive in nature. It is computed in situ while requiring minimal input from the user.

On-site composition and aesthetics feedback through exemplars for photographers

The aesthetic quality of a picture is automatically inferred using visual content as a machine learning problem using, for example, a peer-rated, on-line photo sharing Website as data source. Certain visual features of images are extracted based on the intuition that they can discriminate between aesthetically pleasing and displeasing images. A one-dimensional support vector machine is used to identify features that have noticeable correlation with the community-based aesthetics ratings. Automated classifiers are constructed using the support vector machines and classification trees, with a simple feature selection heuristic being applied to eliminate irrelevant features. Linear regression on polynomial terms of the features is also applied to infer numerical aesthetics ratings.

Growing community of inventors

State College, PA, United States of America

Jia Li