Videomining Corporation

Joonhwa Shin

Rajeev Sharma

Donghun Kim

Youngrock R Yoon

Namsoon Jung

Hankyu Moon

The method and system evaluates the effectiveness of a display location within a store based on a behavioral response analysis of shoppers in the vicinity. The effectiveness of a display location is measured by tracking shoppers in-store, extracting and processing shopper attributes, and extracting metrics based on the processed attributes. The metrics of one location is compared to the metrics of another location to determine overall effectiveness.

Method and system for measuring in-store location effectiveness based on shopper response and behavior analysis

A method and system for creating an anonymous shopper panel based on multi-modal sensor data fusion. The anonymous shopper panel can serve as the same traditional shopper panel who reports their household information, such as household size, income level, demographics, etc., and their purchase history, yet without any voluntary participation. A configuration of vision sensors and mobile access points can be used to detect and track shoppers as they travel a retail environment. Fusion of those modalities can be used to form a trajectory. The trajectory data can then be associated with Point of Sale data to form a full set of shopper behavior data. Shopper behavior data for a particular visit can then be compared to data from previous shoppers' visits to determine if the shopper is a revisiting shopper. The shopper's data can then be aggregated for multiple visits to the retail location. The aggregated shopper data can be filtered using application-specific criteria to create an anonymous shopper panel.

Method and system for creating anonymous shopper panel using multi-modal sensor fusion

A method and system for associating a mobile device carried by a shopper with Point-of-Sale (PoS) data from that shopper's transaction at a retail location. Specifically, the association can utilize face data that was associated with the mobile device to match face data that was generated by association with PoS data. The association with a mobile device can also include the calculation of a mobile trajectory to enable the generation of a shopper profile including shopper behavior. The mobile to PoS association can include tracking of the mobile device throughout a retail location to form a trajectory. The association of a face to a mobile device can utilize the device's MAC address and repeat visit analysis to determine a match. The association of a face to PoS data can utilize the aligning of event time series based on a dynamic time disparity to match checkout events with PoS events.

Association of mobile device to retail transaction

A method and system for cross-channel shopper behavior analysis. Tracking individual shopper behavior across many retail locations (i.e., across multiple channels) can be extremely valuable for product manufacturers and retails. A configuration of vision sensors and mobile access points can be used to detect and track shoppers as they travel a retail environment, forming a trajectory. The trajectory data can then be associated with Point of Sale data to form a full set of shopper behavior data. The shopper's data can then be aggregated for multiple visits to many retail locations in a geographic area. The aggregated shopper data can be filtered using application-specific criteria for further analysis.

Cross-channel in-store shopper behavior analysis

The present invention provides a comprehensive method for automatically and unobtrusively analyzing the in-store behavior of people visiting a physical space using a multi-modal fusion based on multiple types of sensors. The types of sensors employed may include cameras for capturing a plurality of images and mobile signal sensors for capturing a plurality of Wi-Fi signals. The present invention integrates the plurality of input sensor measurements to reliably and persistently track the people's physical attributes and detect the people's interactions with retail elements. The physical and contextual attributes collected from the processed shopper tracks includes the motion dynamics changes triggered by an implicit and explicit interaction to a retail element, comprising the behavior information for the trip of the people. The present invention integrates point-of-sale transaction data with the shopper behavior by finding and associating the transaction data that corresponds to a shopper trajectory and fusing them to generate a complete an intermediate representation of a shopper trip data, called a TripVector. The shopper behavior analyses are carried out based on the extracted TripVector. The analyzed behavior information for the shopper trips yields exemplary behavior analysis comprising map generation as visualization of the behavior, quantitative shopper metric derivation in multiple scales (e.g., store-wide and category-level) including path-to-purchase shopper metrics (e.g., traffic distribution, shopping action distribution, buying action distribution, conversion funnel), category dynamics (e.g., dominant path, category correlation, category sequence). The present invention includes a set of derived methods for different sensor configurations.

Method and system for in-store shopper behavior analysis with multi-modal sensor fusion

A method and system for associating a physically identifiable feature of a person with the unique identifier of a mobile device carried by the person as detected across repeat visits to a physical location or multiple locations. Specifically, associating the face of a repeat visitor with a unique identifier of a mobile device, such as the device's MAC address, by means of a repeat visit analysis without any explicit or voluntary participation of the person, for example, in a form of providing their information by participation in a survey. For each detection of a particular MAC address associated with a mobile device at a particular physical location, a set of candidate faces can be captured at that physical location. Revisiting faces can be identified and grouped together, and a probability that each group of faces is associated with a particular MAC address can then be calculated. After satisfying several constraints, a maximum probability can be determined, resulting in a match of a particular face to a particular MAC address.

Association of unique person to a mobile device using repeat face image matching

A method and system for associating an image of a face of at least a person with Point-of-Sale (PoS) data by aligning a first event time series with a second event time series based on a dynamic time disparity. An exemplary embodiment can generate an event time series containing facial recognition data for a person or persons during the PoS transaction process. These data can form a vision-based checkout event time series. An embodiment can also collect PoS transaction data from the retail checkout system, using timestamp information to create a second event time series. As there may be a time disparity between the time series, they can then be aligned in order to match events from one time series to the other. Faces identified in checkout events in the first time series can be registered to PoS events and the results stored in a database.

Association of unique person to point-of-sale transaction data

The present invention is a system and method for performing ethnicity classification based on the facial images of people, using multi-category decomposition architecture of classifiers, which include a set of predefined auxiliary classifiers that are specialized to auxiliary features of the facial images. In the multi-category decomposition architecture, which is a hybrid multi-classifier architecture specialized to ethnicity classification, the task of learning the concept of ethnicity against significant within-class variations, is handled by decomposing the set of facial images into auxiliary demographics classes; the ethnicity classification is performed by an array of classifiers where each classifier, called an auxiliary class machine, is specialized to the given auxiliary class. The facial image data is annotated to assign the age and gender labels as well as the ethnicity labels. Each auxiliary class machine is trained to output both the given auxiliary class membership likelihood and the ethnicity likelihoods. Faces are detected from the input image, individually tracked, and fed to all the auxiliary class machines to compute the desired auxiliary class membership and ethnicity likelihood outputs. The outputs from all the auxiliary class machines are combined in a manner to make a final decision on the ethnicity of the given face.

Method and system for determining ethnicity category of facial images based on multi-level primary and auxiliary classifiers

The present invention provides a comprehensive method to design an automatic media audience measurement system that can estimate the site-wide audience of a media of interest (e.g., the site-wide viewership of a target display) based on the measurements of a subset of the actual audience sampled from a limited space in the site. This invention enables (1) the placement of sensors in optimal positions for the viewership data measurement and (2) the estimation of the site-wide viewership of the target display by performing the viewership extrapolation based on the sampled viewership data. The viewership extrapolation problem is formulated in a way that the time-varying crowd dynamics around the target display is an important decisive factor as well as the sampled viewership data at a given time in yielding the estimated site-wide viewership. To solve this problem, the system elements that affect the viewership&#x2014;site, display, crowd, and audience&#x2014;and their relationships are first identified in terms of the visibility, the viewership relevancy, and the crowd occupancy. The optimal positions of the sensors are determined to cover the maximum area of the viewership with high probabilities. The viewership extrapolation function is then modeled and learned from the sampled viewership data, the site-wide viewership data, and the crowd dynamics measurements while removing the noise in the sampled viewership data using the viewership relevancy of the measurements to the target display.

Growing community of inventors

State College, PA, United States of America

Joonhwa Shin