Honda Motor Co., Ltd.

Facebook Technologies, LLC

Ananth Ranganathan

Aniket Murarka

Ioannis Pavlidis

Oytun Akman

Jeffrey Witthuhn

Meghana Reddy Guduru

Tao Wu

Jacob V Menashe

Kikuo Fujimura

Ming-Hsuan Yang

Rakesh Gupta

Jongwoo Lim

Andrew Melim

Yan Lu

Roozbeh Mottaghi

In general, this disclosure describes an artificial reality system that provides asymmetric user experiences to users associated with user devices that operate according to different modes of engagement with the artificial reality system. Different user devices may have different capabilities, be used by users having different roles for an artificial reality application, or otherwise be configured to interact in a variety of ways with an artificial reality system.

Artificial reality system having multiple modes of engagement

An artificial reality system is described that implements adaptive degrees-of-freedom (DOF) selection when tracking frames of reference and rendering artificial reality content. In one example, the artificial reality system comprises a head mounted display (HMD) that outputs rendered artificial reality content. A performance monitor determines one or more performance indicators associated with the artificial reality system. A degree-of-freedom (DOF) selector applies one or more policies to the performance indicators to select between a first mode in which a pose tracker computes one or more poses of the HMD within the 3D environment using 6DOF and a second mode in which the pose tracker computes the one or more poses using 3DOF. The pose tracker computes the one or more poses for the HMD within the 3D environment in accordance with the selected mode. A rendering engine renders the content for the artificial reality application based on the computed pose.

Artificial reality system having adaptive degrees of freedom (DOF) selection

Systems and techniques for sign based localization are provided herein. Sign based localization may be achieved by capturing an image of an operating environment around a vehicle, extracting one or more text candidates from the image, detecting one or more line segments within the image, defining one or more quadrilateral candidates based on one or more of the text candidates, one or more of the line segments, and one or more intersections of respective line segments, determining one or more sign candidates for the image based on one or more of the quadrilateral candidates and one or more of the text candidates, matching one or more of the sign candidates against one or more reference images, and determining a location of the vehicle based on a match between one or more of the sign candidates and one or more of the reference images.

Sign based localization

One or more embodiments of techniques or systems for creating a road marking classification template and vehicle localization using road markings are provided herein. A road marking classification template database includes templates of training images taken from different navigation environments. A training image with a road marking can be rectified and enhanced. Corners can be calculated for the road marking using the rectified or enhanced image. Locations can be determined for the corners and stored as part of a template. Similarly, a runtime image can also be rectified, enhanced, boosted, etc. Additionally, corners can be calculated for the runtime image and matched against corners of templates from the template database. A location or a pose of a vehicle can be determined using the match. In this way, vehicle localization is provided such that drift or other issues associated with GPS, such as occlusion, are mitigated, for example.

Localization using road markings

A system and method for mapping, localization and pose correction including, determining a current position of a vehicle along a travel route and a set of currently observable landmarks along the travel route relative to the current position, the set of currently observable landmarks extracted from one or more stereo images obtained from an imaging device, and querying a survey landmark database to identify a subset of surveyed landmarks relative to the current position of the vehicle. The method including determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks and identifying a best transform estimate from the one or more two-dimensional transform estimates that minimizes distances between the set of currently observable landmarks and the subset of surveyed landmarks. The method including correcting a pose of the vehicle based on the best transform estimate.

System and method for image based mapping, localization, and pose correction of a vehicle with landmark transform estimation

System and method for mapping, localization and pose correction of a vehicle based on images

A system and method are provided for learning part-based object models during a learning phase from training images and applying the learned object models to an input image during runtime. The learned part-based object models are augmented by appearance-based models of the objects. The part-based object models correspond to the shapes of the parts of an object. The appearance-based models provide additional appearance cues to the object models for object classification. The approach to learning part-based object models has the capability of learning object models without using viewpoint labels of the objects. The learning is also invariant to scale and in-plane rotation of the objects.

Learning part-based models of objects

The prevention of vehicle accidents is targeted. A road texture model is created based on a vehicle camera image. An initial vehicle location estimate is determined, and map imagery is obtained based on this location estimate. A refined vehicle location is determined using visual egomotion. In particular, 3D features of the vehicle image and the retrieved map imagery are identified and aligned. A map image is selected based on this alignment, and the location associated with the map image is modified by a displacement between the selected map image and the vehicle image to produce a refined vehicle location. A road boundary model is created based on the road texture model and the refined vehicle location, and a road departure model is created based on the road boundary model and vehicle odometry information. The operator of the vehicle is warned of a road departure based on the road departure model.

Road departure warning system

A system and method are disclosed for detecting road marking in a video using learned road marking templates. The system comprises a template learning module configured to learn the feature-based road marking templates from a set of training images. The template learning module is configured to rectify each training image, detect multiple regions of interest, and for each detected region of interest, detect multiple key points. The template learning module extracts feature vectors for the detected key points and builds the road marking templates from the feature vectors. The system also includes a road marking detection module for detecting road markings in a video at runtime using the learned road marking templates. During runtime, these templates are matched using a two-step process of first selecting promising feature matches and subsequently performing a structural matching to account for the shape of the road markings.

Road marking detection and recognition

A system and method are disclosed for detecting and labeling places recognized in a video stream using change-points detection. The system includes a segmentation module and a label learning module. The segmentation module is configured to receive a video stream comprising multiple digital representations of images. The video stream is represented by a measurement stream comprising one or more image histograms of the video stream. The segmentation module segments the measurement stream into multiple segments corresponding to place recognized in the videos stream. The segmentation module detects change-points of the measurement stream and computes probability distributions of the segments over multiple pre-learned place models. The label generation module is configured to generate place labels for the places recognized by the place models.

Detecting and labeling places using runtime change-point detection

A system and method are disclosed for detecting and labeling places in a video stream using change-points detection. The system comprises a place label generation module configured to assign place labels probabilistically to places in the video stream based on the measurements of the measurement stream representing the video. For each measurement in the segment, the place label generation module classifies the measurement by computing the probability of the measurement being classified by a learned Gaussian Process classifier. Based on the probabilities generated with respect to all the measurements in the segment, the place label generation module determines the place label for the segment. In cases where a Gaussian Process classifier cannot positively classify a segment, the place label generation module determines whether the segment corresponds to an unknown place based on the perplexity statistics of the classification and a threshold value.

Detecting and labeling places using runtime change-point detection and place labeling classifiers

A system and method are disclosed for learning a random multinomial logit (RML) classifier and applying the RML classifier for scene segmentation. The system includes an image textonization module, a feature selection module and a RML classifier. The image textonization module is configured to receive an image training set with the objects of the images being pre-labeled. The image textonization module is further configured to generate corresponding texton images from the image training set. The feature selection module is configured to randomly select one or more texture-layout features from the texton images. The RML classifier comprises multiple multinomial logistic regression models. The RML classifier is configured to learn each multinomial logistic regression model using the selected texture-layout features. The RML classifier is further configured to apply the learned regression models to an input image for scene segmentation.

Semantic scene segmentation using random multinomial logit (RML)

An online sparse matrix Gaussian process (OSMGP) uses online updates to provide an accurate and efficient regression for applications such as pose estimation and object tracking. A regression calculation module calculates a regression on a sequence of input images to generate output predictions based on a learned regression model. The regression model is efficiently updated by representing a covariance matrix of the regression model using a sparse matrix factor (e.g., a Cholesky factor). The sparse matrix factor is maintained and updated in real-time based on the output predictions. Hyperparameter optimization, variable reordering, and matrix downdating techniques can also be applied to further improve the accuracy and/or efficiency of the regression process.

Growing community of inventors

Santa Clara, CA, United States of America

Ananth Ranganathan