Gm Global Technolgoy Operations, Inc.

The University of Waterloo

Qingrong Zhao

Bakhtiar Brian Litkouhi

Jinsong Wang

Qi Zhang

Wende Zhang

Nikolai K Moshchuk

SeyedAlireza Kasaiezadeh Mahabadi

Mohsen Khalili

Shuqing Zeng

Wei Tong

Amir Khajepour

Yubiao Zhang

Jingfu Jin

Yingmei Si

Xiaosong Huang

Youssef A Ghoneim

Shih-Ken Chen

Allan K Lewis

Weiwen Deng

Wen-Chiao Lin

David M Sidlosky

Shiming Duan

Arun Adiththan

Robert D Sims, Iii

Sami Ahmed

Michael T Chaney, Jr

Ramesh Sethu

Ehsan Hashemi

Farui Peng

Yechen Qin

Amin Habibnejad Korayem

Yadollah Sabri

Steven T Schweitzer

Valarie V Boatman

Joonho Lee

Edward Joseph Ecclestone

Chao Yu

Tianyu Wang

Simon E Zimmerman

A system and method of automatic image view alignment for a camera-based road condition detection on a vehicle. The method includes transforming a fisheye image into a non-distorted subject image, comparing the subject image with a reference image, aligning the subject image with the reference image, and analyzing the aligned subject image to detect and identify road conditions in real-time as the vehicle is in operation. The subject image is aligned with the reference image by determining a distance (d) between predetermined feature points of the subject and reference images, estimating a pitchof a projection center based on the distance d, and generating an aligned subject image by applying a rectification transformation on the fisheye image by relocating a center of projection of the fisheye image by the pitch angle.

System and method of automatic image view alignment for camera-based road condition detection on a vehicle

A system for learning-model predictive control (LMPC) with multi-step prediction for motion control of a vehicle includes sensors and actuators. One or more control modules each having a processor, a memory, and input/output (I/O) ports are in communication with the sensors and actuators, the processor executing program code portions stored in the memory. The program code portions cause the sensors and actuators to obtain vehicle state information, receive a driver input, and generate a desired dynamic output based on the driver input and the vehicle state information. A program code portion estimates actions of the actuators based on the vehicle state information and the driver input, and utilizes the vehicle state information, the driver input, and the estimated actions of the actuators to select one or more models of a physics-based vehicle model and a machine-learning model of the vehicle to selectively adjust commands to the actuators.

Learning-model predictive control with multi-step prediction for vehicle motion control

A motor vehicle motion control health monitoring system includes sensors and actuators disposed on the motor vehicle. The sensors measure real-time static and dynamic telemetry data about the motor vehicle, and the actuators alter static and dynamic behavior of the motor vehicle. A control module has a processor, a memory, and input/output (I/O) ports. The processor executes program code portions stored in the memory, the program code portions include: an offline portion that collects telemetry data from the motor vehicle, performs failure analysis on the telemetry data and allocates tasks based on the failure analysis; and an online portion that analyzes the telemetry data for failures within specific sensors, actuators, or functions that utilize systems of sensors and/or actuators. The online portion mitigates deviations in the telemetry data by sending a correction to the one or more sensors, actuators, and/or functions of a motor vehicle motion control system.

Architecture and methodology for vehicle motion control health monitoring and mitigation

A method of estimating a performance characteristic of a wheel of a vehicle, includes selecting relevant input features based on wheel dynamics and tire behavior, and collecting experimental data for each of the relevant input features at each of a plurality of vehicle operating conditions. The method further includes manually identifying and labeling wheel stability status over time from the experimental data and calculating tractive limit over time from the experimental data. The method also includes training a tractive limit model and training a wheel stability status model. The method further includes receiving a plurality of testing inputs, wherein each of the plurality of testing inputs is received from a sensor on-board the vehicle or from a controller on-board the vehicle and, processing the received testing inputs in a predetermined machine learning process to calculate in one or more data processors a prediction of the performance characteristic.

Machine learning-based tractive limit and wheel stability status estimation

A universal machine learning based system for estimating a vehicle state of a vehicle includes one or more controllers executing instructions to receive a plurality of dynamic variables and corresponding historical data. The controllers execute a sensitivity analysis algorithm to determine a sensitivity level for each dynamic variable and corresponding historical data and select two or more pertinent dynamic variables based on the sensitivity level of each dynamic variable and the corresponding historical data. The controllers standardize the two or more pertinent dynamic variables into a plurality of generic dynamic variables, wherein the plurality of generic dynamic variables are in a standardized format that is applicable to any configuration of vehicle, and estimate the vehicle state based on the plurality of generic dynamic variables by one or more machine learning algorithms.

Universal machine learning based system for estimating a vehicle state

Systems and method are provided for controlling a vehicle. In one embodiment, a method includes: receiving a first surface value associated with a first road surface area in an upcoming environment of the vehicle; receiving a second surface value associated with a second road surface area in the upcoming environment of the vehicle; determining a change in surface value based on the first surface value and the second surface value; and in response to the change in surface value being greater than a threshold, adapting at least one of vehicle collision warning messages, vehicle braking control, vehicle steering control, and path planning based on the second surface value.

Vehicle lateral motion management with preview road surface information

A system and method of automatically activating a windshield wiper system of a vehicle having a front windshield with a front wiper and a rear windshield with a rear wiper are provided. The method comprises assessing at least one windshield classification of road conditions based on original information and capturing a front image of the front windshield, a rear image of the rear windshield, and an environment image of the environment. The method further comprises classifying the images to define a first windshield class. The method further comprises determining a front windshield perception, a rear windshield perception, and an environment perception of the first windshield class to define a first combination of detection sources. The method further comprises fusing the front windshield perception, the rear windshield perception and the environment perception, defining a first front probability of the first windshield class. The method further comprises activating the front wiper when the first front probability is greater than a front threshold.

System and method of automatically activating windshield wiper of a vehicle

Methods and systems are provided for controlling a vehicle action based on a condition of a road on which a vehicle is travelling, including: obtaining first sensor data as to a surface of the road from one or more first sensors onboard the vehicle; obtaining second sensor data from one or more second sensors onboard the vehicle as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; generating a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classifying, via a processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and controlling a vehicle action based on the classification of the condition of the road.

Apparatus and methodology of road condition classification using sensor data

Presented are automated driving systems for executing intelligent vehicle operations in mixed-mu road conditions, methods for making/using such systems, and vehicles with enhanced headway control for transitional surface friction conditions. A method for executing an automated driving operation includes a vehicle controller receiving sensor signals indicative of road surface conditions of adjoining road segments, and determining, based on these sensor signals, road friction values for the road segments. The controller determines whether the road friction value is increasing or decreasing, and if a difference between the road friction values is greater than a calibrated minimum differential. Responsive to the friction difference being greater than the calibrated minimum differential and the road friction value decreasing, the vehicle controller executes a first vehicle control action. Conversely, if the friction difference is greater than the calibrated minimum but the road friction value is increasing, the controller responsively executes a second vehicle control action.

Automated driving systems and control logic with enhanced longitudinal control for transitional surface friction conditions

Systems and methods are provided for generating adapted tuning parameters for target slip estimation, the parameters being adapted to real-time road surface conditions. The method includes, receiving, from a road surface detection module, a road surface condition, S, from among N road surface conditions S, range of friction, mu, and a confidence level, Ci. The method receives sensor system data from a sensor system, and determines, as a function of S, range of mu, and Ci, initial estimator values including an estimated initial frictional force {circumflex over (&#x398;)}(0), an initial gain, P, and an initial projected range of signal bounds, (P) and (P). The method tunes (i.e., adapts) the initial estimator values to generate therefrom adapted tuning parameters based on received inputs. The method outputs adapted tuning parameters.

System and method for adapting parameters used in target slip estimation

A vehicle subsystem includes an on-vehicle camera that is disposed to monitor a field of view (FOV) that includes a travel surface for the vehicle. A controller captures, via the on-vehicle camera, an image file associated with the FOV and segments the image file into a first set of regions associated with the travel surface and a second set of regions associated with an above-horizon portion. Image features on each of the first set of regions and the second set of regions are extracted and classified. A surface condition for the travel surface for the vehicle is identified based upon the classified extracted image features from each of the first set of regions and the second set of regions. Operation of the vehicle is controlled based upon the identified surface condition.

Method and apparatus for evaluating a vehicle travel surface

Systems, Methods and Apparatuses are provided for detecting surface conditions, which includes: an image scene captured by a camera wherein the image scene includes: a set of a plurality of regions of interest (ROIs); and a processor configured to receive the image scene to: extract at least a first and a second ROI from the set of the plurality of ROIs of the image scene; associate the first ROI with an above-horizon region and associate the second ROI with a surface region; analyze the first ROI and the second ROI in parallel for a condition related to an ambient lighting in the first ROI and for an effect related to the ambient lighting in the second ROI; and extract from the first ROI features of the condition of the ambient lighting and extract from the second ROI features of the effect of the ambient lighting on a surface region.

Systems, methods and apparatuses are provided for enhanced surface condition detection based on image scene and ambient light analysis

A vehicle includes a plurality of on-vehicle cameras, and a controller executes a method to evaluate a travel surface by capturing images for fields of view of the respective cameras. Corresponding regions of interest for the images are identified, wherein each of the regions of interest is associated with the portion of the field of view of the respective camera that includes the travel surface. Portions of the images are extracted, wherein each extracted portion is associated with the region of interest in the portion of the field of view of the respective camera that includes the travel surface and wherein one extracted portion of the respective image includes the sky. The extracted portions of the images are compiled into a composite image datafile, and an image analysis of the composite image datafile is executed to determine a travel surface state. The travel surface state is communicated to another controller.

A method of identifying a condition of a road surface includes capturing at least a first image of the road surface with a first camera, and a second image of the road surface with a second camera. The first image and the second image are tiled together to form a combined tile image. A feature vector is extracted from the combined tile image using a convolutional neural network, and a condition of the road surface is determined from the feature vector using a classifier.

Method of classifying a condition of a road surface

A method for determining wetness on a path of travel. A surface of the path of travel is captured by at least one image capture device. A plurality of wet surface detection techniques is applied to the at least one image. An analysis for each wet surface detection technique is determined in real-time of whether the surface of the path of travel is wet. Each analysis independently determines whether the path of travel is wet. Each analysis by each wet surface detection technique is input to a fusion and decision making module. Each analysis determined by each wet surface detection technique is weighted within the fusion and decision making module by comprehensive analysis of weather information, geology information, and vehicle motions. A wet surface detection signal is provided to a control device. The control device applies the wet surface detection signal to mitigate the wet surface condition.

Growing community of inventors

Madison Heights, MI, United States of America

Qingrong Zhao