Stradvision, Inc.

Stadvision, Inc.

Stradivision, Inc.

Yongjoong Kim

Hongmo Je

Wooju Ryu

SukHoon Boo

Hojin Cho

Taewoong Jang

Kyungjoong Jeong

Donghun Yeo

Myungchul Sung

Woonhyun Nam

Kye-Hyeon Kim

Hak-Kyoung Kim

Insu Kim

Hyungsoo Lee

Myeong-Chun Lee

DongSoo Shin

Dongsoo Shin

Bongnam Kang

Sung An Gweon

Dongkyu Yu

Woonhyu Nam

Sukhoon Boo

Sungan Gweon

Taewong Jang

A method for learning parameters of a CNN using a 1&#xd7;K convolution operation or a K&#xd7;1 convolution operation is provided to be used for hardware optimization which satisfies KPI. The method includes steps of: a learning device (a) instructing a reshaping layer to two-dimensionally concatenate features in each group comprised of corresponding K channels of a training image or its processed feature map, to thereby generate a reshaped feature map, and instructing a subsequent convolutional layer to apply the 1&#xd7;K or the K&#xd7;1 convolution operation to the reshaped feature map, to thereby generate an adjusted feature map; and (b) instructing an output layer to refer to features on the adjusted feature map or its processed feature map, and instructing a loss layer to calculate losses by referring to an output from the output layer and its corresponding GT.

Learning method and learning device for CNN using 1xK or Kx1 convolution to be used for hardware optimization, and testing method and testing device using the same

A method for on-device continual learning of a neural network which analyzes input data is provided to be used for smartphones, drones, vessels, or a military purpose. The method includes steps of: a learning device, (a) sampling new data to have a preset first volume, instructing an original data generator network, which has been learned, to repeat outputting synthetic previous data corresponding to a k-dimension random vector and previous data having been used for learning the original data generator network, such that the synthetic previous data has a second volume, and generating a batch for a current-learning; and (b) instructing the neural network to generate output information corresponding to the batch. The method can be performed by generative adversarial networks (GANs), online learning, and the like. Also, the present disclosure has effects of saving resources such as storage, preventing catastrophic forgetting, and securing privacy.

Method and device for on-device continual learning of a neural network which analyzes input data, and method and device for testing the neural network to be used for smartphones, drones, vessels, or military purpose

A method for updating an object detector of an autonomous vehicle to adapt the object detector to a driving circumstance is provided. The method includes steps of: a learning device (a) (i) inputting a training image, corresponding to a driving circumstance, into a circumstance-specific object detector to apply (i-1) convolution to the training image to generate a circumstance-specific feature map, (i-2) ROI pooling to the circumstance-specific feature map to generate a circumstance-specific pooled feature map, and (i-3) fully-connected operation to the circumstance-specific pooled feature map to generate circumstance-specific object detection information and (ii) inputting the circumstance-specific feature map into a circumstance-specific ranking network to (ii-1) apply deconvolution to the circumstance-specific feature map and generate a circumstance-specific segmentation map and (ii-2) generate a circumstance-specific rank score via a circumstance-specific discriminator; and (b) training the circumstance-specific object detector, the circumstance-specific deconvolutional layer, the circumstance-specific convolutional layer, and the circumstance-specific discriminator.

Learning method and learning device for updating object detector, based on deep learning, of autonomous vehicle to adapt the object detector to driving circumstance, and updating method and updating device using the same

A method for performing on-device learning of embedded machine learning network of autonomous vehicle by using multi-stage learning with adaptive hyper-parameter sets is provided. The processes include: (a) dividing the current learning into a 1-st stage learning to an n-th stage learning, assigning 1-st stage training data to n-th stage training data, generating a 1_1-st hyper-parameter set candidate to a 1_h-th hyper-parameter set candidate, training the embedded machine learning network in the 1-st stage learning, and determining a 1-st adaptive hyper-parameter set; (b) generating a k_1-st hyper-parameter set candidate to a k_h-th hyper-parameter set candidate, training the (k&#x2212;1)-th stage-completed machine learning network in the k-th stage learning, and determining a k-th adaptive hyper-parameter set; and (c) generating an n-th adaptive hyper-parameter set, and executing the n-th stage learning, to thereby complete the current learning.

Method for performing on-device learning of machine learning network on autonomous vehicle by using multi-stage learning with adaptive hyper-parameter sets and device using the same

A method of on-vehicle active learning for training a perception network of an autonomous vehicle is provided. The method includes steps of: an on-vehicle active learning device, (a) if a driving video and sensing information are acquired from a camera and sensors on an autonomous vehicle, inputting frames of the driving video and the sensing information into a scene code assigning module to generate scene codes including information on scenes in the frames and on driving events; and (b) at least one of selecting a part of the frames, whose object detection information satisfies a condition, as specific frames by using the scene codes and the object detection information and selecting a part of the frames, matching a training policy, as the specific frames by using the scene codes and the object detection information, and storing the specific frames and specific scene codes in a frame storing part.

Method and device for on-vehicle active learning to be used for training perception network of autonomous vehicle

A method for explainable active learning, to be used for an object detector, by using a deep autoencoder is provided. The method includes steps of an active learning device (a) (i) inputting acquired test images into the object detector to detect objects and output bounding boxes, (ii) cropping regions, corresponding to the bounding boxes, in the test images, (iii) resizing the test images and the cropped images into a same size, and (iv) inputting the resized images into a data encoder of the deep autoencoder to output data codes, and (b) (i) confirming reference data codes corresponding to the number of the resized images less than a counter threshold by referring to a data codebook, (ii) extracting specific data codes from the data codes, (iii) selecting specific test images as rare samples, and (iv) updating the data codebook by referring to the specific data codes.

Method for explainable active learning, to be used for object detector, by using deep encoder and active learning device using the same

A method for learning a recurrent neural network to check an autonomous driving safety to be used for switching a driving mode of an autonomous vehicle is provided. The method includes steps of: a learning device (a) if training images corresponding to a front and a rear cameras of the autonomous vehicle are acquired, inputting each pair of the training images into corresponding CNNs, to concatenate the training images and generate feature maps for training, (b) inputting the feature maps for training into long short-term memory models corresponding to sequences of a forward RNN, and into those corresponding to the sequences of a backward RNN, to generate updated feature maps for training and inputting feature vectors for training into an attention layer, to generate an autonomous-driving mode value for training, and (c) allowing a loss layer to calculate losses and to learn the long short-term memory models.

Learning method and learning device of recurrent neural network for autonomous driving safety check for changing driving mode between autonomous driving mode and manual driving mode, and testing method and testing device using them

A method for calibrating a pitch of a camera on a vehicle is provided. The method includes steps of: a calibration device (a) inputting driving images from the camera into an object detection network to detect objects and generate object detection information and into a lane detection network to detect lanes and generate lane detection information; (b) profiling the object and the lane detection information to generate object profiling information and lane profiling information, inputting the object profiling information into an object-based pitch estimation module to select a first target object and a second target object to generate a first pitch and a second pitch, and (iii) inputting vanishing point detection information and the lane profiling information into a lane-based pitch estimation module to generate a third pitch and a fourth pitch; and (c) inputting the first to the fourth pitches into a pitch-deciding module to generate a decided pitch.

Method and device for calibrating pitch of camera on vehicle and method and device for continual learning of vanishing point estimation model to be used for calibrating the pitch

A learning method for acquiring at least one personalized reward function, used for performing a Reinforcement Learning (RL) algorithm, corresponding to a personalized optimal policy for a subject driver is provided. And the method includes steps of: (a) a learning device performing a process of instructing an adjustment reward network to generate first adjustment rewards, by referring to the information on actual actions and actual circumstance vectors in driving trajectories, a process of instructing a common reward module to generate first common rewards by referring to the actual actions and the actual circumstance vectors, and a process of instructing an estimation network to generate actual prospective values by referring to the actual circumstance vectors; and (b) the learning device instructing a first loss layer to generate an adjustment reward and to perform backpropagation to learn parameters of the adjustment reward network.

Learning method and learning device for supporting reinforcement learning by using human driving data as training data to thereby perform personalized path planning

A method of adjustable continual learning of a deep neural network model by using a selective deep generative replay module is provided. The method includes steps of: a learning device (a) (i) inputting training data from a total database and a sub-database into the selective deep generative replay module to generate first and second low-dimensional distribution features, (ii) inputting binary values, random parameters, and the second low-dimensional distribution features into a data generator to generate a third training data, and (iii) inputting a first training data into a solver to generate labeled training data; (b) inputting the training data, the low-dimensional distribution features, and the binary values into a discriminator to generate a first and a second training data scores, a first and a second feature distribution scores, and a third training data score; and (c) training the discriminator, the data generator, the distribution analyzer and the solver.

Method for performing adjustable continual learning on deep neural network model by using selective deep generative replay module and device using the same

A method for generating a lane departure warning (LDW) alarm by referring to information on a driving situation is provided to be used for ADAS, V2X or driver safety which are required to satisfy level 4 and level 5 of autonomous vehicles. The method includes steps of: a computing device instructing a LDW system (i) to collect information on the driving situation including information on whether a specific spot corresponding to a side mirror on a side of a lane, into which the driver desires to change, belongs to a virtual viewing frustum of the driver and (ii) to generate risk information on lane change by referring to the information on the driving situation; and instructing the LDW system to generate the LDW alarm by referring to the risk information. Thus, the LDW alarm can be provided to neighboring autonomous vehicles of level 4 and level 5.

Method and device for learning generating lane departure warning (LDW) alarm by referring to information on driving situation to be used for ADAS, V2X or driver safety required to satisfy level 4 and level 5 of autonomous vehicles

A method for achieving better performance in autonomous driving while saving computing power, by using confidence scores representing a credibility of an object detection which is generated in parallel with an object detection process is provided. And the method includes steps of: (a) a computing device acquiring at least one circumstance image on surroundings of a subject vehicle, through at least one image sensor installed on the subject vehicle; (b) the computing device instructing a convolutional neural network (CNN) to apply at least one CNN operation to the circumstance image, thereby to generate initial object information and initial confidence information on the circumstance image; and (c) the computing device generating final object information on the circumstance image by referring to the initial object information and the initial confidence information with a support of a Reinforcement learning (RL) reinforcement learning (RL) agent, and through V2X communications with at least part of surrounding objects.

Method and device for attention-driven resource allocation by using reinforcement learning and V2X communication to thereby achieve safety of autonomous driving

A learning method of a CNN (Convolutional Neural Network) for monitoring one or more blind spots of a monitoring vehicle is provided. The learning method includes steps of: a learning device, if training data corresponding to output from a detector on the monitoring vehicle is inputted, instructing a cue information extracting layer to uses class information and location information on a monitored vehicle included in the training data, thereby outputting cue information on the monitored vehicle; instructing an FC layer for monitoring the blind spots to perform neural network operations by using the cue information, thereby outputting a result of determining whether the monitored vehicle is located on one of the blind spots; and instructing a loss layer to generate loss values by referring to the result and its corresponding GT, thereby learning parameters of the FC layer for monitoring the blind spots by backpropagating the loss values.

Learning method and testing method for monitoring blind spot of vehicle, and learning device and testing device using the same

A method for training an object detection network by using attention maps is provided. The method includes steps of: (a) an on-device learning device inputting the training images into a feature extraction network, inputting outputs of the feature extraction network into a attention network and a concatenation layer, and inputting outputs of the attention network into the concatenation layer; (b) the on-device learning device inputting outputs of the concatenation layer into an RPN and an ROI pooling layer, inputting outputs of the RPN into a binary convertor and the ROI pooling layer, and inputting outputs of the ROI pooling layer into a detection network and thus to output object detection data; and (c) the on-device learning device train at least one of the feature extraction network, the detection network, the RPN and the attention network through backpropagations using an object detection losses, an RPN losses, and a cross-entropy losses.

Learning method and learning device for training an object detection network by using attention maps and testing method and testing device using the same

Processes of explainable active learning, for an object detector, by using a Bayesian dual encoder is provided. The processes include: (a) inputting test images into the object detector to generate cropped images, resizing the test images and the cropped images, and inputting the resized images into a data encoder to output data codes; (b) (b1) one of (i) inputting the test images into the object detector, applying Bayesian output embedding and resizing the activation entropy maps and the cropped activation entropy maps, and (ii) inputting resized object images and applying the Bayesian output embedding and (b2) inputting the resized activation entropy maps into a model encoder to output model codes; and (c) (i) confirming reference data codes, selecting specific test images as rare samples, and updating the data codebook, and (ii) confirming reference model codes and selecting specific test images as hard samples.

Growing community of inventors

Gyeongsangbuk-do, South Korea

Yongjoong Kim