Uber Technologies, Inc.

Microsoft Technology Licensing, LLC

Aaron Matthew Rogan

Taehun Yoon

Gregory Mark Mahowald

Benjamin James Kadlec

Wes Leung

Nicolas Garcia Belmonte

Ramik Sadana

Bradford R Clark

Kapil Gupta

Houtan Shirani-Mehr

Michael Harrell

Lucie Zikova

Alan Hugh Wells

Wei Hao

Systems and methods are disclosed herein for detecting non-compliant use of a micro-mobility vehicle, for example, based on parking the micro-mobility vehicle outside of a designated area. In an embodiment, a processor detects that a micro-mobility vehicle shared by a plurality of users has transitioned from an operational state to a parked state, where the micro-mobility vehicle was operated by a user of the plurality of users when in the operational state. The processor determines a location of the micro-mobility vehicle while the micro-mobility vehicle is in the parked state, and evaluates whether it is permissible to transition the micro-mobility vehicle to the parked state at the location. Responsive to determining that it is not permissible to transition to the parked state at the location, the processor updates a profile of the user to indicate non-compliant use of the micro-mobility vehicle.

Micro-mobility vehicle status compliance determination

Systems and methods are disclosed herein for monitoring a location of a client device associated with a transportation service and generating augmented reality images for display on the client device. The systems and methods use sensor data from the client device and a device localization process to monitor the location of the client device by comparing renderings of images captured by the client device to renderings of the vicinity of the pickup location. The systems and methods determine navigation instructions from the user's current location to the pickup location and select one or more augmented reality elements associated with the navigation instructions and/or landmarks along the route to the pickup location. The systems and methods instruct the client device to overlay the selected augmented reality elements on a video feed of the client device.

Generating augmented reality images for display on a mobile device based on ground truth image rendering

Systems and methods are disclosed herein for recovering inaccurate or missing base map data for an area while providing riders with incentive to refrain from selecting an alternate transportation service while experiencing increased wait times. A service receives a request for a driver and locates a driver to provide the service. The service determines whether the wait time for the rider exceeds a threshold wait time established by the service. If the service determines that the wait time exceeds the threshold wait time, the service identifies an object within a threshold vicinity of the rider known to have inaccurate or missing base map data. The service transmits a prompt for the rider to capture an image of the object. The service receives a rendering of the captured images from the client device, and activates a function via the client device if the images accurately capture the identified objects included in the prompt.

Location determination service based on user-sourced image updates

Systems and methods are disclosed herein for determining a location of a device, for example, by receiving, from a client device, a rendering of an image captured by the client device. The systems and methods compare the received rendering to entries in a database of renderings, each respective entry including a respective rendering and a respective associated location, and determine, from the comparing whether the received rendering matches a respective rendering included in a respective entry in the database of renderings. The systems and methods, in response to determining that the received rendering matches the respective rendering included in the respective entry, determine the location associated with the matching rendering, and transmit the location to the client device.

Locating a client device using ground truth image rendering

Systems and methods are disclosed herein for providing a model of a vehicle to a rider. In an embodiment, the systems and methods determine that a driver has accepted a task to operate a vehicle from a first location to a rider location. The systems and methods then access a model corresponding to the vehicle, determine a direction of movement of the vehicle, and determine a relative orientation of the vehicle with respect to the rider based on the first location, the direction of movement of the vehicle, and the rider location. The systems and methods generate for display, on a client device of the rider, the model at an angle determined based on the relative orientation.

Providing a model of a vehicle to a rider at an accurate orientation

Systems and methods are disclosed herein for determining placement of Augmented Reality (AR) content. In some embodiments, a processor detects input at a first client device of a first user that a transportation session between the first user and a second user has begun. In response to detecting the input, the processor determines a first location of the first client device based on an image captured by the first client device, and determines a geospatial offset of a second client device of the second user from the first client device. The processor determines a second location of the second client device based on the first location of the first client device and the geospatial offset, and generates for display by the second client device AR content, the AR content being selected based on an orientation of the second client device and the second location.

Rider-driver localization for determining placement of AR content for passenger

A network system, such as a transport management system, generates a mutual augmented reality (AR) experience for a user and a provider associated with a service. Responsive to receiving a service request, a service management module matches the user with an available provider and monitors the location of the user and provider client devices as the user and provider travel to the pickup location. When the devices are within a threshold distance of each other, an image recognition module monitors live video streams on the devices for the vehicle and the user. Responsive to the vehicle and user entering the field of view of the devices, an AR control module selects computer-generated AR elements and instructs the devices to visually augment the video streams to identify the user and provider to each other and to allow the user and provider to communicate and share data with each other.

Mutual augmented reality experience for users in a network system

In scenarios involving the capturing of an environment, it may be desirable to remove temporary objects (e.g., vehicles depicted in captured images of a street) in furtherance of individual privacy and/or an unobstructed rendering of the environment. However, techniques involving the evaluation of visual images to identify and remove objects may be imprecise, e.g., failing to identify and remove some objects while incorrectly omitting portions of the images that do not depict such objects. However, such capturing scenarios often involve capturing a lidar point cloud, which may identify the presence and shapes of objects with higher precision. The lidar data may also enable a movement classification of respective objects differentiating moving and stationary objects, which may facilitate an accurate removal of the objects from the rendering of the environment (e.g., identifying the object in a first image may guide the identification of the object in sequentially adjacent images).

Object removal using lidar-based classification

Within machine vision, object movement is often estimated by applying image evaluation techniques to visible light images, utilizing techniques such as perspective and parallax. However, the precision of such techniques may be limited due to visual distortions in the images, such as glare and shadows. Instead, lidar data may be available (e.g., for object avoidance in automated navigation), and may serve as a high-precision data source for such determinations. Respective lidar points of a lidar point cloud may be mapped to voxels of a three-dimensional voxel space, and voxel clusters may be identified as objects. The movement of the lidar points may be classified over time, and the respective objects may be classified as moving or stationary based on the classification of the lidar points associated with the object. This classification may yield precise results, because voxels in three-dimensional voxel space present clearly differentiable statuses when evaluated over time.

Lidar-based classification of object movement

Two-dimensional and three-dimensional data of a physical scene are combined and analyzed together to identify physical objects physically present in the physical scene. Image features obtained from the two-dimensional data and geometric features obtained from the three-dimensional data are combined with one another such that corresponding image features are associated with corresponding geometric features. Automated object detection mechanisms are directed to the combination of image and geographic features and consider them together in identifying physical objects from the physical scene. Such automated object detection mechanisms utilize machine learning such as selecting and tuning multiple classifiers, with each classifier identifying potential objects based on a specific set of image and geographic features, and further identifying and adjusting weighting factors to be applied to the results of such classifiers, with the weighted combination of the output of the multiple classifiers providing the resulting object identification.

Growing community of inventors

Westminster, CO, United States of America

Aaron Matthew Rogan