Phantom Auto Inc.

David Parunakian

Shay Magzimof

Michael Kris

Yuval Deri

Gleb Mezhanskiy

Elrom Silvera

Alexander Efitorov

Vladimir Roumenov Glavtchev

Joe Grundfest

A vehicle remote support system includes a communication system that operates over a plurality of parallel wireless network connections to provide low-latency video from vehicle to a remote support server that provides remote support to the vehicle dependent on real-time video. The vehicle includes a source that encodes multiple versions of the original video segments (e.g., one per wireless network connection) and transmits the multiple versions of the segments to a sink at the remote support server over the respective wireless connections. This redundant multi-path communication system rationally allocates network resources to the managed video streams and balances bandwidth against latency in order to avoid network congestion and safety issues associated with single-path transmissions. In other embodiments, a similar communication system that transmits video or other real-time messages between a source and a sink may be utilized in cloud robotics applications.

Low latency wireless communication system for teleoperated vehicle environments

A system provides on-demand remote support services including teleoperation of vehicles. In response to a remote support request, the system assesses environmental conditions, historical data, and/or parameters of a request for remote support to rank available operators capable of fulfilling the request, and selects an optimal operator for the remote support request based on desired criteria.

Vehicle teleoperator ranking and selection

A communication system facilities low-latency, high-availability multipath streaming between terminals (e.g., mobile terminals) and a server platform. In an example application, a remote support service operating on the server platform provides remote teleoperation, monitoring, or data processing services to a mobile terminal embodied as a vehicle or robot utilizing a low latency communication link. The low latency link enables a remote operator to receive video or telemetry feeds, and timely monitor and respond to hazards in substantially real-time. The low latency communication link may be achieved even when the data streams are transmitted over public networks incorporating at least one wireless leg, and where individual connections have varying quality of service in terms of delivery latency due to congestion or stochastic packet losses. Assignment of data streams to particular communication channels may be made on an optimization model derived from a machine-learning process or simulation.

Platform for redundant wireless communications optimization

A vehicle policy server maintains a set of policies for constraining operations of one or more remote vehicles. The policies may specify areas, locations, or routes that specified vehicles are restricted from accessing based on a set of acquired information. An application programming interface (API) enables programmatic updates of the policies or related information. Policies may be enforced by transmitting control signals fully or in part to onboard vehicle computers or to a teleoperation support module providing remote support to the vehicles using human teleoperators and/or artificial intelligence agents. The control signals may directly control the vehicles or teleoperation, or may cause a navigation system to present known restrictions in a suitable fashion such as generating an augmented reality display or mapping overlays.

Restricting areas available to autonomous and teleoperated vehicles

In a vehicle system that can receive remote support from a remote support server (e.g., interfacing with a human or computer teleoperator), a local normalization engine locally normalizes operation of the vehicle based on locally available sensor data that may not be accessible to the remote support server. The local normalization engine applies transformations to control commands received from the remote support server to transform the command to compensate for conditions that are locally sensed and may be unknown to the remote support server. Alternatively, or in addition, the local normalization engine controls auxiliary functions of the vehicle (e.g., by activating one or more auxiliary actuators) that may not be under direct control of the remote support server.

Normalization of intelligent transport system handling characteristics

A remote support system facilitates assignment of vehicles to remote support agents for providing teleoperation or other remote support services. The remote support system may generate assignments based on a mapping function that optimizes various parameters based on sensed data associated with the vehicle, a requested service mode of the vehicle, or other factors. In some situations, the remote support server assigns a redundant set of remote support agents to a vehicle that provide similar command streams. The vehicle selects between the command streams to minimize latency or another performance parameter. Alternatively, the remote support server assigns multiple diverse remote support agents to a vehicle that generate diverse command streams. A proxy agent then generates a consensus command stream for providing to the vehicle.

Mapping of intelligent transport systems to remote support agents

In a connected vehicle environment, network connection parameters such as a network congestion window and bit rate are automatically adjusted dependent on a location of a vehicle in order to optimize network performance. A geospatial database stores learned relationships between network performance of a connected vehicle at different physical locations when configured in accordance with different network parameters. The vehicle can then adjust its network parameters dynamically dependent on its location. A vehicle may maintain multiple connections to different networks concurrently for transmitting duplicate data of a data stream, with the vehicle independently adjusting parameters associated with different networks to optimize performance.

Optimizing wireless networking using a virtual geographic information system overlay

In a vehicle teleoperation session, a speed limit is determined for which the vehicle can be safely teleoperated. A safety system senses data relating to the vehicle environment and generates a depth map from which obstacles in the vicinity of the vehicle can be identified. Based on the detected obstacles and a motion state of the vehicle, a speed limit is determined at which the teleoperator is predicted to be able to utilize an emergency braking command to avoid a collision. The speed limit may be automatically applied to the vehicle or may be provided to the teleoperator to enable the teleoperator to adjust the vehicle speed.

Automatic imposition of vehicle speed restrictions depending on road situation analysis

A smart container yard includes systems for intelligently controlling operations of vehicles in the container yard using teleoperation and/or autonomous operations. A remote support server controls remote support sessions associated with vehicles in the container yard to provide teleoperation support for loading and unloading operations. Aerial drones may be utilized to maintain positions above a teleoperated vehicle and act as signal re-transmitters. An augmented reality view may be provided at a teleoperator workstation to enable a teleoperator to control vehicle operations in the smart container yard.

Teleoperation in a smart container yard

A vehicle routing evaluation system makes predictions about network performance data along road segments relevant to navigation of a vehicle and generates routing instructions or recommendations based in part on the network performance data. The vehicle routing evaluation system may send control signals to automatically control navigation of the vehicle to follow the routing generated routing or may present the navigation instructions to a teleoperator that controls the vehicle. Alternatively, the vehicle routing evaluation system generates a display with a navigational map overlay providing visual cues indicative of predicted wireless performance along various road segments to enable the teleoperator to select between possible routes.

Vehicle routing evaluation based on predicted network performance

A control platform generates commands for coordinating use of network resources between a plurality of vehicles within a geographic region. In an embodiment, game-theoretical modelling is employed to determine allocation of resources in a manner that provides an optimal solution for a given allocation strategy. This model may reward controllers of vehicles that comply with a coordination policy while penalizing controllers of vehicles that defect from compliance.

Platform for vehicle cooperation and coordination of services

A video encoder compresses video for real-time transmission to a video decoder of a remote teleoperator system that provides teleoperator support to the vehicle based on the real-time video. The video encoder recognizes one or more generic objects in captured video that can be removed from the video without affecting the ability of the teleoperator to control the vehicle. The video encoder removes regions of the video corresponding to the generic objects to compress the video, and generates a metadata stream encoding information about the removed objects. The video decoder generates replacement objects for the objects removed the compressed video. The video decoder inserts the rendered replacement objects into relevant regions of the compressed video to reconstruct the scene.

Growing community of inventors

Moscow, Russia

David Parunakian