Tata Consultancy Services Limited

Pradip Pramanick

Chayan Sarkar

Brojeshwar Bhowmick

Snehasis Banerjee

Balamuralidhar Purushothaman

Hrishav Bakul Barua

Arpan Pal

Abhijan Bhattacharyya

Ashis Sau

Ruddra Dev Roychoudhury

Indrajit Bhattacharya

Ajay Kattepur

Soumyadip Maity

Sayan Paul

Dipanjan Das

Kritika Anand

Tapas Nayak

Ruchira Singh

Ruddra Roy Chowdhury

Avik Mitra

The disclosure generally relates to methods and systems for complex natural language task understanding for embodied robots or agents. Conventional works on relation extraction generally find relevant triplets in a natural language phrase, but neither ground the task nor ground the arguments. The present disclosure implements a Grounded Argument and Task Extraction (GATE) technique that extracts a set of tasks and relevant arguments from the complex natural language instruction. The GATE uses an encoder-decoder neural network with nested decoding technique. The extracted tasks are mapped (grounded) to the known skill set of the robot and arguments are mapped (grounded) to objects within the environment, classifies the tokens as many times as possible which existing sequence labeling cannot do. The encoder-decoder neural network of the present disclosure extracts grounded task-argument pairs from a natural language instruction in a generative mechanism, and grounds the arguments based on object detector vocabulary.

Methods and systems for complex natural language task understanding for embodied agents

Accuracy of transcript is of foremost importance in Automatic Speech Recognition (ASR). State of the art system mostly rely on spelling correction based contextual improvement in ASR, which is generally a static vocabulary based biasing approach. Embodiments of the present disclosure provide a method and system for visual context aware ASR. The method provides biasing using shallow fusion biasing approach with a modified beam search decoding technique, which introduces a non-greedy pruning strategy to allow biasing at the sub-word level. The biasing algorithm brings in the visual context of the robot to the speech recognizer based on a dynamic biasing vocabulary, improving the transcription accuracy. The dynamic biasing vocabulary, comprising objects in a current environment accompanied by their self and relational attributes, is generated using a bias prediction network that explicitly adds label to objects, which are detected and captioned via a state of the art dense image captioning network.

Method and system for visual context aware automatic speech recognition

This disclosure addresses the unresolved problems of tackling object disambiguation task for an embodied agent. The embodiments of present disclosure provide a method and system for disambiguation of referred objects for embodied agents. With a phrase-to-graph network disclosed in the system of the present disclosure, any natural language object description indicating the object disambiguation task can be converted into a semantic graph representation. This not only provides a formal representation of the referred object and object instances but also helps to find an ambiguity in disambiguating the referred object using a real-time multi-view aggregation algorithm. The real-time multi-view aggregation algorithm processes multiple observations from an environment and finds the unique instances of the referred object. The method of the present disclosure demonstrates significant improvement in qualifying ambiguity detection with accurate, context-specific information so that it is sufficient for a user to come up with a reply towards disambiguation.

Methods and systems for disambiguation of referred objects for embodied agents

The embodiments of present disclosure herein address unresolved problem of cognitive navigation strategies for a telepresence robotic system. This includes giving instruction remotely over network to go to a point in an indoor space, to go an area, to go to an object. Also, human robot interaction to give and understand interaction is not integrated in a common telepresence framework. The embodiments herein provide a telepresence robotic system empowered with a smart navigation which is based on in situ intelligent visual semantic mapping of the live scene captured by a robot. It further presents an edge-centric software architecture of a teledrive comprising a speech recognition based HRI, a navigation module and a real-time WebRTC based communication framework that holds the entire telepresence robotic system together. Additionally, the disclosure provides a robot independent API calls via device driver ROS, making the offering hardware independent and capable of running in any robot.

Telepresence robots having cognitive navigation capability

Existing cognitive robotic applications follow a practice of building specific applications for specific use cases. However, the knowledge of the world and the semantics are common for a robot for multiple tasks. In this disclosure, to enable usage of knowledge across multiple scenarios, a method and system for ontology guided indoor scene understanding for cognitive robotic tasks is described where in scenes are processed based on techniques filtered based on querying ontology with relevant objects in perceived scene to generate a semantically rich scene graph. Herein, an initially manually created ontology is updated and refined in online fashion using external knowledge-base, human robot interaction and perceived information. This knowledge helps in semantic navigation, aids in speech, and text based human robot interactions.

System and method for ontology guided indoor scene understanding for cognitive robotic tasks

Conventional tele-presence robots have their own limitations with respect to task execution, information processing and management. Embodiments of the present disclosure provide a tele-presence robot (TPR) that communicates with a master device associated with a user via an edge device for task execution wherein control command from the master device is parsed for determining instructions set and task type for execution. Based on this determination, the TPR queries for information across storage devices until a response is obtained enough to execute task. The task upon execution is validated with the master device and user. Knowledge acquired, during querying, task execution and validation of the executed task, is dynamically partitioned by the TPR across storage devices namely, on-board memory of the tele-present robot, an edge device, a cloud and a web interface respectively depending upon the task type, operating environment of the tele-presence robot, and other performance affecting parameters.

Knowledge partitioning for task execution by conversational tele-presence robots in a geographically separated environment

The disclosure herein relates to methods and systems for enabling human-robot interaction (HRI) to resolve task ambiguity. Conventional techniques that initiates continuous dialogue with the human to ask a suitable question based on the observed scene until resolving the ambiguity are limited. The present disclosure use the concept of Talk-to-Resolve (TTR) which initiates a continuous dialogue with the user based on visual uncertainty analysis and by asking a suitable question that convey the veracity of the problem to the user and seek guidance until all the ambiguities are resolved. The suitable question is formulated based on the scene understanding and the argument spans present in the natural language instruction. The present disclosure asks questions in a natural way that not only ensures that the user can understand the type of confusion, the robot is facing; but also ensures minimal and relevant questioning to resolve the ambiguities.

Methods and systems for enabling human-robot interaction to resolve task ambiguity

This disclosure provides systems and methods for robotic task planning when a complex task instruction is provided in natural language. Conventionally robotic task planning relies on a single task or multiple independent or serialized tasks in the task instruction. Alternatively, constraints on space of linguistic variations, ambiguity and complexity of the language may be imposed. In the present disclosure, firstly dependencies between multiple tasks are identified. The tasks are then ordered such that a dependent task is always scheduled for planning after a task it is dependent upon. Moreover, repeated tasks are masked. Thus, resolving task dependencies and ordering dependencies, a complex instruction with multiple interdependent tasks in natural language facilitates generation of a viable task execution plan. Systems and methods of the present disclosure finds application in human-robot interactions.

Robotic task planning for complex task instructions in natural language

This disclosure relates to system and method for enabling a robot to perceive and detect socially interacting groups. Various known systems have limited accuracy due to prevalent rule-driven methods. In case of few data-driven learning methods, they lack datasets with varied conditions of light, occlusion, and backgrounds. The disclosed method and system detect the formation of a social group of people, or, f-formation in real-time in a given scene. The system also detects outliers in the process, i.e., people who are visible but not part of the interacting group. This plays a key role in correct f-formation detection in a real-life crowded environment. Additionally, when a collocated robot plans to join the group it has to detect a pose for itself along with detecting the formation. Thus, the system provides the approach angle for the robot, which can help it to determine the final pose in a socially acceptable manner.

System and method for enabling robot to perceive and detect socially interacting groups

This disclosure relates generally to human-robot interaction (HRI) to enable a robot to execute tasks that are conveyed in a natural language. The state-of-the-art is unable to capture human intent, implicit assumptions and ambiguities present in the natural language to enable effective robotic task identification. The present disclosure provides accurate task identification using classifiers trained to understand linguistic and semantic variations. A mixed-initiative dialogue is employed to resolve ambiguities and address the dynamic nature of a typical conversation. In accordance with the present disclosure, the dialogues are minimal and directed to the goal to ensure human experience is not degraded. The method of the present disclosure is also implemented in a context sensitive manner to make the task identification effective.

Conversational systems and methods for robotic task identification using natural language

A method and a robotic system for online localized fatigue-state detection of a subject in a co-working environment using a non-intrusive approach is disclosed. A force sensor, mounted on the robotic system is capable of capturing effective force applied by local muscles of the subject co-working with the robotic system, providing a non-intrusive sensing. The captured force is analyzed on-line by the robotic systemto detect current fatigue state of the subject and proactively predict the future state of the subject. Thus, enables alerting the subject before time avoiding any possible accident.

Growing community of inventors

Kolkata, India

Pradip Pramanick