Deep compositional frameworks for human-like language acquisition in virtual environments

Abstract

Described herein are systems and methods for human-like language acquisition in a compositional framework to implement object recognition or navigation tasks. Embodiments include a method for a model to learn the input language in a grounded and compositional manner, such that after training the model is able to correctly execute zero-shot commands, which have either combination of words in the command never appeared before, and/or new object concepts learned from another task but never learned from navigation settings. In embodiments, a framework is trained end-to-end to learn simultaneously the visual representations of the environment, the syntax and semantics of the language, and outputs actions via an action module. In embodiments, the zero-shot learning capability of a framework results from its compositionality and modularity with parameter tying.