Reading out scene analysis for users wearing head-mounted devices

Abstract

In one embodiment, a method includes accessing sensor signals associated with a scene in a field of view of cameras of a head-mounted device associated with a first user from the head-mounted device, determining whether one or more criteria are satisfied for triggering a proactive analysis of the scene based on the sensor signals by machine-learning models, analyzing the scene to identify objects of interest to the first user in the scene based on the determination that the criteria are satisfied, generating a description of the objects based on an analysis of the objects and the sensor signals, wherein the description comprises relational information associated with one or more of the objects, and providing instructions to the head-mounted device to present an audio readout of the description to the first user.