Surface-based user input for extended reality

Abstract

Systems and methods in the present disclosure relate to surface-based user input for extended reality (XR) devices. An XR device tracks a hand of a user across multiple captured image frames to obtain positions of the hand in a real-world environment. The XR device detects an input plane associated with a physical surface in the real-world environment and projects the positions onto the input plane. The XR device continuously monitors an input state with respect to the input plane to identify when the user is providing user input via the input plane and to differentiate between ongoing user input and finalized user input. Based on the monitoring of the input state, the XR device records one or more of the projected positions as input data. The input data is processed to interpret the user input. The XR device executes an action based on the interpreted user input.