Salesforce, Inc.

Junwen Bai

Caiming Xiong

Yingbo Zhou

Weiran Wang

Embodiments described herein combine both masked reconstruction and predictive coding. Specifically, unlike contrastive learning, the mutual information between past states and future states are directly estimated. The context information can also be directly captured via shifted masked reconstruction—unlike standard masked reconstruction, the target reconstructed observations are shifted slightly towards the future to incorporate more predictability. The estimated mutual information and shifted masked reconstruction loss can then be combined as the loss function to update the neural model.

Growing community of inventors

Ithaca, NY, United States of America

Junwen Bai