Amazon Technologies, Inc.

Ming Tan

Bing Xiang

Atul Deo

Andrew Oliver Arnold

Yuchen Tian

Ankur Deepak Desai

Zijian Wang

Praphruetpong Athiwaratkun

Mingyue Shang

Random token segmentation may be implemented for next token prediction. Text data may be received for training a machine learning model to predict a next token given input text tokens. Multiple tokens may be determined from the text data. Different ones of the multiple token may be randomly segmented in to sub-tokens. The machine learning model may then be trained using the multiple tokens including the respective sub-tokens as a training data set.

Random token segmentation for training next token prediction models

Growing community of inventors

Jersey City, NJ, United States of America

Ming Tan