Amazon Technologies, Inc.

Praphruetpong Athiwaratkun

Sudipta Sengupta

Bing Xiang

Ramesh M Nallapati

Parminder Bhatia

Yuchen Tian

Srinivas Iragavarapu

Rama Krishna Sandeep Pokkunuri

Atul Deo

Andrew Oliver Arnold

Matthew Lee

Yanitsa Donchev

Ankur Deepak Desai

Zijian Wang

Vamshidhar Krishnamurthy Dantu

Sujan Kumar Gonugondla

Sri Ranga Akhilesh Bontala

Jiacheng Guo

Ramana Keerthi

Zixuan Lin

Mingyue Shang

Haifeng Qian

Xiaokai Wei

Bohan Yao

Hantian Ding

Techniques for using a quantized and/or fused model are described. In some examples, a service is to receive a request to use a trained model, the request including input data; apply a trained model to the input data, the application of the trained model includes fusing one or more matrix multiplication operations with element-wise operations; and output a result from the trained model.

Large language models utilizing element-wise operation-fusion

Evaluation data sets may be programmatically generated for code generation models. An evaluation data set is obtained that includes items that correspond to different evaluation tests for a code generation system. The individual items of the evaluation data set maybe converted, including the conversion of a function signature for the items, the test statements for the items and using a code generation system to generate the body of the function.

Programmatically generating evaluation data sets for code generation models

Pre-fix matching may constrain the generation of next token predictions. Input text to perform a next token prediction may be received. Multiple tokens may be determined from the input text, including a partial token. From possible tokens, one or more matching possible tokens with the partial token may be identified. Next token predictions may then be filtered using the identified possible tokens in order to ensure that the partial token is matched.

Constrained prefix matching for generating next token predictions

Growing community of inventors

Jersey City, NJ, United States of America

Praphruetpong Athiwaratkun