Friendliai Inc.

Samsung Electronics Co., Ltd.

Snu Rdb Foundation

Soojeong Kim

Byung-Gon Chun

Gyeongin Yu

Geon-Woo Kim

Joo Seong Jeong

Yoosang Hwang

Keunnam Kim

Hyosub Kim

Chulkwon Park

Seungbae Jeon

Manbok Kim

Kyunggeun Lee

An accelerator resource management method and apparatus are disclosed. The accelerator resource management method includes receiving a task request for a neural network-related task and a resource scheduling policy for the neural network-related task, obtaining information on a current resource utilization status of an accelerator cluster comprising a plurality of accelerators, in response to the task request, and allocating an accelerator resource for performing the task based on a utility of a resource allocation that is based on the resource scheduling policy and the information.

Accelerator resource management method and apparatus

Semiconductor devices may include an active pattern, a gate structure in an upper portion of the active pattern, a bit line structure on the active pattern, a lower spacer structure on a lower portion of a sidewall of the bit line structure, and an upper spacer structure on an upper portion of the sidewall of the bit line structure. The lower spacer structure includes first and second lower spacers sequentially stacked, the first lower spacer contacts the lower portion of the sidewall of the bit line structure and does not include nitrogen, and the second lower spacer includes a material different from the first lower spacer. A portion of the upper spacer structure contacting the upper portion of the sidewall of the bit line structure includes a material different from the first lower spacer.

Semiconductor devices

An inference system applies a machine-learning transformer model to a batch of requests with variable input length or variable target length or variable internal state length by selectively batching a subset of operations in the transformer model but processing requests in the batch individually for a subset of operations in the transformer model. In one embodiment, the operation to be processed individually is an attention operation of an encoder or a decoder of the transformer model. By selective batching, the inference system can allow batching operations to be performed for a batch of requests with variable input or target length or internal state length to utilize the parallel computation capabilities of hardware accelerators while preventing unnecessary computations that occur for workarounds that restrain the data of a batch of requests to a same length.

Selective batching for inference system for transformer-based generation tasks

An inference system applies a machine-learning transformer model to a batch of requests with variable input length or variable target length or variable internal sate length by selectively batching a subset of operations in the transformer model but processing requests in the batch individually for a subset of operations in the transformer model. In one embodiment, the operation to be processed individually is an attention operation of an encoder or a decoder of the transformer model. By selective batching, the inference system can allow batching operations to be performed for a batch of requests with variable input or target length or internal state length to utilize the parallel computation capabilities of hardware accelerators while preventing unnecessary computations that occur for workarounds that restrain the data of a batch of requests to a same length.

Dynamic batching for inference system for transformer-based generation tasks

Growing community of inventors

Seoul, South Korea

Soojeong Kim