Testing benchmark generation method and testing benchmark generation system capable of adjusting prompt data

Abstract

A testing benchmark generation method includes providing a product, generating prompt data of the product by a prompt generation module, generating a code snippet by a large language model according to the prompt data, semantically checking the code snippet by an inference module for determining if the code snippet is successfully verified, compiling the code snippet and executing the code snippet to drive the product after the code snippet is successfully verified, analyzing an execution result corresponding to the code snippet after the code snippet is executed under the product, comparing the execution result corresponding to the code snippet with at least one constraint of the product by a profiling module for determining if the code snippet matches the at least one constraint, and adding the code snippet to testing benchmark data of a testing benchmark generation module.