Automatic Test Generation from Execution State Using Large Language Models.

Automatic Test Generation from Execution State Using Large Language Models.

Parshin M.A. (SPBU, St. Petersburg, Russia)
Stepanov D.S. (ITMO University, St. Petersburg, Russia)

Abstract

In software systems with a long lifecycle, a substantial number of high-level integration and end-to-end tests tend to accumulate over time, historically serving as the most pragmatic means of verifying functionality. A promising direction is the automatic generation of unit tests for individual components based on these existing tests (unit test carving), which improves the precision of defect localization and provides rapid feedback when changes are introduced. In this paper, we propose a method for automatic unit test generation based on program execution state. During the execution of an existing high-level test, the target object state, call arguments, return value, and interactions with dependencies are captured. From the collected data, an execution descriptor is constructed and subsequently used to synthesize a unit test: first, preliminary code is deterministically generated, after which a large language model performs its refactoring and stylistic adaptation. Unlike existing test carving tools, the method relies on a large language model to produce the final test code, and its implementation targets large JVM projects. We describe an implementation of the method for JVM programs built on top of the JDI debugging interface. The method is evaluated on 70 pairs of a test and a target method from seven open-source Java projects. We compare tests built by an AI agent without execution data, from the execution descriptor and by refactoring the preliminary code, as well as the preliminary code itself taken as the final test without refactoring. An AI agent with access to the data collected by the proposed method built unit tests that, on average, reproduce 86% of the target method calls from the original test with the same arguments and result, and 63% when the values of the target object's fields are also taken into account. For tests generated by the agent without access to execution data, these figures are 72% and 43%, respectively.

Keywords

automatic test generation; unit tests; large language models; Java Debug Interface; JVM.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 67-92

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(6)-26

For citation

Parshin M.A., Stepanov D.S. Automatic Test Generation from Execution State Using Large Language Models.. Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 67-92 DOI: 10.15514/ISPRAS-2026-38(6)-26.

Full text of the paper in pdf (in Russian) Back to the contents of the volume