Research highlight
LLM Program Search for Games
A benchmark study of using large language models to guide iterative program search across game-related tasks in Python and Java.
Why this project is interesting
Game-playing programs are difficult to write and tune across varied game rules. This project tests whether language models can help search for usable programs, and how task type and search strategy affect the result.
The problem
A single successful code-generation example does not show whether language models can reliably produce useful programs across different games and programming tasks. The work needed a broad, repeatable evaluation rather than a demonstration on one task.
Approach
The study used hill-climbing program search, with language models controlling initial programs and mutations. The benchmark covered 29 tasks and 11 models across Python and Java. The Python tasks included miniature Atari games, Baba Is You levels, an Asteroids-inspired environment and maze generation; the Java tasks used 12 games implemented in TAG.
My contribution
My role focused on designing the TAG-based benchmark, implementation of the LLM interaction with the existing Java codebase, automatic extraction of the TAG API and prompt engineering. I've also built a user interface for running the benchmark, visualising the results, inspecting the generated code and iterating on it with further human prompts to influence the LLM search.
Outcomes and evidence
- The paper reports that task choice affected performance more than model size in the evaluated benchmark.
- Multiple search runs with fewer iterations performed better than one run with more iterations in the reported experiments.
- The benchmark connects LLM-guided program search with both video-game tasks and tabletop games implemented in TAG.