Can a MUD evaluate LLMs? A $99 proof of concept

**Researchers Explore MUD-Based Evaluation of Large Language Models**

In a novel experiment, a group of researchers has investigated the potential of using text-based games, also known as MUDs (Multi-User Dungeons), to evaluate the performance of Large Language Models (LLMs). By leveraging these early text-based games, the researchers aimed to assess the capabilities of LLMs in a unique and challenging environment. Utilizing personal computers and just $99 in API credits, the team conducted an extensive experiment, running their test on nights and weekends over several months.

The study's results revealed a fascinating leaderboard, but more surprisingly, the measurements of each LLM on four behavioral dimensions highlighted inconsistencies in the models' performance. When two of the dimensions, which heavily relied on an LLM classifier, were removed, one of the frontier models fell significantly in the rankings. Moreover, an analysis of the agreement between the classifier and a second judge showed a wide range of per-model agreement, from 85% to 22%. The aggregate kappa value of 0.04 for probe detection indicated that the instrument was noisy, but it did not pinpoint which models were affected. Notably, the most impacted model shared a model family with the classifier, raising questions about potential bias.

While the study's findings are intriguing, the researchers emphasize that this is merely a proof of concept and not a validated benchmark. The paper, data, and code are publicly available under open-source licenses, allowing others to review and build upon the work. The team is currently designing Phase 2 of the experiment, which aims to address limitations such as limited human raters, a small environment, and a single objective. By incorporating human baselines, multiple judges, and more objectives, the researchers hope to create a more robust evaluation framework for LLMs.