LisanBench

Edit Type Distribution

Share of insertions, deletions, and substitutions in each model's chains.

Bridge Pattern Rate

How often models use the insert-substitute-delete bridge pattern.

Word Length Over Chain Steps

Longest-trial word length by step for each starting word.

Code & Details

More details, the benchmark implementation, and the source data are available on GitHub.

Why Word Chains?

Words as building blocks. Every language model is familiar with words. By making words the basic unit, evaluation becomes trivial: each answer is checked with simple string matching and a dictionary lookup. There is no LLM judge, no rubric-writing, and no human annotation.

Simple rules, emergent complexity. The problem is easy to state, but difficulty emerges naturally and can be adjusted by changing the starting words. The benchmark also scales effortlessly: you can add more starting words, more trials, or even switch languages without redesigning the task or changing how evaluation works. If you want to isolate reasoning more directly, you can also provide all valid next moves at each step and measure whether the model chooses well.

Why complexity emerges. As stronger models produce longer chains, the task gets harder on its own. They have to track a longer history, recall more accurately which words have already been used, and avoid painting themselves into a corner. Every chosen word removes one future node from the search space, so each move changes the structure of what remains.

Capabilities that matter in practice. Word chains probe a combination of abilities that real-world tasks depend on. To score well, models need to follow instructions, draw on broad and sometimes obscure knowledge, plan ahead strategically, remember what they have already done, and keep going until the task is actually finished:

• Rule Following — stay within the problem's constraints at every step
• Knowledge — know enough words to navigate sparse regions
• Planning — avoid short-sighted moves that lead to dead ends
• Recall — keep track of what has already been used
• Persistence — continue until no valid moves remain

These abilities compound rather than substitute for one another. Weakness in any one of them shows up quickly: models that stop early score poorly, models with weak recall repeat words, models without enough vocabulary get stuck in sparse regions, and models that know many words but plan poorly still fail to escape low-connectivity parts of the graph. That makes LisanBench difficult to game and better aligned with real-world usefulness than benchmarks that can be optimized around narrow tricks.

Methodology & Scoring

Each model is evaluated on the same 50 starting words, chosen to cover a range of difficulty. Some starting words, like hat, sit in dense parts of the graph with many valid continuations. Others, like countries, lie in much sparser regions, where valid moves are scarce and poor choices lead to dead ends more quickly. Each starting word is tested 3 times per model to reduce variance.

All chains are validated programmatically: each word must exist in the dictionary, each transition must be exactly edit-distance 1, and no repeats are allowed.