Can My Puzzle Write-ups 'Lift' Cheap LLMs?

• Intro
• Takeaways
• Experiment
  • Harness
  • Did the models already know the answers?
  • Eval conditions
• Results
• Other tidbits from the solution runs
• Conclusion

Intro

I am interested in “lifting” cheaper models to higher tasks with the right guidance. I have written some puzzle solution blog posts recently. I wondered how much of my solution I would need to show a cheaper model in order for it to solve the puzzle.

I set up a simple testing environment using the Inspect eval harness and a Jupyter-style python env. I made sure the models didn’t have the answer memorized from their training data and then tested the models on the base puzzle and incrementally more of my solution write-ups. I recorded each model and setting’s pass/fail rate, token use, API cost, and turns to completion.

Takeaways

Some takeaways:

• Sonnet 5 didn’t need my lifting. It is a far stronger model than I thought. It solved all 3 puzzles without any of my notes at least once. I thought these would need beefier models to get a solution within my turn constraints!
• My “lifting” idea didn’t really work with Haiku 4.5. It helped on one of the three puzzles, but did not help on the other two. My guidance may transfer the solution idea, but the model may still not have the chops to implement e.g. backtracking search.
• Haiku 4.5 needed my write-up for only one of the puzzles. Otherwise, it either got the answer on its own or couldn’t solve the puzzle even with my full solution write-up.
• Sonnet costs about 3x more per attempt than Haiku, but it solved 85% of attempts while Haiku solved about a third of its attempts, even though Sonnet was only run on the puzzle settings that Haiku couldn’t solve every time.
• Extended thinking mode did nothing for Haiku and sometimes gave Sonnet “analysis paralysis” where it spent its full output limit thinking without running code

Experiment

Harness

I used the Inspect eval harness and a Jupyter-style python kernel. I tried four versions of the harness before I was happy with the results.

V0: I used Inspect’s built-in python() tool and only 30 tool calls.

V1: many attempts would hit the turn cap and not submit an answer, so I increased the tool budget to 50 calls and added a final turn that forces an answer if the model uses its whole tool budget.

V2: the models kept trying to import packages that weren’t available and assumed their python kernel state carried over between their calls, wasting turns on Python errors, so I updated the system prompt to list the installed packages, state that variables did not persist between python() calls, and tell the models how many turns they could take to explore and run code.

V3: the models still tried to reuse variables from previous tool calls and were also confused by OOM errors in their scripts, so I replaced the built-in python() tool with a Jupyter-style python kernel with a persistent environment between calls and bumped their containers up to 4 GB of memory from 2.

Did the models already know the answers?

This testing would be measuring nothing if the models have the solutions memorized. Two of the puzzles are recent enough that they should be past Sonnet’s knowledge cutoff, and all three for Haiku, but I wanted to test this anyway.

I first tested each model for contamination by asking it if it recognizes the puzzle by title, related puzzles in the same series, or the text of the write-up. The model could lie, but it’s still good to check this.

Neither model mentioned a correct answer in any reply or in its thinking.

Then, I tested each model for contamination by requesting a puzzle solution without allowing any turns with the python env. If the models had jumped right to a correct solution it would indicate training contamination. These were run without thinking mode.

*Haiku ignored my prompt to answer immediately, began solving, and hit its reply limit.

Both of these came back negative, so I proceeded to the actual experiment.

Eval conditions

The three puzzles and write-ups I used were Robot Baseball, ‘Pent-Up’ Frustration 3 / Knight Moves 7, and Andy’s Afternoon Amble.

I tested claude-haiku-4-5, claude-haiku-4-5 +thinking, and claude-sonnet-5 +thinking on each of the three puzzles.

I tested each model on the puzzle text + the first \(i\) sections of each write-up, where \(i\) ranges from 0 to the number of sections in the write-up. Let’s call these s0, s1, … s4

I tested each model x condition with 3 replicates. If a “weaker” model had solved a condition 3/3 times, then I did not re-run that condition with any stronger models.

I used the default temperature for these models, 1.0. I gave Haiku thinking an 8,000-token budget and set Sonnet’s thinking effort to “high”. I allowed 50 Python calls before I forced a final answer, and the models were informed about this limit in the system prompt.

I gave each Python tool call a 60s runtime limit and allowed 30 mins wall time max per puzzle attempt. I gave the models’ containers 4 GB memory, 2 CPUs, and no network access.

My grading looked for an exact answer for the Andy and Pent-Up puzzles, and allowed a 5e-11 tolerance for Robot Baseball.

Results

Correct answers out of 3 attempts per condition:

Andy

*The other two attempts used their whole token limits

Pent-Up

Robot Baseball

Token use and API cost:

Cost per correct answer, by puzzle:

Median Python calls in solved attempts (budget of 50):

Other tidbits from the solution runs

• Haiku hallucinated scipy.optimize.golden_section_search 11 times
• Sonnet solved Robot Baseball in a median of 5 Python calls vs Haiku’s 13–25
• Haiku frequently repeated the same mistakes between runs, including misunderstanding the original puzzle prompt

Can my puzzle write-ups “lift” cheap LLMs?

Not with Haiku 4.5 and Sonnet 5 didn’t need my help!

https://matthewritch.com/blog/2026/09/28/Puzzles-And-Lifted-Models/

Conclusion

I still think that given the right instruction, Haiku could also solve some of these puzzles. It’s possible that my write-ups are not effectively “teaching” concepts to the reader. I wrote them as I solved the puzzles, so they are a trace of my learning, but they skip over many of the dead ends I hit along the way and have been pruned to be tight reads. Maybe including more of that fluff would help Haiku? Or maybe it needs something else entirely. Maybe I’ll ask Sonnet to do it.