Quote: Flaky Expression
Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear.
This quote and the blog post's finding, line up with a mental model of LLMs that I've found useful. I might go into this in a longer post someday, but there's an interesting correspondence between how LLMs appear to function and the philosophy of language developed by Ludwig Wittgenstein in Philosophical Investigations.
LLMs "understand" language statistically. Wittgenstein makes an argument that languages are games, and that to understand language is to share a context between the players of the game, to play the game as they do. This illuminates why LLM "knowledge" is so faulty — the model only encodes enough context to understand the language used, to be able to accurately play the game. They are general purpose, universal language machines. As long as the context of a language can be encoded into the model, the machine has a good chance of speaking the "language".
If you turn this on its side, the reason LLMs are so capable of writing code or responding in Pig Latin is because it's just shifting concepts from one language to another. It appears from observation that there is a cost to shifting and nesting languages. Embedding code inside JSON like in Model Context Protocol (MCP) produces more errors, and even deeper nesting (python inside of json inside of xml expressed as English sentences for example) will be incorrect much more frequently. Unpack and simplify the expression of a solution to your problem if you want LLMs to act "smarter", as it keeps them from tripping up over the complexity of tracking nested language rules.