Less Is More (Maybe): Thinking Through Tiny Recursion Models

Key Takeaways

• Tiny Recursion Models (TRMs) represent a paradigm shift: fully learned recursive reasoning architectures that achieve ~45% on ARC-AGI-1 and ~8% on ARC-AGI-2 with a mere ~7M parameters. They challenge the dogma that performance scales strictly with parameter count.
• Aggressive Simplicity: TRM strips away the pretension. It is a single compact network repeatedly refining a latent “reasoning state” and a candidate answer, coupled with a learned halting head. There are no fixed-point theorems, no explicit multi-scale hierarchies, and no forced neurobiological metaphors.
• Empirical Efficiency: TRM outperforms its predecessor, the Hierarchical Reasoning Model (HRM), on Sudoku, mazes, and ARC-AGI, achieving this with fewer parameters and a significantly cleaner training/inference recipe.
• The Real Narrative: This isn’t a fairy tale of a tiny model magically beating trillion-parameter LLMs at their own game. It is a demonstration that carefully designed test-time training and recursive refinement can extract immense value from small networks when the domain is structured and data-poor.
• The Trajectory: Follow-up work, such as curriculum-guided schemes for recursion, suggests we are optimizing the pedagogy of these recursive models. This is not a curiosity; it is a viable, divergent research path.

We exist, currently, in the era of the Gigamodel. The prevailing wisdom suggests that intelligence is a function of weight – that if we simply stack enough transformer layers to blot out the sun, reasoning will emerge as a byproduct of compression. But this perspective risks conflating knowledge retrieval with reasoning. This post dissects Alexia Jolicoeur-Martineau’s “Less is More: Recursive Reasoning with Tiny Networks,” cross-referencing it with the broader context of HRM and ARC-AGI to ask: what happens when we stop building wider and start thinking deeper?

1. Background: Why recursive reasoning, why ARC-AGI, why now?

The current landscape is dominated by “sideways” reasoning. Large Language Models (LLMs) operate in token space, attempting to reason via chain-of-thought (CoT) and brute-force test-time compute tricks (self-consistency, best-of-N sampling). They are, in essence, probabilistic completion engines trying to simulate logic.

Their core failure mode on structured puzzles is cascading error. A single hallucinated token early in a Sudoku grid or an ARC pattern definition causes the entire deductive chain to collapse. CoT bandages this wound, but at a steep cost: latency, tokens, and capital, without curing the underlying brittleness.

A divergent design choice proposes a different path:

Instead of vomiting long verbal proofs, let a small network refine an internal state over several rounds, essentially “chewing” on the problem before decoding a single, clean answer.

This reimagines the “universal transformer” or Adaptive Computation Time (ACT) concepts, cast into a very small, task-specific setting: recurrent refinement over the same representation, governed by a learned halting signal.

The Hierarchical Reasoning Model (HRM) attempted this first, but wrapped it in a brain-inspired narrative:

• H (“high-level”) modules planning abstractly.
• L (“low-level”) modules handling details.
• A biological story about cortical layers.

On benchmarks like Sudoku-Extreme and ARC-AGI-1, a ~27M parameter HRM performed surprisingly well, beating many frontier LLMs. But subsequent analysis revealed an awkward truth for the “brain-inspired” crowd: the hierarchical architecture itself was largely decorative. The performance boost came from the outer refinement loop – the act of re-thinking – and heavy task augmentation.

HRM’s real insight wasn’t biology; it was engineering.

“Recursive refinement + clever training on the exact tasks you care about.”

TRM starts from that realization and strips the rest away.

2. What TRM actually is (and isn’t)

The stance of the paper is refreshing. It effectively says: stop over-engineering the justification.

Contrarily to HRM, TRM requires no fixed-point theorem, no complex biological justifications, and no hierarchy.

Stripped of academic marketing, a Tiny Recursion Model (TRM) is:

• A single small network .
• A latent “reasoning” state and a current answer .
• A question embedding (e.g., an ARC input grid).
• A halting head that estimates probability of completion.

The process is algorithmic:

1. Initialize (often trivial embeddings).
2. For a number of deep refinement steps :
   • Run several inner latent updates (recursive reasoning):
   • After inner steps, update the answer:
   • Compute:
     • a discrete prediction
     • a halting probability
   • If is high, the system decides it is finished.

And crucially, all of this uses the same tiny network, trained end-to-end with “deep supervision.” It learns to think.

A compressed pseudocode-ish version (my own interpretation):

import torch
import torch.nn as nn
from torch import Tensor


class TinyRecursiveCore(nn.Module):
    """A minimal MLP that fuses question, answer, and latent state."""

    def __init__(self, dim: int) -> None:
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(dim * 3, dim),
            nn.GELU(),
            nn.Linear(dim, dim),
        )

    def forward(self, x: Tensor, y: Tensor, z: Tensor) -> Tensor:
        # x, y, z: [B, D]
        concat = torch.cat([x, y, z], dim=-1)
        return self.mlp(concat)


class TinyRecursionModel(nn.Module):
    """Simplified TRM: recursive refinement with deep supervision."""

    def __init__(self, dim: int, num_tokens: int) -> None:
        super().__init__()
        self.input_emb = nn.Embedding(num_tokens, dim)
        self.answer_emb = nn.Embedding(num_tokens, dim)
        self.core = TinyRecursiveCore(dim)
        self.answer_head = nn.Linear(dim, num_tokens)
        self.halt_head = nn.Linear(dim, 1)

    def latent_recursion(
        self, x: Tensor, y: Tensor, z: Tensor, inner_steps: int
    ) -> tuple[Tensor, Tensor]:
        """Run inner latent refinement loop."""
        for _ in range(inner_steps):
            z = self.core(x, y, z)  # update latent "reasoning" state
            y = self.core(y, z, y)  # refine answer using latent state
        return y, z

    def forward(
        self,
        x_tokens: Tensor,
        y_init_tokens: Tensor,
        inner_steps: int = 6,
        deep_steps: int = 16,
    ) -> tuple[Tensor, Tensor]:
        # x_tokens: [B, Lq], y_init_tokens: [B, La]
        x = self.input_emb(x_tokens).mean(dim=1)
        y = self.answer_emb(y_init_tokens).mean(dim=1)
        z = torch.zeros_like(y)

        logits_per_step: list[Tensor] = []
        halt_logits_per_step: list[Tensor] = []

        for _ in range(deep_steps):
            y, z = self.latent_recursion(x, y, z, inner_steps)
            logits_per_step.append(self.answer_head(y))
            halt_logits_per_step.append(self.halt_head(y))

        # Training loss is computed over all steps ("deep supervision")
        return torch.stack(logits_per_step), torch.stack(halt_logits_per_step)

If this feels like a “universal transformer” applied to small MLPs and grid puzzles, your pattern matching is functioning correctly.

A compact mental model

We can reconstruct the core idea as a flowchart of intent:

The structural bias here is singular and profound: “I will keep re-thinking the same puzzle internally before I commit.” No explicit programs, no search trees, just recurrent refinement in latent space.

3. TRM vs HRM: What actually changed?

Let’s strip the paint and look at the structural differences.

3.1 Architecture & training

Under the hood, both share the same essence: a recurrent refinement loop and a halting head. HRM tried to justify a specific H/L decomposition using fixed-point theorems; TRM bluntly asserts that the outer loop and latent recursion do the heavy lifting.

3.2 Performance on structured puzzles

On Sudoku-Extreme, the difference is stark:

TRM achieves higher accuracy with fewer parameters, proving that architectural clarity often beats complexity.

3.3 ARC-AGI numbers and the “tiny model beats trillion-param” meme

The numbers for ARC-AGI-1 and ARC-AGI-2 (pass@2) tell a compelling story:

Yes, you can correctly claim:

A 7M parameter TRM trained from scratch beats many giant LLMs on ARC-AGI-1 and ARC-AGI-2.

But – and here the rationalist alarm bells should ring – the training regimes are fundamentally different. TRM is trained directly on the ARC tasks, acting as a transductive system. Many LLM results are closer to “few-shot ARC prompting.”

From a systems perspective, TRM looks less like “tiny AGI” and more like a highly specialized neural program synthesis substrate. That is still remarkable, but it is a distinct category from general-purpose token prediction.

4. Training dynamics: Deep supervision and test-time recursion

The magic isn’t just in the architecture; it’s in the pedagogy.

4.1 Deep supervision over refinement steps

TRM applies loss at every refinement step, not just the end:

This forces the network to attempt correctness immediately and continuously. It aligns the halting head with the concept of “done” rather than “tired.”

4.2 Recursion at training time matters more than at inference

The data suggests a counter-intuitive reality: training with deep recursion creates a robust model that can often infer with fewer steps. It’s akin to training with heavy weights to make the race feel easy.

Follow-up work like curriculum-guided adaptive recursion makes this explicit: gradually increasing recursion depth during training allows for massive efficiency gains.

4.3 Task augmentation as implicit program search

The dataset for ARC is aggressively augmented (rotations, flips, color permutations). The result is that the model is effectively forced to learn the rule rather than the bit pattern. It is neural program synthesis in disguise.

5. A more formal view: What kind of computation is TRM implementing?

Viewed through a formal lens, TRM is a parametric recurrent dynamical system:

1. Embed the puzzle:
2. Initialize answer and latent:
3. For (deep refinement):
   • For (inner recursion):
   • Answer update:
   • Halt score:
4. Decode answer:

It is a differentiable variant of “search until solution found,” where “search” is latent refinement.

The crucial metric is effective depth. For Sudoku-Extreme, TRM reports effective depths of 40–50. A 7M parameter model is being given significant compute time per token. The surprise is not that it solves puzzles; the surprise is the efficiency of the substrate.

6. How to play with TRM in practice

For the builders, the barrier to entry is low. The architecture is clean. A minimal “toy” usage in PyTorch might look like this:

import torch
from tiny_recursive_model import TinyRecursiveModel, MLPMixer1D

# Tiny model config – *not* the exact ARC config, just a small demo
trm = TinyRecursiveModel(
    dim=32,
    num_tokens=256,
    network=MLPMixer1D(
        dim=32,
        depth=2,
        seq_len=64,   # length of flattened puzzle or answer
    ),
)

# Dummy puzzle batch: input & initial answer tokens
B, L = 8, 64
x_tokens     = torch.randint(0, 256, (B, L))   # "question" grid
y_init       = torch.zeros(B, L, dtype=torch.long)  # e.g. blank answer
y_target     = torch.randint(0, 256, (B, L))   # pretend ground truth

# Forward pass with recursive refinement
pred_logits_over_steps, halt_logits_over_steps = trm(
    x_tokens,
    y_init,
    max_deep_refinement_steps=12,
)

# Cross-entropy at all steps (deep supervision)
logits = pred_logits_over_steps      # [T, B, L, vocab]
T = logits.size(0)

ce_loss = 0.0
for t in range(T):
    ce_loss = ce_loss + torch.nn.functional.cross_entropy(
        logits[t].view(-1, logits.size(-1)),
        y_target.view(-1),
    )

ce_loss = ce_loss / T
ce_loss.backward()

TRM as a tool for LLMs

A natural engineering pattern emerges: Use an LLM as the parser/orchestrator, handing off structured problems to a domain-specific TRM. The TRM becomes the “System 2” plugin for the “System 1” LLM.

7. How “real” is the generalization?

We must address the inevitable skepticism. Is this general reasoning or just sophisticated memorization over a cleverly augmented dataset?

The analysis suggests TRM is transductive. It learns to solve the tasks it is shown (and their variants) exceptionally well. It does not necessarily possess the broad, fluid intelligence of a frontier LLM.

However, TRM is strong evidence that recursive refinement with small networks is a powerful pattern in structured domains. It is a proof-of-concept for efficient, depth-first reasoning.

8. Relationship to other “tiny but deep” models

TRM sits at the intersection of critical trends:

• Tiny models with clever structure: Like FastGRNN or TinyLlama, proving that inductive bias matters.
• Universal transformers / ACT: The idea of trading depth for flexibility is recurrent (pun intended).
• Neural program synthesis: Using gradient descent to synthesize logic.

TRM’s contribution is to strip away the noise. It shows that a single small network, recursing on its own latent state, is enough to crack difficult benchmarks. It is the “minimum viable recursive reasoner.”

9. Open questions and speculative directions

Where do we go from here? A few questions I find myself circling:

1. Beyond grids
   ARC, Sudoku, and maze tasks are all essentially grids with local rules.
   How well does a TRM-style architecture extend to:
   • Textual reasoning (math word problems, multi-hop QA)?
   • Symbolic domains (graphs, programs)?
   • Continuous control?
2. Integrating induction and transduction
   ARC Prize’s winning entries make clear that combining program synthesis (induction) with transduction works very well.
   Can we:
   • Use TRM as a learned proposal generator for symbolic program search?
   • Or train TRM to emulate a small internal DSL, making its latent recursion more interpretable?
3. Task-general TRMs
   Right now, TRM is mostly “train a separate model per puzzle family (ARC vs Sudoku vs Maze)”.
   Is there a way to:
   • Meta-train a TRM over a variety of domains/tasks so that new tasks only need a small amount of adaptation?
   • Use an LLM to configure / prompt a general TRM instead of retraining?
4. Compute vs parameters tradeoff
   TRM is tiny in parameters but not cheap in training compute: ARC runs assume multiple modern GPUs for days.
   Follow-ups like curriculum-guided recursion improve the situation, but we still have to ask:
   • For a given budget, when is “small but deeply recursive” better than “medium but shallow”?
   • How do these results scale if we give TRM 70M parameters instead of 7M?
5. Safety and interpretability
   Recursive latent reasoning is opaque. If we want to trust these models on safety-critical tasks:
   • Can we visualize trajectories in -space to detect failure modes?
   • Can we extract approximate rules (e.g. via probing or distillation into explicit programs)?

10. The Verdict

If I had to synthesize a view on TRM:

• Recursive refinement is a non-negotiable ingredient of future reasoning systems. HRM and TRM demonstrate that “big LLM magic” is often just a crude approximation of what a small network can do with a proper loop.
• The narrative isn’t David vs. Goliath. It is that David brought a specialized tool to a problem Goliath was trying to solve by sitting on it.
• ARC remains a weird, beautiful laboratory. It is doing its job by forcing us to look at inductive biases.

For the practitioner, the lesson is clear: When the domain is structured and data is scarce, do not reach for the biggest hammer. Reach for the sharpest recursive loop.

And if you’re a researcher (or an applied ML engineer with research envy), TRM is a genuinely nice playground:

• Clear code.
• Non-trivial benchmarks.
• Room for architectural creativity (graph variants, hybrid symbolic loops, curriculum tricks).

I suspect the next couple of years will see a lot of “TRM-but-for-X” work. Some of it will be hype, some will quietly advance the frontier. In that sense, Less is More is aptly titled: not because smaller is always better, but because once you strip away the unnecessary parts of an architecture, what’s left is a clearer lens on what really matters.