· Aaron Darr · engineering · 6 min read
Looped Transformers: How NVIDIA's Dory Thinks Harder Without Getting Bigger
NVIDIA's Dory1.0-4B reruns one block of layers for each token. You choose how many times at deploy time, and the weights in memory never change.
Most of the ways we make a language model smarter cost something. A bigger model needs more GPU memory. Longer chains of thought take more time and more tokens. NVIDIA’s Nemotron Labs has released a research model, Dory1.0-4B, that takes a different approach. It runs part of itself more than once for every token.
Dory is a looped (or latent-recurrent) transformer. It has 4 billion parameters, so it fits on ordinary hardware. And you choose how hard it thinks when you deploy it.
What Dory is
- Size: 4B parameters
- Architecture: a looped normalized-GPT (nGPT), with sliding-window attention layers interleaved with full-attention layers
- Context: up to 256K tokens
- License: OpenMDW 1.1, for commercial or non-commercial use
- Formats: BF16, MXFP8, NVFP4 and a Q5_K_M GGUF for llama.cpp
- Training: 6 trillion tokens of pre-training, with the first ~4.5 trillion natively in NVFP4 and the rest in BF16. Then 300 billion tokens of context extension, supervised fine-tuning, and multi-environment reinforcement learning with GRPO.
The weights and model cards are in NVIDIA’s Nemotron Labs Dory collection on Hugging Face.
How the loop works
A normal transformer runs each layer once, in order. Dory splits its 36 layers into three groups:
- An input block with 8 layers, which runs once.
- A recurrent block with 20 layers, which loops.
- An output block with 8 layers, which runs once.
Each token goes around the recurrent block k times. vLLM and llama.cpp support 1 to 4 loops. On each loop, the model refines its internal picture of the token.
One detail makes this work. The input is reinjected into every loop. This is a bit like the skip connections in a ResNet, but Dory uses the same weights on every pass. Each loop starts with a reminder of the original input, like rereading the question before trying again. This keeps the representation from drifting over many loops.
The same checkpoint works at every loop count. You do not retrain anything. You pick the loop count when you start the server.
A deeper model with the same weights
Every extra loop sends each token through 20 more layers. The weights in memory stay the same.
| Loops | Layers per token (8 + 20 × k + 8) |
|---|---|
| 1 | 36 |
| 2 | 56 |
| 4 | 96 |
So at 4 loops, a token passes through almost three times as many layers as at 1 loop, with no extra weights. For anyone who is limited by GPU memory, this changes the usual trade-off. You get depth from compute, not from VRAM.
NVIDIA also reports that looping is token-efficient. As the loop count goes up, the model solves the same tasks with fewer generated tokens. The extra work goes into thinking inside the model, not into longer answers.
The catch: the KV cache
Memory still grows in one place: the KV cache. Dory has two cache modes:
- Per-loop: each loop keeps its own cache, so cache memory grows with the loop count.
- Last-loop: each recurrent attention layer keeps one cache, and each loop overwrites the last. The cache stays at the one-loop size, whatever the loop count.
NVIDIA says accuracy with last-loop caching stays close to per-loop mode, with output lengths almost unchanged. If you serve long contexts or many users at the same time, try last-loop mode first. In vLLM, set recurrent_kv_cache_mode to "last_loop".
More loops, better scores
NVIDIA publishes an internal composite estimate (Artificial Analysis v4.3.2) for each loop count:
Two things stand out:
- The second loop does most of the work. The score jumps from 13.8 to 17.7. Going to 4 loops adds only 0.7 more.
- At 2 loops, Dory already passes Gemma 4 26B A4B (16.35). That is a mixture-of-experts model with about 4B active parameters per token, but far more total weights.
Where Dory wins and where it loses
NVIDIA’s model card compares Dory with Gemma 4 26B A4B on ten benchmarks. Dory wins 1 of 10 at 1 loop, 4 of 10 at 2 loops, and 5 of 10 at 4 loops. Here is the 4-loop result:
| Benchmark | Category | Dory (4 loops) | Gemma 4 26B A4B |
|---|---|---|---|
| SWE-bench Verified | Coding & agentic | 61.8 | 57.4 |
| τ³-bench (Banking) | Coding & agentic | 14.69 | 13.66 |
| GDPval-AA-V2 (Elo) | Coding & agentic | 967 | 751.6 |
| IFBench | Instruction following | 79 | 77.89 |
| AA-LCR | Long context | 73 | 64.6 |
| SciCode | Coding & agentic | 35.11 | 40.32 |
| Terminal-Bench 2.1 | Coding & agentic | 22.2 | 35 |
| GPQA Diamond (no tools) | Reasoning | 72.16 | 80.43 |
| HLE (text-only, no tools) | Reasoning | 10.43 | 17.79 |
| AA-Omniscience (accuracy) | General knowledge | 10.03 | 21.13 |
The pattern tells you what looping is good for:
- Dory wins on agentic tasks, instruction following and long context.
- Dory loses most on knowledge-heavy tests: AA-Omniscience, HLE and GPQA Diamond. Its AA-Omniscience score stays at about 10 whatever the loop count.
This makes sense. Looping gives a model more time to think, but not more room to store facts. Facts live in the weights, and Gemma has far more of them.
Deploying Dory today
Dory runs in vLLM and, through a Dory-enabled build, in llama.cpp. In both, you set the loop count when you start the server (n_recurrent_loops in vLLM’s --hf-overrides, or dory.num_loops in llama.cpp). To offer more than one loop count, you run more than one instance.
NVIDIA’s advice, and a sensible setup:
- Use 2 loops as the default. It gives most of the gain.
- Use 1 loop when cost per token matters most.
- Use 4 loops when accuracy matters more, especially for agentic work like SWE-bench.
- Use last-loop KV caching for long contexts or high concurrency.
- Measure on your own workload. More loops cost more per token, but they can also mean fewer tokens, so check total latency and cost.
If you run more than one loop count, put a router in front, so hard requests go to the deeper instance.
Why this matters
Dory is a research release, and it will not replace production models yet. Its value is the idea. Most reasoning models today get better answers by writing longer chains of thought. A looped model thinks harder per token instead of writing more tokens, and you decide how hard at deploy time.
Dory is only 4B parameters, and at 2 loops it already beats a 26B mixture-of-experts model on NVIDIA’s composite score. If the idea scales to larger models, looped transformers could become a serious option for teams that need strong reasoning on hardware they already own.
How we can help
If you’re thinking about running an open model on your own hardware, or weighing one up against a hosted API, we can test both on your workload and compare accuracy and cost per request side by side.
Our AI development service covers model selection and the serving stack. Our AI agents and chatbots service covers the agents and tools built on top of it.
Get in touch with what you want the model to do and the hardware you have, and we’ll tell you what’s realistic.
Source: NVIDIA-Nemotron-Labs-Dory1.0-4B model card. All benchmark numbers were measured by NVIDIA. The composite scores are NVIDIA’s internal estimates and may differ from official Artificial Analysis scores.
