AREX-2 Tested: What Happens When the Agent Loop Lives in the Model

If you have built an agent in the last year, you have almost certainly written this bit of code. The model makes an attempt. You run it and check the result. You decide whether to retry. If you retry, you keep the best version so far, and at some point you decide to stop.

All of that logic lives in the harness. The model gives you one attempt at a time. Your code does everything else.

So when the Beijing Academy of Artificial Intelligence (BAAI) released AREX-2, a model trained to run that loop by itself, we wanted to understand it properly. Give it a budget, say five hours on a problem, and it works out how to spend those hours making its answer better.

We read the paper, went through the model card and looked at the evaluation code. Then we ran it ourselves. Here is what we found, and what we think it means for the harnesses we are all building.

What AREX-2 is

The basics first:

• 27B parameters, dense (every parameter is used for every token), fine-tuned from Qwen3.8-27B.
• Open weights under Apache 2.0, so you can use it commercially.
• 262K-token context window. That matters here, because the whole method depends on the model remembering a long session.
• It comes from BAAI, the lab behind the BGE embedding models and the earlier AREX deep-research models.

The headline claim is that a 27B fine-tune scores 81.8 on MLE-bench Lite, an ML engineering benchmark. In BAAI’s own table, that is ahead of GPT-5.6 Sol at 72.7.

We think the model is less interesting than the idea behind it, though. So we start there.

The core idea: reflection times horizon

The paper’s framing is simple. For a model to keep getting better over many rounds, it needs two separate abilities.

Reflection. The model looks at its current situation and the feedback it just got, and asks: can I make a better attempt? This decides how much each round is worth.

Long-horizon execution. Can it keep doing that for hours? That means remembering what it already tried, and recovering when something goes backwards. This decides how many rounds stay useful before the model stalls and loses its way.

The two multiply. If reflection is weak, more rounds just burn tokens. If the horizon is short, even sharp reflection plateaus.

The part we found most useful is that this split works on any agent, even if you never fine-tune a model. When your agent stalls, ask which half broke.

If it makes bad changes, the problem is reflection. Usually that means a weak feedback signal: a vague pass or fail when it needs a real score, full logs and the exact error.

If it loses the thread and redoes things it already tried, the problem is the horizon. That is a state and memory problem. Look at what drops out of context between rounds.

From outside, both look like “the agent stopped improving”. They need different fixes.

One more piece: AREX-2 is told its budget up front, both the number of rounds and the wall-clock time. It is trained to pace itself, using early rounds to measure and later rounds to act on what it measured.

How you train a model to improve

This is where the paper gets clever.

Most agent training data is made like this. You pose a task, the model has a go, a verifier says pass or fail, and you keep the passes. That shows the model what a correct answer looks like. It shows nothing about how an answer gets better. The failed attempts, the feedback and the fixes all get thrown away.

And the most useful runs for learning improvement are the long, messy ones that start badly and end well. Those are usually the first to go.

So BAAI built its own data. They picked two fields where the feedback is clear:

• Machine learning engineering. A validation score goes up or it doesn’t.
• Algorithmic programming. Tests pass or they don’t.

Both also leave a lot of room to keep climbing, from a basic baseline to a tuned ensemble, or from brute force to a near-optimal program.

The pipeline has three steps:

1. Turn repos into environments. A teacher model takes GitHub repositories and online-judge problems and turns each one into a task with a scoring function and a sandbox. An environment is kept only if the reference solution runs and a simple baseline scores well below it, so there is room to improve.
2. Let a teacher agent work for hours. For each task, the teacher reads the code, searches papers, runs small experiments and submits. It does this over hundreds of tool calls, knows its budget, and gets feedback after every submission.
3. Keep whole runs, failures included. A run is kept based on how it finished, not judged step by step. The failed rounds stay in context. But the training loss (the signal the model learns from) only applies to the steps that moved things forward: diagnosing a failure, repairing it, changing strategy, running an experiment, submitting an improvement. Repeated polling, empty tool calls and near-duplicate turns are not trained on.

That last step is the clever bit. The model sees the mess, and it learns how to respond to it.

One thing to be clear about: this is not reinforcement learning. It is fine-tuning on these example runs, mixed with deep-research data from the earlier AREX models. The paper does not name the teacher model. As far as we can tell, the training data and training code are not released. The GitHub repo has evaluation code only.

If you fine-tune on your own agent traces, you can reuse this recipe directly. Pick runs by how they end, not step by step. Keep the failures in context. Mask the loss so you only train on recovery and progress.

Does it work?

On the two coding benchmarks, the model card reports:

• MLE-bench Lite: 81.8 (Any Medal, mean of three seeds, 12 hours per task). That is the top score in the table, ahead of GPT-5.6 Sol (72.7) and every open model listed, including Kimi-K3 at 2.8T parameters.
• Frontier-CS: 70.7 on the 188-task Agent Track. That is the best open-weights score, but about six points behind GPT-5.6 Sol (76.4).

Keep in mind that this is a 27B model next to systems that are almost certainly much larger.

The chart we found most telling is score over time on Frontier-CS. AREX-2 does not start out in front. It reaches 54.4 after one hour, 65.9 after two, and 70.7 after five. DeepSeek-V4-Pro levels off at 44.7 after two hours and does not move after that.

That ability to keep improving with more time is what everyone is chasing.

The long horizon also carries over to research tasks. AREX-2 beats the earlier AREX models on all four deep-research benchmarks in the paper, including the 122B AREX model. It scores 84.0 on BrowseComp, 52.6 on HLE, 92.2 on GAIA and 93.8 on DeepSearchQA.

A caution: every number here comes from BAAI, run with BAAI’s own harness. When we searched, we found no independent evaluation yet. Treat these as claims to check.

Skills vs. training: what actually moved the number

This is the part we would point anyone to. The MLE-bench Lite score is not the model alone. It was run with skills in context. Skills are written notes on how to do the work: general ML practice, plus task-specific notes built by researching each problem.

The paper includes an ablation, which is a test that removes one piece at a time to see what each piece is worth. We were glad to see it.

The base model, with no training at all, goes from 28.8 to 68.2 on skills and more rounds. Swap in AREX-2 with the same skills and the same rounds, and you get 81.8.

So skills drive most of the gain. But the training adds about another 14 points on top, which is real. It is just a smaller lever than good skills. If you have built agents with well-written skills, that probably does not surprise you.

What this means for your harness

If more models are trained like this, do harnesses go away? We don’t think so. But their job changes.

The harness will need to worry less about controlling the loop: when to retry, what to keep, when to stop. Its job moves to three things:

1. Give the model a good feedback signal. Real scores, real logs, real errors. Reflection is only as good as what the model is shown.
2. Give it working knowledge. That is the skills part, and the ablation shows how much it is worth.
3. Set the budget. Tell the model how long it has, so it can plan its long run.

We expect friction on the way. If your framework has its own reflect-and-retry nodes, and the model is trained to pace itself over one long session, you now have two loops that may fight each other. And a harness that resets context every few steps, to save tokens, could make a model like this worse, because it cuts off the long memory the model was trained to use.

Is this really self-improvement?

BAAI frames the work as a step toward recursive self-improvement. That needs a careful reading.

What the paper shows is self-improvement at test time, inside one task. The model gets better at the problem over many rounds. The weights do not change.

Think of a chess player who works out a strong opening over a long night of analysis, then wakes up the next day with no memory of it. They are good at analysing. They do not get better at chess.

True self-improvement would feed the model’s own runs back in as its next training data, so it needs fewer attempts next time. To be fair to BAAI, the paper says exactly this. It names that as the path forward.

How to test it yourself

The benchmark numbers come from BAAI’s setup. What matters is whether the behaviour carries over to your work. We ran this test on AREX-2 ourselves. Here is how to build it:

1. Pick a task with a numeric score where you can see steady progress, such as a route-finding or packing heuristic, a slow function to speed up, or a small tabular model.
2. Write a bare loop. Each round, the model submits a full solution. You run it in a sandbox and send back the score, the logs and the number of rounds left.
3. Run AREX-2 and base Qwen3.8-27B with the same prompt, budget and settings. Plot best-score-so-far against the round number.
4. Change one thing at a time:
   • Strip the prompt to the bare task, with no planning advice. If the loop really lives in the model, AREX-2 should still climb and the base model should not.
   • Tell it 3 rounds, then 12. Does its first round change?
   • Wipe the history every few rounds and pass only a summary forward. Does that hurt AREX-2 more than the base model?

Hardware: this is a dense 27B model. In BF16 the weights alone are about 54 GB, before the KV cache. Running both models side by side needs a big GPU or two cards. Quantised builds already exist, such as a 4-bit EXL3 build, and the model is hosted on Featherless if you would rather not host it.

Our take

AREX-2 is less interesting as a benchmark number and more interesting as a direction. The loop we have all been writing into our harnesses is starting to move into the model. That should not surprise anyone. Model makers already tune their models to work well with their own coding agents, and this is the same move one layer down.

If you run long agent jobs, such as ML experiments, coding problems or research tasks, it is worth a try. Even if you never run it, it is a good excuse to look at your own harness and list what it actually does for the model. Some of those jobs the next model will do itself. Feedback, knowledge and budget will stay with you.

How we can help

If you run long agent jobs, or are planning to, we can look at your harness with you and work out which half is holding it back: the feedback the model sees, or what it keeps between rounds. We can also build the side-by-side test above on one of your own tasks, so you can see whether a model like AREX-2 earns a place before anything changes in production.

Our AI agents and chatbots service covers the agent, its tools and the harness around it. Our AI development service covers choosing a model and serving open-weights models like this one, on your own hardware or through a hosted provider.

Get in touch with what your agent does today and where it stalls, and we’ll tell you what’s worth changing.

Sources: AREX-2 paper (arXiv 2609.38288) · Model card on Hugging Face · Evaluation code on GitHub