Agents that learn from past runs: an independent evaluation with Toloka

We wanted an independent assessor to test a claim we often make: agents become better and more reliable when they learn from their own past work. Toloka connected xmemory to TolokaForge, its open-source engine for running and grading agents, and evaluated it with 11 models in about 20,000 graded runs on airline cargo support tasks from Toloka Arena. [1]

The results support both of our expectations. Lessons from past runs raised scores by +0.084 on the tasks used to write them and by +0.048 on tasks that were not used. As the policy store, xmemory cut the total bill by 26% because it answers with compact records instead of whole passages. The benchmark didn’t cover long-running agents whose working state keeps changing, which is the setting xmemory is built for. That is the next experiment we want to run with an independent assessor.

What we set out to show

We went in with two goals:

• Learning is necessary. An agent that starts every task from scratch repeats the same mistakes. We wanted to measure, on someone else’s tasks and someone else’s grading, how much lessons from past runs change quality and reliability.
• Answers are compact. In earlier experiments with other design partners, xmemory’s answers were much shorter than the plain-text documents or vector-search chunks an agent would otherwise read. We expected that to show up as lower token cost here too, and wanted that measured independently rather than by us.

Toloka designed the tasks, ran the agents and graded every trial. We built the xmemory side of the integration.

The setup

Each task places the agent at a support desk. It talks to a simulated customer, looks up company policy, and operates the desk’s systems through tools such as a CRM, a cargo system, billing and claims. The benchmark has 50 tasks, and each model attempts every task five times in every configuration.

Each trial is scored from 0 to 1:

• Half comes from whether the final database state matches the reference.
• Half comes from an LLM judge’s rubric on the process the agent followed.

Every comparison is paired: the same model on the same tasks, with only the memory configuration changed. Brackets are 95% paired bootstrap intervals over tasks.

Configuration	Policy answers from	Memory of past runs
Document search (baseline)	a full-text index over the nine policy articles	none
xmemory as policy store	xmemory, loaded with the same articles, answering the same tool	none
Full stack	xmemory, as above	read-only lessons and playbooks written by an analyst agent from the graded runs of 35 of the 50 tasks; the other 15, from the same request families, were held out
Control (three models)	document search, as in the baseline	an earlier version of the same lessons, served by the document-search engine

The full run is on airline cargo. In smaller pilots across four domains, xmemory was level with or ahead of document search on airline, while document search kept the edge on HR, retail brokerage and telecom. Airline was also the domain where document search scored lowest, so it left the most room for memory to help.

Learning from past runs is what moves quality

With the full stack, all eleven models scored at least as well as document search. For three of them (GPT-5.6 Sol, Opus 5 and Gemini 3.7 Flash), the gain is within noise. These three already score 0.79–0.82 on document search, among the highest of the eleven, so they have the least room to improve.

The gain comes from the lessons:

• Tasks used to write the lessons: scores rose by +0.084 pooled over models, and ten of eleven models improved.
• Held-out tasks: scores rose by +0.048. Six models improved, and four of them have intervals clear of zero.

The lessons help most below the top tier. The strongest models already follow the process well and gain the least. With the lessons, an open-weight mid-size model, Qwen3.8 27B, scored 0.75. Two proprietary mid-tier models without lessons scored lower: GPT-5.6 Terra at 0.69 and Sonnet 5 at 0.68.

Closing the gap to the top tier

With lessons, mid-tier models scored 0.75 to 0.79, close to the 0.79 to 0.83 that the top-tier models reach on document search without them.

With the full stack, GPT-5.6 Terra reached 0.79, Sonnet 5 0.77 and Qwen3.8 27B 0.75. Without lessons, the four strongest models, Grok 4.6, GPT-5.6 Sol, Opus 5 and Gemini 3.7 Flash, scored between 0.79 and 0.83.

Toloka’s two-part grading shows where the improvement comes from. On held-out tasks, the judged process half of the score rose for all eleven models, while the final-state half stayed flat. With lessons, agents verify the account first, look for an existing case before opening a duplicate, and confirm before closing. These are the habits that make an agent dependable on the next task, not only correct on this one.

Compact answers, confirmed independently

The second expectation held up. A document-search hit returns whole articles, 26–65 KB on average per model. That text stays in context and is billed on every later turn. An xmemory answer is 2–9 KB of table, so agents read 3.5–15× less policy text for the same output.

As the policy store, xmemory cut the total bill by 26%, and all eleven models were cheaper. With the full stack, which adds the lessons to every trial, nine of the eleven models cost no more than 1.03× the baseline per trial. Grok 4.6 and GPT-5.6 Terra cost more because they pull in the most memory text per trial.

Integration: ask questions, not keywords

Document search and agent memory are used in different ways. With document search, the agent sends a bag of keywords and reads whatever articles come back. Broad queries work because each hit brings the whole article, including everything around the match. xmemory answers the question it is asked, with records shaped to that question. That is where the token saving comes from. It also means a rule that sits next to the answer in the policy text is left out unless the agent asks for it.

So the benefits of xmemory are unlocked by an agent that asks targeted questions. Every model in the run was instructed to do so, but not every model followed the instruction. One billing-dispute task from the full run shows the difference: a client disputes an invoice whose base rate doesn’t match the contract, and the account already has a resolved case for it.

	Grok 4.6, document search	Grok 4.6, xmemory	Kimi K3, xmemory
Policy reads	2 keyword queries, returning 80 KB and 29 KB	7 narrow questions, up to 6 KB each	1 broad question, returning 3 KB
Existing case	found and reopened	found and reopened	never checked; opened a duplicate
Mean score over five attempts	0.98	0.98	0.09

The pattern holds across the run. On xmemory, Grok 4.6 went from fewer than three policy reads per task to more than eight, each about twenty times shorter, and reached parity with document search. Kimi K3 kept asking one broad question per task and lost 0.045.

What comes next: long-running agents

This benchmark tested two kinds of knowledge: policy the agent looks up, and lessons from past runs. Our internal measurements show that xmemory works well for operational state that changes while the agent works, but this assessment did not test it, and we want an independent assessor to.

The next experiment would give the agent working state that keeps changing as it acts, with tasks that require combining many records and keeping them current along the way.

That is the setting of long-running agents, the same one behind our partnership with Temporal. [2]

Thanks to the Toloka team for designing, running and grading the evaluation. For evaluations and datasets, reach out to Toloka. [3]

Read Toloka’s own write-up of the evaluation. [4]