Contents
Every assistant that claims to know you has the same problem underneath: your life is not in its context window. It is scattered across a year of photos, receipts, screenshots and chat logs, and the useful questions are the ones where no single file holds the answer. “How much did I pay for the second hotel on my Portugal trip?” is trivial to ask and genuinely hard to answer — you need the trip, the ordering of hotels within it, and the amount, from three different places.
That is what ATM-Bench-Hard measures. And for the last few months we have been running it over and over: different models, different coding agents, different reasoning tiers, different ways of handing an agent someone’s history.
The tally so far: 118 runs, 98 distinct model × harness × effort configurations, 55 models from ten vendors, 7 coding agents, 1.9 billion tokens, and a little over $800 of compute at list prices.
Here is why it was worth the electricity.
The baseline nobody benchmarks against
The result we keep coming back to is that a general-purpose coding agent, pointed at a directory of files with nothing but a shell, beats the systems built specifically to solve this.
Mem0, A-Mem, MemoryOS, MemPalace, SimpleMem, HippoRAG2 — real projects, real ideas, designed for exactly this task. Most of them lose to grep plus a competent agent that has no memory architecture at all.
We are not saying memory research is pointless. We are saying the trivial baseline is far stronger than the literature treats it as, and that a memory system which does not clear it has not yet earned its complexity. If you are building one, this is the number you should be embarrassed not to beat.
Retrieval is the whole problem
Hand a model the right evidence and it answers well. Make it find that evidence itself and the score collapses.
The distance between those two conditions is the benchmark. It is not a reasoning gap — the same weights, same prompt, get it right when the evidence is already in context. Everything lost in between is retrieval. Which means the field’s habit of chasing better answer models is aimed at the wrong half of the pipeline.
The harness is a bigger lever than the model
This is the finding that changed how we run everything else.
Hold the model fixed. Change only the coding agent driving it — same weights, same questions, same memory, same judge. The score moves more than most model upgrades buy, and on one model it moves more than the entire reasoning-effort ladder of a frontier system.
The mechanism, where we could trace it, is search effort. A harness that gives up early spends a fraction of the tokens and answers “Unknown” where a more persistent one finds the record — and it looks cheaper on the invoice while doing it. That is worth internalising if you are picking an agent on cost.
The related surprise: vendors’ own agents barely beat third-party ones on the same weights. The native-agent premium is real but small.
More reasoning is often worse
Turning the effort dial up does not reliably help, and on several models it hurts monotonically — with the best setting also being the cheapest.
The damage concentrates in open-ended questions while exact-match questions do not move at all. That is the signature of a model reasoning itself out of an answer it already had, rather than failing to find one. One vendor’s ladder is stranger still: the score ordering and the cost ordering run in opposite directions, because the cheaper reasoning tier plans retrieval worse and re-reads the same context over and over.
There is a Pareto frontier, and it is steep
Every priced run sits on one chart of total cost against score, with the frontier — the best score available at each price — drawn as a staircase.
It rises sharply at the cheap end and flattens hard at the top. A very small budget gets you most of the way; the last few points cost orders of magnitude more, and the most expensive run on the board is nowhere near the best. If you are building on this, the interesting engineering is at the cheap end of that curve, not the top.
Caveats
Single runs, no repeats. The effects worth trusting are the large ones. Small differences are noise, and we would not defend them.
Costs are list-price recomputations from token counters, not invoices. Subscription runs are priced at the equivalent metered rate to keep them on the same axis as metered ones.
Two runs are excluded from the published board, both for broken integrations rather than bad scores, and both named in the repo record. Removing a bad result quietly is how a comparison becomes an advertisement.
Every number behind this is on the ATM-Bench leaderboard, with all three charts live and hoverable. Submissions are one PR against a plain JavaScript array.
What we would most like is for someone to break the first finding. If your memory system beats a coding agent with a filesystem here, we want that row on the board.
Comments