Only One of Seven Agents Kept the Audit Trail
Across 11 realistic professional projects, DeliverBench found that even its best agent system scored 65 out of 100 for final delivery. In one audit project, all seven systems addressed review findings, but only one preserved traceability. Finishing a task and delivering work that can be checked are different achievements.1
That finding sits awkwardly beside some remarkable progress reported this week. Anthropic appears to be improving its models faster, a leading law firm is dedicating lawyers' working time to building specialist tools, and shopping agents are attempting purchases for customers. The next useful test is whether the work survives a change of requirements, a missing detail or a request to show the evidence.
Seven systems, one audit trail
DeliverBench was designed around an ordinary problem with extraordinary consequences: professional assignments do not arrive as one clean question. Clients send new documents, change the brief, challenge a calculation and ask for a revised version. A successful result therefore has to preserve what was learned at each stage, including the reasons an earlier decision was made. A polished final document can conceal a surprising amount of confusion behind it.
The researchers reconstructed 11 projects covering areas such as financial due diligence, legal contract drafting and drug safety reporting, working with domain experts and drawing on realistic materials. Together the projects contain 150 linked subtasks, spread over several rounds of fresh information and feedback. Importantly, agents were not handed ideal answers after each stage. They had to carry their own choices and errors into the next stage, as anyone working on a real client assignment would.1
The best of seven tested model-and-agent configurations received a final delivery score of 65 out of 100. At a demanding threshold of 90 points, it passed only half of the individual subtasks. Those figures describe this benchmark and these particular systems; they are not a universal failure rate for agents. But the design makes the result more informative than a test where every question starts with a clean slate and ends before the work becomes complicated.1
The audit example is particularly revealing. All seven systems addressed the issues raised in a review, yet only one preserved the traceability that connected the changes to the supporting work. In a professional setting, the revised number is only part of the answer. A colleague may need to find which source document established it, which assumption changed and whether the correction reached every related section. If that chain is broken, the team has to repeat the investigation before it can rely on the result.1
Consider a hypothetical due diligence report in which a newly supplied invoice changes a revenue estimate. Updating a spreadsheet cell is relatively simple; the task also affects the written analysis, supporting calculations, cited evidence and perhaps a conclusion about the purchase price. Fixing three of those four places can leave a persuasive report that is internally inconsistent. This is why checking one response at a time can flatter a system that performs poorly across the entire assignment.
There is also a design lesson here for people building AI content tools and other workplace assistants. A draft caption, a product description or a customer reply can look complete in isolation while contradicting information approved earlier in the week. A product launch moves, an offer expires or a client asks for a different emphasis; the later material needs to reflect that change without erasing the useful history. The benchmark's audit example is far more demanding than an Instagram post, but the underlying problem of continuity appears in both.
Anthropic's faster research cycle
The week's most consequential capability news came from METR, which independently examined how AI is affecting research and development at Anthropic. Based on an 11-week assessment that included internal data and staff interviews, METR estimated that Anthropic's progress on a set of well-specified benchmark tasks has accelerated by about 1.9 times its earlier rate since October 2025. Its stated 95% confidence interval for the benchmark-based estimate was 1.5 to 2.8 times. That measures the pace of improvement, not a claim that every current model is 1.9 times better at every job.2
METR also concluded that the acceleration could not be explained by faster growth in training compute alone. The researchers judged that extensive use of AI in research was probably responsible for much of the change, although they acknowledged other possible causes and significant uncertainty. Anthropic staff reported using AI to write code, run experiments and support model development. The picture is of capable people becoming more productive with increasingly useful tools, rather than models independently designing and building their successors.2
Some important evidence remains out of public view. Anthropic redacted substantial portions of the material that METR considered informative, and METR said this limits how well outsiders can judge its conclusions. The report is still notable because the researchers had unusually deep access and were candid about where their evidence fell short. It would be a mistake to turn a carefully qualified estimate into a neat prediction that development will keep doubling indefinitely.2
For product teams, the practical implication is that evaluations have a shorter shelf life. A workflow rejected six months ago because a model repeatedly failed at one step may now be worth testing again. Equally, a system that performed well when it was installed can acquire new capabilities, dependencies and failure modes as vendors update it. Evaluating a model once and treating that result as a permanent property of the product is increasingly difficult to defend.
The METR and DeliverBench findings should be read together without pretending they tested the same thing. One concerns the rate at which a frontier laboratory's models improve on measured capabilities. The other examines whether a set of agent systems can complete changing professional projects coherently. Both can be true: research capability can accelerate while delivery remains uneven. That distinction matters most to organisations deciding what to let software produce, recommend or change on their behalf.
A law firm budgets expertise
A&O Shearman announced an investment of $1.1 billion over five years in developing AI tools and services. The striking detail was how it proposed to spend much of that amount: each legal practice is expected to allocate at least 3% of its working time to the effort. This does not mean every lawyer individually sets aside an identical block of hours; practice leaders decide how to assign the work. Reuters reported 17 active projects under the approach, which the firm says began in July.3
A large part of the commitment is therefore the opportunity cost of specialists working on new systems instead of ordinary fee-earning matters. That is a very different definition of AI investment from a software subscription or a contract for extra computing capacity. The firm's lawyers understand which clauses change the meaning of a deal, where a client is likely to ask for an exception and what evidence must accompany a legal conclusion. Those details take time to put into a tool, even when generating code and text becomes faster.
The firm already has experience with specialist legal technology, including ContractMatrix, developed with Harvey and Microsoft. Its new allocation of working time suggests that organisations are beginning to price the knowledge that makes automation useful, rather than assuming general-purpose intelligence arrives ready to practise law. A system that can draft a clause is valuable. One that can also explain why it used that clause, relate it to the client's instructions and carry a negotiated change across related documents is much closer to useful professional work.3
Snowflake made a complementary announcement this week. Its Business Ontology offering is intended to give multiple agents shared, approved definitions for business terms such as net revenue, rather than making each system infer what a database column means. Cortex AI Gateway, announced alongside it, is aimed at managing how agents access models and tools. These are vendor announcements about product capabilities, not independent proof that all the resulting answers will be correct.4
The definitions problem sounds narrow until it changes a decision. Imagine finance counts net revenue after refunds, while sales talks about bookings before refunds. Both teams may use the word revenue, and a language model may calculate either figure perfectly. Unless the business specifies the intended definition and the data behind it, two agents can deliver incompatible answers without either making an arithmetic mistake. The meaning of the number belongs in the system, along with the number itself.
MariaDB's new agent-memory and semantic-caching features address a related technical problem: how to retain useful information across sessions and avoid repeating unnecessary model calls. Remembering previous work, using current business definitions and connecting those definitions to evidence are separate jobs. An assistant can recall an old customer request and still miss that the price changed this morning. Persistent memory helps continuity only when there is a way to distinguish what remains valid from what needs to be checked again.5
Smaller organisations cannot dedicate hundreds of lawyers to this task, but they have an advantage in knowing which exceptions matter. A shop owner can identify products with unusual returns rules; a restaurant manager knows when substitutions are possible; a marketing team knows which promises have already been made to customers. Documenting those details before automating the work is less glamorous than trying a newer model. It also gives the business something concrete against which to judge whether the tool has improved.
Visa's agents forgot to ask
A test reported by Fortune offers a consumer-facing version of the same problem. Visa asked 22 AI shopping agents to find suitable TVs, laptops and hair appliances. Across the evaluation, the agents succeeded about 57% of the time, with the strongest reaching 82%. Several struggled to ask for missing requirements, and some recommended unsuitable products. The numbers describe Visa's test conditions, not the likelihood that every real-world agent purchase will go wrong.6
A shopping assistant can retrieve a long list of laptops and still fail the customer who has not mentioned that a particular game needs to run well. It can find a TV at the requested price without resolving a requirement about size or compatibility. Product selection is full of constraints that emerge through conversation, and a genuinely useful assistant has to recognise when a recommendation would be premature. A fluent answer is no substitute for asking the question that would change the choice.
The commercial effect extends beyond the shopper. A retailer's inventory, returns policy, delivery restrictions and variant names all influence whether an agent can recommend the right item. If that information is scattered across product pages and old promotions, errors become more likely. The retailer may then deal with unnecessary returns and disappointed customers, even though its catalogue was technically available to an assistant.
This is a useful warning for the emerging idea of agentic commerce. Completing a checkout is an action with a visible end point, but a successful purchase means the customer receives an item that satisfies their requirements. A benchmark that only measures whether an agent reaches a payment screen would miss a substantial part of the job. Visa's more demanding test at least begins to expose the difference between moving through a shopping flow and making a sound selection.6
It would be easy to respond by adding more prompts or forcing every shopping agent through an enormous questionnaire. That would recreate the friction these products were meant to remove. A better design asks a small number of consequential questions, remembers the answers while they remain relevant and makes the reasons for the recommendation inspectable. The customer should be able to see why a product was selected, particularly when a requirement rules out a cheaper or more popular alternative.
The evidence must survive delivery
These stories point to a practical change in how organisations should evaluate an AI system. Instead of giving it ten separate questions, give it a realistic assignment with a changing brief, a correction halfway through and a final handover to somebody who did not watch the work unfold. Ask that person whether the result is usable and whether its important claims can be traced back to source material. Count the rework, not only the tasks the system says it completed.
Faster model development will make this test more demanding, because systems will become capable of attempting larger assignments. A law firm investing its own specialists' time and a database company standardising business definitions are both signs that model improvements alone cannot supply the working knowledge a finished job requires. Those investments may prove worthwhile, but their value should be measured in fewer corrections, more consistent outputs and less time spent reconstructing decisions. The best benchmark is increasingly the one that resembles the work people pay to have finished.
For anyone building with generative AI, the uncomfortable finding in DeliverBench is also an opportunity. Six systems in one audit example could address the visible review comments without preserving the chain of evidence, which is precisely the kind of failure a polished demo rarely reveals. A product that reliably keeps the brief, changes and supporting material connected has earned a much stronger claim than one that produces a convincing first draft. That is the standard worth testing as the underlying models improve.1