Salesforce's 35% is a success rate

In June 2025 a trade headline said "enterprise AI agents fail 65% of multiturn tasks". The number comes from CRMArena-Pro, a benchmark from Salesforce AI Research. It is easy to read as a typical agent failing two jobs in three. The paper says something narrower, and in one way worse.

What the 35% is

The 35% is one model's score. Gemini 2.5 Pro, the best of the nine models tested, succeeded on 35.1% of multi-turn tasks in the paper's B2B environment. On the B2C environment the same model scored 30.0%.

It is a success rate, averaged over four business skills: workflow execution, policy compliance, text reasoning and database querying. The 58% that usually travels with it is the same model's single-turn score on the B2C environment (58.3%).

The paper's abstract says that leading agents reach about 58% single-turn success, dropping to about 35% in multi-turn settings. The headline kept the numbers and dropped the word "leading". CIO wrote that "agents achieved a 58% success rate in single-step tasks … but dropped to just 35% in multi-step workflows", with no model named.

The median model did worse

Read as a failure rate, 65% is the best case, not the typical one. Here is every model's multi-turn score from the paper's Table 2, averaged over the four skills:

The median model succeeded on 23.7% of multi-turn B2B jobs and 20.0% of B2C jobs: fewer than one in four.

What a failure means in this benchmark

A failure here is a final answer that did not match the expected one. The benchmark uses exact match for IDs and values and token-level F1 for free text. In the multi-turn setting, an LLM-simulated user with a sampled persona gives out the details of the request a little at a time, often incompletely at first, so the agent has to ask the right follow-up questions.

The models are the ones available in spring 2025: o1, GPT-4o, the Gemini 2.5 and 2.0 family, Llama 3.1 and Llama 4. The paper has one version, dated May 2025, so its numbers describe those models and not today's.

The number the benchmark does not report

CRMArena-Pro scores the final answer against the ground truth and stops there. It does not report whether an agent's own claim of completion matched the outcome.

In a deployment, the agent's summary may be the first thing a person sees, before anyone checks the records it changed. If the agent reports a refund to the wrong order as done, nothing in that report says otherwise. So the question for anyone deploying agents is narrower than "what share fail?" It is "of the jobs that fail, how many come back marked done, and what would catch them before they ship?" CRMArena-Pro does not answer that, and it was not designed to.

How to read the next agent benchmark headline

1. Which model? Here, the 35% is one model, the best of nine, in one of two environments.
2. Which setting? Single-turn and multi-turn can differ by nearly 30 points: Gemini 2.5 Pro scored 58.3% and 30.0% on the same B2C environment.
3. Success or failure? Here, the 65% is the complement of the best model's 35.1% B2B multi-turn score, the most favorable reading the paper allows.
4. What was scored? An exact-match answer, a final state, or a judge's opinion are three different measurements.
5. What did the agent report? If the benchmark doesn't say, nobody knows how many of the failures would have looked like successes.

That last question is the one we work on. Excellent treats an agent's "done" as a claim: it re-runs your frozen checks on the finished change instead of trusting the report.

Sources

• Huang et al., CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions, Salesforce AI Research, arXiv v1, May 2025. Abstract, section 3.6 and Table 2.
• CRMArena-Pro dataset card, Hugging Face.
• Luis Rijo, "Salesforce study reveals enterprise AI agents fail 65% of multiturn tasks", PPC Land, June 29, 2025.
• Mastufa Ahmed, "Salesforce study warns against rushing LLMs into CRM workflows without guardrails", CIO, June 17, 2025.