Evaluating AI agents
How to tell a useful AI agent from a confident chat box — separation of duties, evidence trails, checks, receipts, and the questions that filter the noise.
On this page
Every product launched in 2026 calls itself "agentic." Most are not. The difference matters because the capability you need is not a confident answer. It is work that can be checked, returned, accepted, and improved from evidence.
This is the framework we use to tell them apart.
The two-axis map
Plot every "AI feature" on two axes:
- Authority — does the AI just suggest, or does it act?
- Accountability — is there a separate party that verifies the action?
That gives you four quadrants:
| Low authority | High authority | |
|---|---|---|
| No accountability | Copilot | Loose cannon |
| Verifier required | Reviewable draft | Real agent |
You want products in the bottom-right. Most products are in the top-left. The top-right is what makes the news when it goes wrong.
What real agent capability looks like
Six properties, in priority order.
1. Named roles, not "the AI"
A real agent system has roles you can name:
- Shipper — does the work.
- Verifier — checks it.
- Triager — sorts the inbound.
- Scribe — writes the artifacts.
- Planner — sequences the work.
If the product just calls everything "the AI," ask which role you're getting. If they can't answer, the system doesn't have roles — it has one undifferentiated model with prompts.
2. Separation of duties
The shipper can't approve the shipper. The verifier is a different identity, ideally a different model or a different prompt with a different system context. This isn't a process — it's a property of the platform. The MCP server should refuse to mark work done when the actor is the same one that moved it to review.
If a single agent can finish its own work, you don't have separation of duties — you have a checkbox.
3. An evidence trail you can read after the fact
Every action the agent took should be queryable: which model, which tool, what arguments, what return value, what cost. The auditors aren't going to accept "we trust the agent." They'll accept a log.
The minimum bar: per-session, per-tool, per-cost timeline of every autonomous action.
4. Budgets the agent can't exceed
A real agent system has a budget — in dollars per session, tokens per task, or tool-calls per goal. The system refuses to continue past the cap and asks for supervision. If your AI feature can run indefinitely with no cap, it's a foot-gun.
5. Goals, not tasks
A copilot lives in a chat window and reacts to the last message. An agent has a durable goal that persists across sessions. "Triage the inbound queue daily" is a goal. "Summarize this email" is a task. The presence of goals is the cleanest tell between the two.
6. Inspectable prompts and tools
You should be able to see the system prompts, the tool definitions, and the policy file that governs what the agent can do. Vendors that hide the prompts hide the failure modes. If you can't read it, you can't reason about it.
How to test a vendor's claim
A practical evaluation, in four steps.
Step 1 — Read the audit log
Ask the vendor for a sample audit log from a real workflow. Look for: actor IDs, model names, tool calls with arguments, costs, and a clear before/after delta. If the "log" is INFO: AI processed task, walk away.
Step 2 — Try to make the agent approve its own work
Sign in as one user. Mark a task in progress. Move it to review. Try to move it to done as the same identity. The platform should refuse. If it doesn't, the verification gate is a UI convention, not a property of the system.
Step 3 — Look for the budget knob
Find the place where you set "this role can spend at most $X per session" or "at most Y tool calls per goal." If there isn't one, the vendor's confidence in the agent is unearned.
Step 4 — Read the prompts
If the prompts are inspectable, read them. They tell you exactly what the agent is told to do, what tools it can call, and what guardrails are nominal vs. structural. If the prompts are hidden, the failure modes are hidden with them.
Red flags
- "Just trust the AI." No.
- "Magic" or "intelligence" in the spec sheet. Spec sheets do not contain magic.
- A single AI module that does everything. That's marketing, not architecture.
- No way to disable the AI per-record or per-workflow. You will need to.
- Inference is mandatory in the vendor's cloud. Your customer data shouldn't be the price of admission.
What Excellent verifies
For comparison, Excellent does not ask you to trust a claimed agent layer. It records the work and checks the result:
- Work — the request, scope, owner, expected checks, and proof envelope are written down before the agent starts.
- Attempts — each run records what the agent claimed, what changed, and which evidence it attached.
- Checks — tests, browser proof, deployment proof, data checks, and custom gates run as independent evidence.
- Results — a result is accepted only when the evidence matches the work, not because the agent said it is done.
- Receipts — accepted work leaves behind a durable receipt that can be read later by a human or another agent.
- Experiments — model, prompt, and process changes are compared against observed outcomes instead of opinion.
You don't have to take our word for any of that — every one of those bullets is visible in the product and the docs.
Next up: the AI-agent vendor comparisons score specific vendors against this same six-point framework, so you can apply it without re-doing the homework.