Skip to content

Excellent vs LLM-as-judge evals: models propose, oracles label

A judge model is a model grading a model. In a repository, the test suite is the better oracle.

curl -fsSL https://excellent.so/install.sh | sh
Jump to comparison

LLM-as-judge evals, in one line

Eval platforms such as Braintrust, LangSmith, Langfuse and Promptfoo, which score model output with LLM judges, code graders and human annotation against datasets you curate.

Read unchecked agent work, by the numbers for the evidence on what model review misses.

Best for
Measuring an LLM feature. If your product's output is text, an eval platform with a judge model and a dataset is the right instrument for the job.
Limits
A judge model is a model grading another model. On a generic dataset that is often the only option available. In a repository it is a worse oracle than the one already sitting there: the test suite, the build and the browser.
Stronger than us at
Everything to do with evaluating LLM applications: dataset management, experiment tracking, annotation queues, tracing at volume, side-by-side prompt and model comparison. Excellent does none of it and should not be your eval platform.
Price 2026-09-23
$0–$2,499 / mo (Published tiers across Braintrust, LangSmith and Langfuse)

Excellent in one line

A verification system for agentic tasks. It records what the agent attempted, runs your checks against the claim, and binds the result to evidence you can open.

Best for
Teams that want agent work checked before they trust or promote it.
Limits
There is no hosted option. You run it on your machines. Verification evidence stays on the machine that produced it and does not sync to teammates yet. No SSO, and no Mac app has shipped. It does not write tests, and it does not review your diff.
Stronger than them at
Keeping the attempt record, running your own checks as the oracle, and trialling prompt, model and tool changes on held-out work from your repository.
Price 2026-09-23
Free to install and run (Consulting from $10,000 / month)

The capabilities that decide it

Capability by capability, with LLM-as-judge evals on the left and Excellent on the right. Prices are the vendor's own, read on 2026-09-23.

CapabilityLLM-as-judge evalsExcellent
What decides pass or failModels propose; oracles label. A model may do the work, but something deterministic decides whether it passed.An LLM judge, a code scorer or a human annotatorTests, builds, browser checks and CI
What is being scoredModel output against a dataset you curateA code change in your repository, against the checks that repository already has
Runs on your own held-out workTrials use held-out tasks from your own repository before a prompt, model or tool change is promoted.Your datasets and tracesYes
Compares prompts and models side by sideYesOn held-out repository work, not on a generic dataset
General LLM tracing and observabilityYesNo
Records each agent attempt and the claim it madeTraces of LLM callsAttempts, checks, results and receipts for code work
Published price (checked 2026-09-23)Braintrust Starter $0 and Pro $249 per month, Enterprise custom; LangSmith Developer $0 per seat and Plus $39 per seat per month, both plus usage, Enterprise custom; Langfuse Hobby $0, Core $29, Pro $199 and Enterprise $2,499 per month, free to self-host. Promptfoo is open source; we did not re-check its enterprise pricing.Free to install and run. Consulting starts at $10,000 / month.

The rest of the field, named

Close enough to belong in this comparison, not different enough to need a page of their own.

LangSmith

Visit

Online and offline evals, datasets, annotation queues and tuned evaluators, plus an eval-engineering skill that builds evals from repository context. Developer $0 per seat and Plus $39 per seat per month, both plus usage; Enterprise custom (checked 2026-09-23).

Langfuse

Visit

The open-source, self-hostable option: traces, prompt versioning, datasets, experiments and LLM-as-judge. Hobby $0, Core $29, Pro $199, Enterprise $2,499 per month, with 100k units included and $8 per additional 100k graduating down at volume; free to self-host (checked 2026-09-23).

Promptfoo

Visit

Open-source evals and red-teaming, being acquired by OpenAI (announced March 2026). It publishes a guide for evaluating coding agents against the Codex SDK, Claude Agent SDK and OpenCode. That guide is the closest thing in this category to testing an agent change before you promote it. We did not re-check its pricing.

When to pick which

If you are evaluating an LLM application, use an eval platform; Excellent is not one. If you are verifying code an agent wrote, the oracle is already in the repository. Our rule is that a model may propose the work, but a test, a build or a browser decides whether it passed.

Stay with LLM-as-judge evals if

Measuring an LLM feature. If your product's output is text, an eval platform with a judge model and a dataset is the right instrument for the job.

Add Excellent if

You want agent work captured as attempts, checked against evidence you can open, and promoted only when the result is defensible to someone outside the team.

Trust the evidence, not the agent's word

curl -fsSL https://excellent.so/install.sh | sh