# Excellent docs (full text) > Every page of the Excellent documentation (https://www.excellent.so/docs) as markdown, in the order the docs sidebar shows them, then the buyer guides and every research post (https://www.excellent.so/research), newest first. Each page ends with its source URL. Release availability: the CLI currently published as npm latest is 0.1.0 (checked October 27, 2026). It predates the intended v1 checked workflow described below, including run, verify and receipt verification. Those commands are not available in that npm release. Before recommending an installation or command, read the [quickstart version note](https://www.excellent.so/docs/quickstart) and [changelog](https://www.excellent.so/changelog). --- # What Excellent does > The shortest description of the product: Excellent runs next to your AI agents, records each attempt and checks the result with independent evidence. Excellent runs next to your AI agent, records the attempt, checks the result, and tests changes that may make the agent more reliable. ```bash excellent init ``` Expected result: `AGENTS.md` and `.excellent/{context.md,cli.toml}` are scaffolded in the current repository, ready for checked work. If it fails, the command exits with the missing prerequisite or permission it could not use. The agent's "done" is a claim. Excellent's job is to be the thing that goes and looks, with evidence the agent did not author, reached by a verifier that is not the agent. Start at the [quickstart](https://www.excellent.so/docs/quickstart). Source: https://www.excellent.so/docs/what-excellent-does --- # Install > Install the CLI in a repository you already use with your agent, initialize it with excellent init, check the install, see where its files live, and uninstall it. Install Excellent in a repository you already use with your agent. **Step 1: Install the CLI** ```bash curl -fsSL https://excellent.so/install.sh | sh ``` **Step 2: Initialize your repository** ```bash cd your-repo excellent init ``` Expected result: the `excellent` command is available and the repository has a confirmed local configuration. If installation fails, read the command output first. It should name the missing platform support, permission, or package step. Prerequisites, what the installer touches, and where Excellent runs are on the [install page](https://www.excellent.so/install). ## Verify the install ```bash excellent --version ``` That prints `excellent` followed by a version number. The current published version is stated on [/install](https://www.excellent.so/install). If the command is not found, open a new terminal so your shell picks up `~/.local/bin`. ```bash excellent doctor ``` It reports the runtime, the local database, detected model providers and coding CLIs, your PATH and the keystore. Anything marked `!` is a thing to fix before you rely on a result. Real output, from a development machine. This capture is from a monorepo checkout, not an npm install, which is why the header reads `cli 0.0.0`; installed from npm, it shows the published version. Your `runtime`, `providers` and `coding CLIs` lines will differ. ```console $ excellent doctor Excellent CLI doctor (node v24.20.0 · cli 0.0.0) ✓ runtime companion (monorepo) — monorepo checkout at ~/Sites/micro home ~/Sites/micro ✓ database ~/.excellent/data.db (exists) 0 workspace(s) ✓ providers no HTTP keys set local runtimes: ollama, lmstudio, vllm ✓ coding CLIs claude 2.1.280, codex 0.156.1, grok 1.0.34 ✓ PATH ~/.local/bin on PATH ✓ keystore present, recovery NOT set ✓ telemetry opt-in, off by default; 16 scrubbed event types disclosed ✓ single-writer off — no writer elected ✓ ready — run excellent to start chatting. ``` ## Where the files live | Path | What it holds | |---|---| | `~/.excellent/data.db` | Local database | | `~/.excellent/cli.toml` | CLI preferences | | `~/.excellent/chat-sessions/` | Saved chat sessions | | `./.excellent/context.md` | Per-repository context, written by `excellent init` | | `./.excellent/cli.toml` | Per-repository preference overrides | | `./AGENTS.md` | Agent instructions, written by `excellent init` | Set `EXCELLENT_DATA_DIR` to move the local database and sessions somewhere else. ## Uninstall There is no `excellent uninstall` command. Remove it by hand. First delete the `excellent` entry from each agent's config file, so no agent is left starting a server that is gone: [the files are listed here](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file). Then: ```bash npm uninstall -g @excellent-so/cli rm -rf ~/.excellent ``` The last command deletes every local record. It is irreversible: everything Excellent recorded is in that directory. Source: https://www.excellent.so/docs/install --- # Quickstart > Learn the intended v1 setup for a repository, an AI agent, checks, and results. The published CLI 0.1.0 predates this workflow; review the release note. This page walks from nothing to a checked result. Every block marked **real output** was copied from an actual terminal. Where a step has no transcript yet, it says so rather than showing you invented output. ## 1. Install ```bash curl -fsSL https://excellent.so/install.sh | sh ``` You need macOS and Node.js 22.3 or newer. The script checks the Node version, downloads the release, verifies it against the published `SHA256SUMS`, and installs `@excellent-so/cli` globally. If the checksum is missing or does not match, it stops and installs nothing. ## 2. Confirm the CLI is there ```bash excellent --version ``` That prints `excellent` followed by a version number. No transcript here on purpose: `excellent --version` reports whatever is in the `package.json` it was installed from, so the number depends on when you installed and any output pasted here would be wrong by the next publish. The current published version is stated once, on [/install](https://www.excellent.so/install). If the command is not found, open a new terminal so your shell picks up `~/.local/bin`. ## 3. Scaffold a repository Run this inside a repository you already use with your agent. Real output: ```console $ cd your-repo $ excellent init ✓ created AGENTS.md ✓ created .excellent/context.md ✓ created .excellent/cli.toml Edit these, then run 'excellent' here — the chat reads them as context. ``` `AGENTS.md` is what your agent reads. `.excellent/context.md` is the repository context Excellent itself loads. Edit both before your first real task. They are the difference between a useful check and a vague one. ## 4. Check the local setup Real output, from a development machine. Absolute paths are shortened to `~`, and the line naming the signed-in model account is removed. Your `runtime`, `providers` and `coding CLIs` lines will differ. This capture is from a monorepo checkout, not an npm install. That is what the `runtime companion (monorepo)` line means, and it is why the header reads `cli 0.0.0`. The monorepo carries a placeholder version; the real number is set at publish time. Installed from npm, that header shows the published version instead. ```console $ excellent doctor Excellent CLI doctor (node v24.20.0 · cli 0.0.0) ✓ runtime companion (monorepo) — monorepo checkout at ~/Sites/micro home ~/Sites/micro ✓ database ~/.excellent/data.db (exists) 0 workspace(s) ✓ providers no HTTP keys set local runtimes: ollama, lmstudio, vllm ✓ coding CLIs claude 2.1.280, codex 0.156.1, grok 1.0.34 ✓ PATH ~/.local/bin on PATH ✓ keystore present, recovery NOT set ✓ telemetry opt-in, off by default; 16 scrubbed event types disclosed ✓ single-writer off — no writer elected ✓ ready — run excellent to start chatting. ``` Anything marked `!` is a thing to fix before you rely on a result. ## 5. Your agent is already wired The installer registered the Excellent MCP server in every agent it found on your machine and knows how to configure: Claude Code, Cursor, Codex and 25 others. You do not need to configure anything by hand. To see exactly which file it wrote for each agent and how to undo it, read [Claude Code, Cursor and other agents](https://www.excellent.so/docs/claude-code-and-cursor). If you installed the CLI some other way, register it now: ```bash excellent install-agent-mcps ``` > **Not in the CLI on npm yet.** `@excellent-so/cli` 0.1.0, the release npm serves > (checked October 23, 2026), predates `install-agent-mcps`, `excellent run` and > `excellent verify`. With it, the installer skips agent setup and says so; add your > agent's entry [by hand](https://www.excellent.so/docs/claude-code-and-cursor#adding-the-entry-by-hand) instead. > Steps 5 and 6 describe the next release; the [CLI reference](https://www.excellent.so/docs/cli) says which > commands 0.1.0 has. ## 6. Run a task and get it checked Work in Excellent is a **Work item**, an **attempt** against it, and a **verification** of that attempt. The agent does not get to close its own loop: the verifier must be a different actor in a different execution boundary, which is why the result means something. [How verification decides](https://www.excellent.so/docs/how-verification-decides) explains the rules. Two ways in. **Through your agent.** This is the normal path. With the MCP server registered, the agent opens an attempt, submits its output, and asks for a status, using the `universal_verification_open_attempt`, `universal_verification_submit_attempt` and `universal_verification_status` tools. You do not type these; the agent calls them. **From the terminal**, against an existing task id: ```bash excellent run --agent --task ``` `--agent` is a role name and `--task` must be an existing numeric task id. A free-text objective is refused with a structured hint. File it as a task first. Then: ```bash excellent verify ``` > **Not yet transcribed.** We have not published a real end-to-end transcript of > step 6 against a fresh repository. The commands and tool names above are read from > the CLI source, but the output is deliberately omitted rather than invented. ## 7. Verify a receipt yourself A verification produces a signed receipt, and you do not have to take our word for one. The public verifier is a single file with no dependencies, served from this site beside an example receipt pack and the public key it was signed with. After the download it needs no network, no database and no Excellent installation: ```bash curl -fsSLO https://www.excellent.so/verify-receipt.mjs curl -fsSLO https://www.excellent.so/receipt-example.json curl -fsSLO https://www.excellent.so/receipt-example-key.pub.pem node verify-receipt.mjs receipt-example.json --pubkey receipt-example-key.pub.pem ``` It prints `PASS` or `FAIL` for each of ten checks and ends `RESULT: VERIFIED`, exit 0. Change any character of the receipt's content and run it again: the payload-hash check fails, it ends `RESULT: NOT VERIFIED` and exits 8. [Receipt format](https://www.excellent.so/docs/receipt-format) has the full real output and what each check means. ### From an Excellent monorepo checkout This route is only for people who already have a checkout of the Excellent monorepo. Its source is not public, so if you do not have one, the public verifier above is the check to run. It verifies the repository's release receipt, the one shown on [Sample receipt](https://www.excellent.so/docs/sample-receipt), with the repository's own standalone verifier. Copy the verifier and signed receipt outside the repo; that is the point of the check: ```sh mkdir /tmp/receipt-check && cd /tmp/receipt-check cp /scripts/lib/offline-receipt-verifier.mjs . cp /docs/v0/release-receipt.json . cp /docs/v0/release-receipt-public-key.pem . node offline-receipt-verifier.mjs release-receipt.json release-receipt-public-key.pem ``` Replace `` with the path to your Excellent monorepo checkout. The real output is: ```json { "bound": true, "receiptId": "receipt_v0_release_4df3dd89b3e4d0a0", "tenantId": "tenant_v0_release", "decisionDisclosure": "UNDISCLOSED", "claimedKeyId": "v0-ed25519-2026-08", "claimedSignatureAlgorithm": "ed25519", "signatureAlgorithm": "ed25519", "payloadHashMatches": true, "signatureValid": true, "excellentReachable": false, "excellentResolutionError": "ERR_MODULE_NOT_FOUND", … } ``` The repository also includes a deliberately altered receipt. Copy it into the same directory and run the verifier again: ```sh cp /docs/v0/adversary/receipt-tamper-fixtures/tampered-receipt.json . node offline-receipt-verifier.mjs tampered-receipt.json release-receipt-public-key.pem ``` It demonstrates the negative result shape: ```json { "bound": false, "receiptId": null, "tenantId": null, …, "claimed": { "receiptId": "receipt_exp_v0_proof_cand_v0_proof", … }, "claimedKeyId": "v0-ed25519-2026-08", "payloadHashMatches": false, "signatureValid": false, "tamperedSignatureRejected": true, "tamperedContentRejected": true } ``` A tampered receipt can still produce exit 0 because the verifier reached an answer; read `bound` and the signature/hash fields. See [Sample receipt](https://www.excellent.so/docs/sample-receipt) for the complete document and explanation. ## Where to go next - [CLI reference](https://www.excellent.so/docs/cli): every command that exists, and its flags. - [How verification decides](https://www.excellent.so/docs/how-verification-decides): evidence, authority, and why a model can never sign off a pass. - [Claude Code, Cursor and other agents](https://www.excellent.so/docs/claude-code-and-cursor): the exact config each agent gets. ## Undo it There is no `excellent uninstall` command. Remove it by hand. First delete the `excellent` entry from each agent's config file, so no agent is left starting a server that is gone: [the files are listed here](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file). Then: ```bash npm uninstall -g @excellent-so/cli rm -rf ~/.excellent ``` The last command deletes every local record Excellent kept. It is irreversible. Source: https://www.excellent.so/docs/quickstart --- # Connect an agent > Use the agent and model provider you already chose. Excellent attaches to Claude Code, Cursor and other MCP-capable agents as an MCP server. Excellent attaches to the agent you already use, as an MCP server. The installer does this for you; run it again yourself at any time: ```bash excellent install-agent-mcps ``` Expected result: Excellent finds every agent on the machine that it knows how to configure (28 agents, among them Claude Code, Cursor, Codex, Gemini CLI and goose) and writes an `excellent` server entry into each one's own config file: `~/.claude.json` for Claude Code, `~/.cursor/mcp.json` for Cursor, `~/.codex/config.toml` for Codex, and so on. It adds that one entry and leaves the rest of the file alone, and prints a line for each agent it configured. Restart those agents afterwards, or they will not see the new server. It never guesses at a config path. An agent it does not know, or knows but cannot configure safely, gets no entry and no line in the output. > **Not in the CLI on npm yet.** `@excellent-so/cli` 0.1.0, the release npm serves > (checked October 23, 2026), predates `install-agent-mcps`. With it, add the entry by > hand: [Adding the entry by hand](https://www.excellent.so/docs/claude-code-and-cursor#adding-the-entry-by-hand). Undo it by deleting the `excellent` entry from that agent's config file. `excellent-mcp uninstall-skills` does this for you only in a source checkout. For every agent's file and exact entry, the tools each agent sees, and the Claude Code hooks, read [Claude Code, Cursor and other agents](https://www.excellent.so/docs/claude-code-and-cursor). Source: https://www.excellent.so/docs/connect-coding-agent --- # Run the first task > Start with one useful task, not a demo. Run a real task with excellent run, read the attempt it records, and know what the exit codes mean. Run a task that has a real expected outcome in your repository. ```bash excellent run --agent --task ``` Both flags are required. `--task` must be an existing numeric task id: a free-text objective is validated and then refused with a structured hint, because the underlying pathway targets task ids. File the objective as a task first. Expected result: the role attempts the work, Excellent records the attempt, and a `correlationId` is printed on stderr. Exit 0 completed, 1 not completed, 2 usage. If Excellent cannot tell whether the work passed, the result is `Not checked` or `Needs a decision`, not a silent pass. Undo the accepted change with the rollback instruction attached to the result. Source: https://www.excellent.so/docs/run-the-first-task --- # Read a result > Understand what passed, failed, or could not be checked: read a verification result from the CLI or from inside your agent. Read the result after an attempt completes. ```bash excellent verify --json ``` Expected result: a verification starts and returns its id. Passing a `ver_*` id instead reads the status of an existing one. Exit 0 ok, 1 runtime, 2 usage, 3 authorization, 4 not found. From inside an agent, the same read is the `universal_verification_status` tool. A result shows what was requested, what the agent changed, what the agent claimed, what Excellent checked, what passed, what failed, and what could not be checked. If the local UI cannot open, use the CLI output and logs named by the failed command. Source: https://www.excellent.so/docs/read-a-result --- # How verification decides > Evidence, authority, and why a model can never sign off a pass: the six verdicts, what counts as evidence, retries, flaky tests and anti-gaming. "Done" is a claim an agent makes about its own work. A verdict is what happens when something else goes and looks. This page is the rulebook: what counts as evidence, who is allowed to conclude what, and what stops the whole thing from being gamed. ## The six verdicts A verification resolves to exactly one of six states. | Verdict | Meaning | |---|---| | **PASS** | Every obligation holds, at or above the authority floor. | | **MISMATCH** | An independent record was consulted and **contradicted** the claim. This is an accusation. | | **FAIL** | Conclusive, not a pass, and no independent contradiction was demonstrated. A refusal to certify. | | **UNKNOWN** | No finding. Coverage was partial or integrity did not hold, and nothing the agent does will change that. A human is asked to fix an instrument. | | **NOT_RUN** | Nothing was looked at. | | **PENDING** | Too early to tell, and there is a deadline by which it will not be. | The distinction between `FAIL` and `MISMATCH` matters: only the oracle seam, which reads a record the *verifying* side populates, can mint a `MISMATCH`. An evaluator reads the bundle the submitter chose, so an evaluator can never author an accusation. When several checks are folded into one verdict, the order of precedence is: ``` MISMATCH > FAIL > UNKNOWN > NOT_RUN > PENDING > PASS ``` `PASS` is last. One unresolved check is enough to sink an otherwise clean set. An empty set aggregates to `NOT_RUN`, not to `PASS`. `PENDING` is resolved at read time, never by writing back, because a receipt is signed and append-only. A pending verdict that runs out of deadline resolves to `UNKNOWN`, `FAIL`, `MISMATCH` or `NOT_RUN`. It is a type-level guarantee that it can never expire into `PASS`. ## Who is allowed to conclude what Every check records the authority of whatever decided it: - `deterministic`: a machine re-ran something and observed the outcome. - `human`: a person looked. - `semantic`: a model judged. - `imported`: the result came from somewhere else. **Only `deterministic` and `human` can carry a set to PASS.** This is a membership test, not an exclusion, so it fails closed: a new authority added later is untrusted until it is deliberately added to that list. A `PASS` also requires a deterministic evidence-integrity proof. Without one, the determination is rejected outright. ## When model output counts as evidence It counts. It just cannot be the thing that says yes. A semantic worker can move a verdict **down** (a cited conflict becomes `FAIL`) or route it to a person. It can never move one up. The routing table has six outcomes, and every one of them resolves to `UNKNOWN` except a cited conflict, which resolves to `FAIL`: | Route | Taken when | |---|---| | `unknown_uncited` | The judgment cited nothing. | | `uncalibrated_model_needs_review` | The model version has no calibration on record. | | `unknown_model_reported` | The model itself said it did not know. | | `cited_conflict_exception` | The model found a cited contradiction. Resolves to `FAIL`. | | `high_consequence_support_needs_human` | The judgment supports the claim, but consequence is high or critical. | | `cited_support_needs_review` | The judgment supports the claim and a person still reviews it. | For a semantic judgment to be considered at all, it must be **grounded**: bound to a frozen evaluation case, with every citation resolving to an admitted artifact carrying a `sha256:` quote hash. A judgment with no typed predicate is not refused. It resolves `NOT_RUN`, which aggregates to `UNKNOWN`. The practical consequence, stated in the product's own posture: > A semantic worker may never issue an authoritative PASS, and must cite the evidence > it relied on or return `unknown_uncited`. So the strongest outcome an injected > instruction can reach is routing a judgement to a person — never minting a pass, > never closing a finding. That is also the answer to prompt injection. An attacker who fully controls the model's output can, at worst, cause a human review. ## What counts as evidence Evidence is typed on several axes at once, and the combination is what gives it weight: - **type**: `artifact`, `observation`, `attestation`, `review`, `import` - **admission source**: `producer`, `collector`, `reviewer`, `human`, `imported` - **independence**: `independent`, `self_attested`, `derived`, `unknown` - **mutability**: `immutable`, `versioned`, `mutable` - **freshness**: `fresh`, `stale`, `expired`, `unknown` - **accessibility**: `accessible`, `inaccessible`, `redacted`, `unknown` Independence is assessed on three separate questions (is the actor distinct, is the execution boundary distinct, is custody distinct), and the assessment must state its basis. It is resolved from structure, not accepted as a label. A producer that declares itself independent is not independent. A claim only reaches `PASS` or `FAIL` through a **decidable observable**. At the current tier that means one thing: `evidence.content-digest`, where the expression is exactly `(= "sha256:<64 hex>")`. Anything else binds and answers `UNKNOWN`. Notably, `evidence.present` is explicitly declined: the absence of cited evidence is not falsehood. ## Retries, re-runs and flaky tests **A re-run is a new attempt, not an overwrite.** Receipts are signed and append-only. The lawful way to change a verdict is to issue a new receipt carrying `supersedesReceiptId`. Nothing edits history, so a fail followed by a repair and a pass leaves both on the record. Retry limits that exist today: | Where | Limit | |---|---| | Semantic judgment per check | 2 attempts, then `semantic-retry-limit-exceeded` | | Verification API executor | 3 attempts | | Connector queue | 3 attempts | | Webhook delivery | 6 attempts, backing off 1m, 5m, 25m, 2h05, 10h25 | > **Flaky tests are not modelled.** There is no quarantine state, no flake score and no > "re-run until green" path in the verification kernel. That is by design, because the shape > of "retry until it passes" is exactly what the system exists to refuse. A test that > passes and fails on the same input produces two receipts that disagree, and that > disagreement is the finding. If you want flake handling today, it belongs in your test > runner, before the result becomes evidence. ## Anti-gaming Six rails, all enforced in code rather than convention. **An agent cannot verify its own work.** A completion is legitimate only if at least one actor who never claimed the task performed the verification. The check is scoped to the current cycle, so a stale approval from an earlier round cannot satisfy it after a reopen, and an unnamed actor cannot count as a "distinct" claimer. **An agent cannot verify code it wrote**, even if someone else claimed the task. The verifier is refused if it appears as the committing actor of any link anchoring the work, including superseded links, so a rebase does not erase the original author. **The verifier must not share the worker's execution boundary.** Same actor is refused as `worker-self-verification`; same mutable execution boundary is refused as `shared-execution-boundary`. A worker also cannot accept the contract for its own attempt. **A producer's account of what changed is not evidence.** A change manifest is only admitted if it was independently *observed*: a diff computed in a store that is never mounted into the container that produced the change. A producer-declared manifest returns `UNKNOWN`, never `FAIL` and never `PASS`, with the reason `change-manifest-not-independently-observed`. **The baseline is captured before the work starts.** Any predicate comparing before and after needs an authoritative pre-mutation baseline. Without one it produces `UNKNOWN`. The correlation token cannot be minted except by the capture step, so no caller can fake having taken it. **An obligation that cannot fail is refused.** Falsifiability is decided by witness: a witness that fails proves the obligation was falsifiable. Vacuous predicates, unreachable fail branches and undecidable predicate kinds are all rejected. A false refusal costs an argument; a false admission signs a verdict that means nothing. On top of those, a structural gate walks the transitive import closure of every hard-gate decision path and refuses a build in which any of them can reach the statistical scorer. No learned component sits in a gate decision, and the gate refuses rather than passes if its own scan shrinks. ## What this does not claim Independence today is **process-distinct, not party-distinct**. A verifier running in a separate boundary on the same machine, under the same operator, is structurally separated from the producer, but it is not an adversarial third party. Read a receipt for what it is: tamper-evident proof of what was bound and checked together, by a process that could not also have written the code. Verifying a signature proves the receipt has not been altered since the key holder signed it. It does not prove the key is the tenant's key. If you obtained the key from the same system that issued the verdict, a forged receipt re-signed under an attacker's key verifies exactly as cleanly as a genuine one. Confirm the fingerprint through a channel this system does not control, then pin it. ## Related - [Sample receipt](https://www.excellent.so/docs/sample-receipt): what a signed verdict actually contains. - [Receipts](https://www.excellent.so/docs/receipts): verifying one. - [Claims and evidence](https://www.excellent.so/docs/claims-and-evidence): the shorter version of this page. Source: https://www.excellent.so/docs/how-verification-decides --- # Agents and versions > Track the setup used for each attempt. An agent version records how the agent was configured, so every result belongs to one setup. An agent version is the recorded setup used to run an agent. Every attempt is bound to the setup that produced it, so a result is always a result *for a particular configuration* rather than for "the agent" in general. Run `excellent doctor` to see what the CLI resolved on this machine: runtime, database, model providers, detected coding CLIs and versions. If no agent has been configured, initialize the repository and run the first task. Roll back a promoted version from the result or release page that promoted it. Source: https://www.excellent.so/docs/agents-and-versions --- # Work and attempts > Work is the request; attempts are the tries. How Excellent records each attempt against the repository state and agent setup, and links retries. Work is the requested outcome. An attempt is one try to complete it. ```bash excellent run --agent --task ``` Expected result: Excellent creates an attempt record and binds it to the repository state and the agent setup. If an attempt is returned, the next attempt links back to the prior one. The prior attempt is not rewritten. Undo accepted work with the rollback path shown in the result. Source: https://www.excellent.so/docs/work-and-attempts --- # Claims and evidence > Done is a claim, not proof. How Excellent weighs evidence from Git, tests, CI and people, and why a model's judgment can never carry a pass. Excellent treats an agent's "done" message as a claim. Evidence can come from Git, tests, browser checks, CI, APIs, databases, deployments, another model, or a person. It is not all worth the same: evidence carries a type, an admission source, an independence classification, a mutability and a freshness, and those together decide what it can establish. Two consequences worth knowing up front. A model's judgment can move a verdict down or send it to a person, but it can never carry a pass. And an account the producer wrote of what it changed is a claim, not evidence. Only an independent observation of the change can establish anything. If evidence is missing, the result says what was not checked. See [How verification decides](https://www.excellent.so/docs/how-verification-decides). Source: https://www.excellent.so/docs/claims-and-evidence --- # Checks and results > The plain result labels used by Excellent: Verified, Returned, Needs a decision and Not checked, and why a check that cannot run stays visible. Checks decide whether a claim held. A check returns one of four plain states: ```text Verified Returned Needs a decision Not checked ``` If a check cannot run, the result should remain visible as `Not checked` rather than disappearing from the denominator. Source: https://www.excellent.so/docs/checks-and-results --- # Experiments and candidates > Test better ways of running the agent. Experiments compare candidate setups (instructions, model, tools, retries) against the current one. Experiments compare candidate agent setups against the current setup. Candidates may change instructions, model selection, tool descriptions, context, checking, retrying, and stopping rules. If a candidate sees the held-out tasks or lacks required evidence, it is invalid. Source: https://www.excellent.so/docs/experiments-and-candidates --- # Promotion and rollback > Promote only what holds up. How a candidate agent setup is promoted on held-out evidence, and how to roll back from the promotion receipt. Promotion changes the agent setup after a candidate holds up on work it did not see. Promotion is gated on evidence the candidate did not see, and the exact diff is shown before anything changes. If regressions appear, roll back to the prior version from the promotion receipt. A receipt is append-only, so the rollback is a new record that supersedes the old one rather than an edit to it. Source: https://www.excellent.so/docs/promotion-and-rollback --- # Receipts > Verify what was bound and signed. What a receipt binds, how to check its signature against a key you pin, and what a valid receipt does not prove. A receipt binds input, checks, evidence, decision, and software version. ```bash excellent-mcp receipt verify --pubkey ``` Expected result: one JSON line reporting whether the canonical payload hash and the Ed25519 signature match, and whether the pack anchors to the key you pinned. Exit codes: 0 verified, 2 usage, 8 unverifiable or invalid. Pin the key yourself. A receipt re-signed under an attacker's key, handed to you with that attacker's key, verifies exactly as cleanly as a genuine one. See a real receipt and a real tamper being caught on [Sample receipt](https://www.excellent.so/docs/sample-receipt). The format itself is published: [Receipt format](https://www.excellent.so/docs/receipt-format) has the JSON Schema, an example pack and a zero-dependency verifier you can run on a machine that has never had Excellent installed. A receipt does not prove every external source was truthful. It proves which records were bound and signed together. If verification fails, do not rely on the result until the digest or signing key issue is resolved. Source: https://www.excellent.so/docs/receipts --- # Security and data > Local storage, model providers, export, and delete: what Excellent keeps on your machine, what your agent still sends out, and how to remove it. Excellent runs locally by default and records work on your machine. ```bash excellent doctor ``` Expected result: Excellent prints the runtime it resolved, the local database path, detected providers and coding CLIs, the keystore state, and whether telemetry is on. Telemetry is opt-in and off by default. Your agent still sends context to its model provider according to that agent's own settings. Excellent does not make that traffic local. Local data lives under `~/.excellent/`, or wherever `EXCELLENT_DATA_DIR` points. The database is a file; copy it to export it. There is no `excellent export` command and no `excellent uninstall`. Delete everything Excellent recorded: ```bash rm -rf ~/.excellent ``` That is irreversible. Source: https://www.excellent.so/docs/security-and-data --- # Run your own team sync server > Self-host the relay that keeps a team's workspaces in sync. This guide comes from a real end-to-end run, including the parts that are still hands-on. **Beta.** Excellent is local-first: every person's app keeps a complete copy of the workspace on their own machine. To keep those copies converged, your team runs one small always-on server, the **relay**. You host it. Excellent never hosts it and never sees your data. This page was written by provisioning a real relay in Docker and running real nodes against it: first on 2026-09-23, then re-run end to end on 2026-09-24 after the team-join fix landed. Every command and every error below was executed. Where a path was read in the product source but never run, it says so instead of implying otherwise. ## Read this before you start **A second device can now join an existing workspace, and it works from the command line only.** On the 2026-09-24 run, a clean second node received the first node's rows on its first tick, and its own rows reached the first node: both directions, with no hand-edited workspace id and no hand-edited cursor file. It is still beta, and two things will surprise you if nobody says them first. **1. The in-app switch cannot join an existing workspace.** *Settings → Team → Primary & sync* takes a relay address but no workspace id, so a node configured that way is always the **owner** of its own workspace. A teammate joining yours runs `excellent-mcp sync setup --team-bundle` in a terminal. See [§5](https://www.excellent.so/docs/team/self-host#5-adding-a-device-to-the-roster). **2. Adding a teammate is a hand edit of a file on your server.** There is no roster API and no invite link. They send you one line, you paste it into `devices.json`, and the relay picks it up without a restart. Removing the line revokes them the same way. **All of this is newer than the published CLI.** The join commands (`sync setup --team-bundle`, `sync invite-team`, `sync workspace-id`), the roster hot-reload that removes the restart, and `GET /health` all landed on 2026-09-24. Check before you start, on **every** machine and on the relay, because the relay installs the CLI from npm too: ```sh excellent-mcp sync # the subcommand list must include workspace-id and invite-team curl -s http://:9700/health # must answer {"ok":true,"service":"excellent-relay",…} ``` If either check comes back the old way, that install predates the fix: a second device on it will pair, sync, and pull zero rows, and roster edits will not take effect until you restart the relay. Upgrade first; [§9](https://www.excellent.so/docs/team/self-host#9-upgrading) has the steps. ## What syncs, and what does not The relay carries your **workspace**. The sync registry classifies 127 tables as team-shared, merged conflict-free so two people can edit the same record, online or offline, without losing anything. Genuine conflicts (money, pipeline stage, lifecycle status, archival, a person's identity) are surfaced for a human instead of silently resolved. Project rows were what the test run exercised: the owner's project reached the joining node on its first tick, and the joiner's project reached the owner. **Verification evidence stays on the machine that produced it.** `verification_runs` is local-only. Gate-run receipts, DoD judgment receipts and the webhook delivery records are excluded from the team exchange. The entire Universal Verification store (attempts, evidence snapshots, run manifests) lives in a separate database that the sync registry does not govern at all, and evidence blobs over 1 MiB are written to local disk. Searching all 127 team-synced table names for `verif|receipt|attempt|evidence|gate|proof` returns exactly one hit: `dod_criteria`, the *definition* of done, not any receipt. A teammate can see that you shipped something. They cannot open the proof behind it from their own laptop. If that is what you need, this server does not provide it. See [Receipts](https://www.excellent.so/docs/receipts) for what a receipt binds and how to verify one offline. ## 1. Prerequisites **Every machine that people *edit* on must be an Apple-Silicon Mac.** The conflict-free merge engine (cr-sqlite) is vendored for `darwin-arm64` only. On any other platform it throws rather than half-loading, and the in-app switch refuses with `platform-unsupported`. This is a hard limit, not a warning. `CRSQLITE_EXT` exists as an escape hatch for an extension you build yourself; no binary for any other platform ships. **The relay host does not need to be a Mac.** It is a plain Node HTTP server and never loads the merge engine. Verified: the relay ran in a `node:22-slim` Linux container for the whole run. On the relay host you need: - Docker with Compose, **or** Node and the published CLI. - Outbound access to the npm registry. There is no prebuilt relay image: the container runs `npm i -g @excellent-so/cli` on **every start**, which took about 24 seconds in testing and is the relay's only hard external dependency. A registry outage breaks every restart. - One TCP port reachable by every teammate (9700 by default). - A non-empty allowlist file. The relay exits with status 2 on an empty roster. - If it crosses the public internet: a domain name and the ability to set its DNS. On each teammate's Mac: Excellent installed, and **Node 22**. The run started on Node 24 and every command that opens the database failed: ```console $ node -v v24.20.0 $ excellent-mcp sync pair Error: The module '.../better-sqlite3-multiple-ciphers/build/Release/better_sqlite3.node' was compiled against a different Node.js version using NODE_MODULE_VERSION 127. This version of Node.js requires NODE_MODULE_VERSION 137. code: 'ERR_DLOPEN_FAILED' ``` The product pins `node: ">=22.0.0 <23"` and its `.nvmrc` says `22.23.2`. Everything below ran on Node 22.23.2. `EXCELLENT_DATA_DIR` is the one knob that moves a node's data directory; it defaults to `~/.excellent`. ## 2. What it costs Excellent charges nothing for this: no per-seat fee, no licence check. Your cloud provider bills you directly. AI features run on each teammate's own Claude Code subscription, so there is no central AI bill either. Provider list prices as the product records them, checked 2026-06. Confirm them in your provider's console; providers change prices. | Where it runs | Size | Estimated USD/month | |---|---|---| | A machine you already run (Docker) | Any | **$0** | | DigitalOcean | 1 vCPU · 512 MB | $4 | | DigitalOcean *(provisioner default)* | 1 vCPU · 1 GB | **$6** | | DigitalOcean | 1 vCPU · 2 GB | $12 | | DigitalOcean | 2 vCPU · 2 GB | $18 | | DigitalOcean | 2 vCPU · 4 GB | $24 | | DigitalOcean | 4 vCPU · 8 GB | $48 | | AWS Lightsail *(provisioner default)* | 512 MB · 2 vCPU | $5 | | AWS Lightsail | 1 GB · 2 vCPU | $7 | | AWS Lightsail | 2 GB · 2 vCPU | $12 | | Fly.io (usage-billed, from) | shared-cpu-1x · 256 MB | from $3 | | Fly.io (usage-billed, from) | shared-cpu-1x · 1 GB | from $6 | | A server you already have | Any | whatever you already pay | We do not publish a team size any of these fits. No benchmark has been run, so the only honest number is the price. Fly.io and Cloudflare appear here as places you could run a container by hand. The one-click provisioner does not target them (see [Not yet supported](https://www.excellent.so/docs/team/self-host#not-yet-supported)). ## 3. Stand up the relay ### The Docker path (this is what was run) The Compose file below is the product's own generated recipe, not something written by hand. The test run imported the product's `buildRelayComposeRecipe` and ran its output unchanged except for renaming the container, the volume and the published port so it could be torn down cleanly. In an empty directory on the relay host, create `devices.json`, the allowlist. It must not be empty. Start with your own device; §4 shows where that line comes from. ```json { "244805373dc4e11fb92164033546a2db": "MCowBQYDK2VwAyEAPuhy+xCh1I9u5+RqySduhDTG43PuQ0dSILyVcDFslu4=" } ``` `docker-compose.yml`: ```yaml services: relay: image: node:22-slim container_name: excellent-relay restart: unless-stopped ports: - "${RELAY_PORT:-9700}:9700" environment: EXCELLENT_DATA_DIR: /data command: ["sh", "-c", "npm i -g @excellent-so/cli && exec excellent-mcp sync relay serve --host 0.0.0.0 --port 9700 --devices /etc/excellent/devices.json --log /data/relay-log.ndjson"] volumes: - relay-data:/data - ./devices.json:/etc/excellent/devices.json:ro volumes: relay-data: name: excellent-relay-data ``` ```sh docker compose up -d docker compose logs -f relay ``` Startup-line excerpt from the 2026-09-24 run, about 24 seconds after the container starts. The npm install runs first, and the relay is unavailable for that whole window on every restart: ```console relay listening on http://0.0.0.0:9700 (1 device(s); log /data/relay-log.ndjson) ``` The old output also told every node to run the owner's setup command. Those obsolete instruction lines are omitted here. Use a relay URL reachable from every Mac; `0.0.0.0` above is a listen address, not the address to give teammates. The current CLI help gives this sequence for switching from single-owner to team sync: freeze writes, configure the owner, then join teammates with the owner's workspace bundle. Pair every device and add its public key to `devices.json`; retry pairing after the roster edit if the device was initially refused. Start the owner to seed the relay, then start Excellent on the other machines. Only the owner uses `setup --mode team`; teammates use `setup --team-bundle`. Follow [§4](https://www.excellent.so/docs/team/self-host#4-point-a-mac-at-the-relay) and [§5](https://www.excellent.so/docs/team/self-host#5-adding-a-device-to-the-roster) for the commands. This guidance was checked against current source; it is not additional terminal output captured in the dated run above. ### The in-app provisioner Excellent ships a one-click provisioner for the relay under **Settings → Team**. It generates exactly the Compose file above, seeds the roster with your own device, and polls until the relay answers. Its targets are **Docker, DigitalOcean and AWS Lightsail**; give it a domain and it also stands up Caddy for HTTPS. Its success message tells you to add each teammate's device to `devices.json` on the relay host. The relay re-reads that file when it changes, so additions and revocations take effect without a restart. The manual roster edit is still required; see §5. > The provisioner's screens were read in the product source but not clicked > through: walking them needs the full app running with an authenticated owner > session, which this run did not do. So the screen-by-screen walkthrough is not > published here. What is published is the Compose file it emits, which was > generated by its own builder and run. ### Running it without Docker The container runs this command; on a host with Node 22 and the CLI installed it is the same command: ```sh npm install -g @excellent-so/cli excellent-mcp sync relay serve \ --host 0.0.0.0 \ --port 9700 \ --devices /etc/excellent/devices.json \ --log /var/lib/excellent/relay-log.ndjson ``` | Flag | Meaning | |---|---| | `--host` | Bind address. `0.0.0.0` is what the generated recipe uses; a loopback bind is invisible to other machines. | | `--port` | Listen port. The recipe's default is 9700. | | `--devices ` | The JSON allowlist, `{ "": "" }`. Required, and must not be empty. | | `--log ` | The append-only change log. | Wrap the process in systemd, launchd or a supervisor so it survives a reboot. Only the containerised form was exercised in testing. ## 4. Point a Mac at the relay The `sync` subcommands that exist, from the CLI's own error message: ```console $ excellent-mcp sync --help unknown 'sync' subcommand: --help (try setup|join|adopt|status|now|pair|workspace-id|invite-team|relay serve) ``` If `workspace-id` and `invite-team` are missing from that list, your CLI predates the join fix. Upgrade (§9) before going further. Which `setup` you run depends on which side you are on. **The first machine, which owns the workspace everyone else will join:** ```sh excellent-mcp sync setup --mode team --relay http://:9700 excellent-mcp sync pair --relay http://:9700 ``` **A teammate joining that workspace** uses the bundle the owner sends them (§5) instead of `--mode team`: ```sh excellent-mcp sync setup --team-bundle excellent-mcp sync pair --relay http://:9700 ``` Running `--mode team` on the second machine is the mistake that costs you an afternoon: it configures that node as the owner of its *own* workspace, and the relay will correctly send it nothing, forever. The sync loop now says so out loud after a few empty ticks, naming the command that fixes it. `sync setup` writes `sync.env` in the data directory, mode 0600, holding `EXCELLENT_SYNC_MODE='team'` and `EXCELLENT_RELAY_URL`. Source it before starting Excellent. `sync pair` mints this machine's Ed25519 device key and prints the one line the relay operator needs: ```console $ excellent-mcp sync pair --relay http://:9700 device: 244805373dc4e11fb92164033546a2db public key (give this to the relay operator / devices.json): "244805373dc4e11fb92164033546a2db": "MCowBQYDK2VwAyEAPuhy+xCh1I9u5+RqySduhDTG43PuQ0dSILyVcDFslu4=" ``` Against a relay that has not been told about this device, pairing is refused: ```console pairing with http://:9700 failed: relay /pair/challenge → HTTP 404 {"ok":false,"reason":"unknown-device"} Is this device provisioned on the relay? (sync relay serve --devices …) ``` That refusal is the allowlist doing its job. > **This used to be refused outright.** Until 2026-09-24, `sync setup --mode team` > exited 2 with a solo-node identity message, because agent ids were derived from > the process tree and were not unique across machines. They are now salted with a > per-device fingerprint, and team sync is allowed. The `remote-worker` role is > still refused, and for a reason that has not changed: it needs a distributed > lock that does not exist. The product's own runbook names the in-app equivalent as **Settings → Team → Primary & sync → Team sync (conflict-free)**. That path was not walked in this run, so nothing about its screens is published here. As §5 says, it cannot join someone else's workspace in any case. ## 5. Adding a device to the roster There is no roster API. The relay's complete route list is `GET /health`, `POST /pair/challenge`, `POST /pair/verify`, `POST /register`, `POST /push`, `POST /pull`, and (only when webhooks are enabled) `POST /inbox/pull` and the `/hooks//` pair. Nothing else exists to call. Onboarding a teammate is therefore two separate things, and **both** have to happen. Doing only one is the failure mode this section exists to prevent. ### a. Send them the workspace, so their node knows what to join On the machine that already has the workspace: ```sh excellent-mcp sync invite-team ``` It prints one base64url line, then instructions. That line carries a relay URL and a workspace id and **nothing else**. It is not a credential, it grants no access, and on its own it gets a teammate nowhere. The command's own output says so. Send it however you would send an address. `excellent-mcp sync workspace-id` prints just the workspace id on stdout, one line, if you would rather assemble things yourself. The teammate runs, on their machine: ```sh excellent-mcp sync setup --team-bundle excellent-mcp sync pair --relay http://:9700 ``` `setup --team-bundle` points their node at your workspace instead of the one their install minted for itself, and it says so when it does: ```console aligned workspace 1 ('default') to the joined uid: 01A0D1DD84B0068570230B9458 -> 01A0D1DD7A3315DA4231358547 ``` It **refuses** rather than guessing when the answer is not obvious: if their install has more than one live workspace, or if the local workspace already holds real work that would silently become yours. The refusal names the flag that overrides it. ### b. Put their device on the roster, so the relay will talk to them `sync pair` prints their device line. You paste it into `devices.json` on the relay host: ```json { "244805373dc4e11fb92164033546a2db": "MCowBQYDK2VwAyEAPuhy+xCh1I9u5+RqySduhDTG43PuQ0dSILyVcDFslu4=", "68540a33f46110da273cb3ea789b0ed2": "MCowBQYDK2VwAyEAj1Xcz0Lo56gMNS7Z9GcSZr5CGrsPLRy0Z/H3QXPmElc=" } ``` **No restart.** The relay re-reads that file whenever it changes, before it answers a pairing request and before it resolves any bearer token, so the addition lands on their next attempt. It logs what moved: ```console [relay] roster reloaded from /etc/excellent/devices.json: +1 device(s) (68540a33…), -0 REVOKED; 2 device(s) now allowed. ``` A roster file that is unreadable or does not parse **keeps the last good roster** and says so on stderr. It never falls back to allowing everyone, and it never empties itself and locks the team out. Until their line is in the file, pairing is refused with `404 unknown-device`. That refusal is the allowlist doing its job. ### What a first join actually looks like The joining node receives the workspace on its first tick. It will also record a handful of **parked rows and a couple of conflicts**, every time, and that is expected rather than a sign of damage: both installs seeded a project called `general` and an identical set of loop and rubric rows at first run, so those genuinely-different rows collide on the same unique keys. One copy of each loses and is surfaced at **/database/conflicts** instead of being silently merged. Nothing is lost. It is alarming to look at and it is not caused by the join. ### About invitations There is no invite link, and `excellent-mcp sync join --invite` still cannot join a team relay. That bundle belongs to the older single-owner sync mode: it carries a database URL and a login coupon and has **no field for a relay address**, so `sync join` refuses a bundle without a database URL and then writes a single-owner configuration. `sync invite-team` above is a different thing despite the name. It hands over an address, not an authorisation, and the teammate is still refused by the relay until you have done step (b) by hand. ## 6. HTTPS and a domain The relay speaks plain HTTP and has no TLS of its own. Device authentication is a signed Ed25519 challenge, but the contents of your workspace are readable on the wire. **Plain HTTP is acceptable only on a trusted LAN or a VPN. Do not expose the relay directly to the internet.** For anything crossing the internet, terminate TLS in front of it. Giving the provisioner a **domain** does this for you: the relay stops publishing its own port and Caddy takes 80/443 with an automatic Let's Encrypt certificate. The generated service: ```yaml caddy: image: caddy:2 container_name: excellent-caddy restart: unless-stopped depends_on: - relay ports: - "80:80" - "443:443" command: caddy reverse-proxy --from relay.your-team.example --to relay:9700 volumes: - caddy-data:/data - caddy-config:/config ``` The manual step is DNS: point the domain's `A`/`AAAA` record at the host's public IP. Until it resolves, the provisioner reports the relay as `pending-dns` rather than reachable. Teammates then use the `https://` URL. > **Not exercised.** The Compose fragment above is what the product generates, and > it was read from the source, but no certificate was ever issued during testing. > Let's Encrypt's HTTP-01 challenge needs a public DNS name resolving to a > publicly reachable host on port 80, which a laptop behind NAT cannot provide. > Treat TLS as unverified until you have run it on a real host with a real domain. ## 7. Backups and restore Your first line of defence is that **every teammate's Mac already holds a complete copy** of the workspace. Cold backups cover what redundancy does not: someone deletes something everywhere, disk loss, ransomware. **On the relay host**, back up two things: the append-only change log, and `devices.json`. With the Docker recipe the log lives in a **named volume**, not in a home directory, so a file-level backup of the host will miss it entirely: ```sh docker cp excellent-relay:/data/relay-log.ndjson ./relay-log.ndjson cp devices.json ./devices.json ``` The log only grows, so a daily snapshot gives you a clean replay point. Stop the relay or snapshot the volume while copying so you do not capture a half-written line. **On each Mac**, back up the whole data directory: `data.db`, `local.db`, `device-key.pem`, `sync.env`, `keys/`. Do it before anything risky. > **Read this before you rely on a Mac backup.** `data.db` is encrypted at rest > with SQLCipher and **the key is not in the data directory**. It lives in the > machine's OS keychain. A copy of `data.db` alone cannot be opened on a different > machine. Either restore onto the same machine with its keychain intact, or > export and preserve the key alongside the backup. Restoring a single Mac end to end, including the keychain step, was not walked in testing, so no procedure for it is published here. ### Restoring the relay Verified by destroying the container **and** its volume, then restoring from a copy of the log: ```sh docker compose up -d docker cp ./relay-log.ndjson excellent-relay:/data/relay-log.ndjson docker compose restart relay ``` Real output after the restore, with all 303 entries back and devices reconnected: ```console relay listening on http://0.0.0.0:9700 (2 device(s); log /data/relay-log.ndjson) $ docker exec excellent-relay wc -l /data/relay-log.ndjson 303 /data/relay-log.ndjson ``` The product's runbook says a brand-new device can replay the whole history from cursor 0. That was tested against this restored log with a clean third node: it received nothing, for the same reason as §5. Do not plan on a fresh device as a recovery path. ## 8. Revoking a device Delete the device's line from `devices.json`. That is the whole procedure. **No restart.** The relay re-reads the roster when the file changes, and a removal drops the device's key, its live session, its bearer tokens and its routing scope together, so a token issued a second earlier stops working too. This was tested on 2026-09-24 with the relay process untouched: the removed device was refused, it never received a change written after its removal, and the container's restart count stayed at zero. ```console pairing with http://:9700 failed: relay /pair/challenge → HTTP 404 {"ok":false,"reason":"unknown-device"} ``` > **On a Linux relay host this is immediate.** The reload is triggered by the > roster file's modification time as the relay process sees it. On the macOS > Docker bind mount used for testing, that propagated about a second late, long > enough that a `/health` snapshot taken half a second after the edit still > counted the old device. A Linux host, which is the documented deployment, has no > such layer. The relay host's own device is a base entry that a file edit cannot remove, so you cannot lock yourself out of your own relay by editing the roster. The revoked machine keeps its own local copy of everything it had already synced. Revocation stops future exchange; it does not reach back and delete. ## 9. Upgrading Migrations are additive-only, so nodes on different builds can straddle a version. Take a backup first. **Each Mac:** ```sh npm install -g @excellent-so/cli@latest ``` **The relay** has no schema of its own: its state is the NDJSON log plus the roster file, and a newer version replays the log unchanged. Because the container installs the CLI at every start, restarting it already picks up the latest published version: ```sh docker compose restart relay ``` Keep `relay-log.ndjson` and `devices.json` in place across the upgrade. > Pin the version in production. The generated command is > `npm i -g @excellent-so/cli` with no version, which means the relay can change > underneath you on an unrelated restart. The published version at the time of > this run was `0.1.0` (checked 2026-09-23, and it is a point-in-time reading; > run `npm view @excellent-so/cli dist-tags` for today's). Pin > `@excellent-so/cli@` in the Compose command once you have one you > trust. ## 10. Is it up? Check that the container is running, that the port answers, and that the log is growing. ```sh docker compose ps docker compose logs --tail 20 relay curl -s http://:9700/health ``` `GET /health` answers with the roster size, the uptime and the head of the change log: ```console $ curl -s http://:9700/health {"ok":true,"service":"excellent-relay","devices":2,"uptime":17,"head":135} ``` A rising `head` is the real signal that changes are flowing. `devices` is the roster as the relay currently reads it, which is how you confirm an addition or a revocation landed without restarting anything. That endpoint is new as of 2026-09-24, and it names itself on purpose: the provisioner's "reachable" poll now requires `ok: true` and `service: "excellent-relay"` before it calls a relay up. It used to accept any status below 500, which is true of any HTTP server that happens to hold the port. > `excellent-mcp sync status` still cannot see team mode. It inspects only the > older single-owner replica flag, and returns `"state": "single-player"` on a > healthy, actively-syncing team node. It returns the same on a revoked one, and on a > machine that was never configured. Do not use it to diagnose the relay; use > `/health` and the node's own tick log. ## 11. Troubleshooting Every error below was produced during testing. **`ERR_DLOPEN_FAILED … NODE_MODULE_VERSION 127 … requires 137`** You are on Node 24. The native modules are built for Node 22. Switch (`nvm use 22`) and retry. **`sync setup: team-sync is refused: this node runs the SOLO-NODE identity implementation…`** Your CLI predates 2026-09-24. Agent ids are now salted with a per-device fingerprint and team sync is allowed; upgrade (§9). The same refusal on `--role remote-worker` is not a version problem and is still correct. **`SqliteError: no such table: main.document_targets`** when team sync starts The database was created by the CLI alone and is missing tables only the app creates. The sync registry classifies those tables as team-shared, so enabling the merge shadow fails closed. Launch Excellent once on that machine first. The terminal-only onboarding path dead-ends here. **`relay /pair/challenge → HTTP 404 {"ok":false,"reason":"unknown-device"}`** The device is not in `devices.json`, or the public key was truncated in transit, or the file no longer parses (a broken roster keeps the last good one and says so on the relay's stderr). Copy the whole `"": ""` line and check the relay's log for the `roster reloaded` line naming the device. No restart is needed to pick up the edit. **Everything reports healthy, and the other machine is empty** That node is not joined to your workspace. It is the owner of its own. It happens when the second machine was set up with `sync setup --mode team` instead of `sync setup --team-bundle`, or from the in-app switch, which cannot join. The node now says so itself after a few empty ticks: ```console [team-sync] 5 ticks and this node has NEVER received a row from a peer. … if you meant to JOIN a teammate's workspace, this node is NOT configured to (EXCELLENT_TEAM_WORKSPACE_UID is unset), so it is registering a scope the relay can never match and will correctly send you nothing, forever. ``` Re-run `sync setup --team-bundle ` on that machine (§5a). The sync cursors are reset automatically when the scope changes, once, so you do not have to touch `team-sync-state.json`. The warning fires only on a node that has never received anything, so a quiet, converged team does not trip it. **The joining machine filled up with conflicts on its first sync** Expected, and not damage (see §5). Both installs seed a project called `general` and an identical loop and rubric set, so those rows collide on their unique keys and one copy of each is parked at **/database/conflicts** rather than silently merged. **A node was working, then stopped after a relay restart** Relay sessions are in-memory. Every restart de-authenticates every device; the app's boot loop re-pairs. If it does not, that device is no longer on the roster. **The relay dies about 90 seconds after it starts** Your CLI predates 2026-09-24, when `sync relay serve` was added to the long-running exemption list in the CLI's own watchdog. Before that the watchdog killed it: silently survivable inside Docker, where the relay is PID 1, and fatal under systemd or in a plain shell. Upgrade (§9). **A node cannot reach the relay at all** In order: is the container up? Is it bound to `0.0.0.0` rather than loopback? Is the port open through the host firewall or cloud security group? Does the URL in `sync.env` match exactly, including scheme and port? For HTTPS, has the certificate actually issued? That is, does DNS point at the box? **The team-sync switch refuses on a machine** That machine is not an Apple-Silicon Mac. The refusal reason is `platform-unsupported`. There is no safe override; the merge engine is not built for it. **Relay restarts are slow, or fail with an npm error** Every start runs `npm i -g @excellent-so/cli`, which took about 24 seconds and needs the npm registry. If the registry is unreachable, the relay does not start. Pin the version (§9) and keep the host's outbound access open. ## Not yet supported Each of these was confirmed absent, not assumed. - **Team-visible verification evidence.** Attempts, gate-run receipts, DoD judgments and evidence blobs stay on the machine that produced them. - **Joining from the app.** The in-app team switch takes a relay address but no workspace id, so it can only ever make a node the owner of its own workspace. A teammate joins from the command line. See §5. - **An invitation flow for the relay.** `sync invite-team` hands over an address, not an authorisation, and the teammate is still added to the roster by hand. `sync join --invite` cannot join a team relay at all: it targets the older single-owner mode, and its bundle has no relay field. - **A roster API.** Adding or removing a device is a hand edit of `devices.json` on the relay host. The relay picks the edit up without a restart, but nothing makes the edit for you. - **Monitoring.** `GET /health` exists and tells you the roster size, uptime and log head, but nothing watches the relay for you and nothing will tell you it is down. - **Team-aware `sync status`.** - **A prebuilt relay image.** The relay reinstalls the CLI from npm on every start. - **Windows, Intel or Linux editing machines.** Apple Silicon only. The relay host itself can be Linux. - **Fly.io and Cloudflare provisioning targets.** The one-click provisioner covers Docker, DigitalOcean and AWS Lightsail. - **Owner transfer, and a guard against removing the last owner.** - **Device-key rotation.** Rotation exists for the older single-owner tokens. Relay device keys have no rotation path. - **mTLS and certificate pinning.** TLS terminated by your own reverse proxy is the model. - **SSO, SAML, SCIM, directory sync, audit-log export.** - **A managed or Excellent-hosted relay, per-seat pricing, or enterprise plans.** Every team self-hosts. There is nothing to buy here. - **A long-running integrity soak or scheduled two-machine CI.** Neither has been run, so this page publishes no uptime or durability claim. ## When something is wrong Send the relay's container status and last 20 log lines, and the current line count of `relay-log.ndjson`. Those three localize nearly everything in §11. ## Where to go next - [Security and data](https://www.excellent.so/docs/security-and-data): what is stored locally, and what leaves the machine. - [Receipts](https://www.excellent.so/docs/receipts): what a receipt binds, and how to verify one offline. - [CLI reference](https://www.excellent.so/docs/cli): every command the installed CLI actually has. Source: https://www.excellent.so/docs/team/self-host --- # Claude Code, Cursor and other agents > How Excellent attaches to the agent you already use: the file and the exact MCP entry the installer writes for each of the 28 agents it supports, and how to undo it. Excellent does not replace your agent and does not ship a model. It attaches to the agent you already run, as an MCP server, and checks what that agent produces. ## What the installer does to your machine > **If you installed 0.1.0 from npm.** `@excellent-so/cli` 0.1.0, the release npm > serves (checked October 23, 2026), predates `install-agent-mcps`. The installer > skips this step and says so, and no agent config is changed. You can register that > release's server by hand: find your agent in [the table below](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file) > and follow [Adding the entry by hand](https://www.excellent.so/docs/claude-code-and-cursor#adding-the-entry-by-hand). That release does not > have the verification tools this page lists: with no profile set it advertises 234 > tools, and none of the fourteen named under Claude Code is among them. The rest of > this page describes the next CLI release. When you install Excellent, it runs `install-agent-mcps`. That command finds **every** agent on your machine that it knows how to configure, and writes an `excellent` server entry into each one's own config file. Not just the agent you were using at the time. If it finds none, each `excellent` command you run checks again until one setup succeeds. Worth knowing before you run it: it knows 28 agents, so it may touch up to 27 vendor config files, plus OpenClaw's own registry through OpenClaw's command line. It adds one `excellent` entry and leaves the rest of each file's content alone. Every write goes to a temporary file that is then renamed into place. It refuses a config it cannot parse, and a TOML or YAML config whose `excellent` entry is ambiguous. JSON files are written back with two-space indentation; OpenCode and Kilo Code keep the comments in their JSONC. You can run it again safely: once an entry matches this install, it is left as it is. To do it yourself later, or after installing through npm: ```bash excellent install-agent-mcps ``` It prints one line for each agent it configured, naming the file it wrote, and asks you to restart those agents. Agents it cannot configure get no line; they are listed [below](https://www.excellent.so/docs/claude-code-and-cursor#agents-it-detects-but-does-not-configure). Add `--json` for the full report. ## Every agent and its file Names are as the installer prints them. Paths are for macOS, the one platform Excellent supports today. | Agent | File it writes | Entry | |---|---|---| | Amazon Q Developer CLI | `~/.aws/amazonq/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Amp | `~/.config/amp/settings.json` | [`amp.mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#amp) | | Antigravity CLI | `~/.gemini/config/mcp_config.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Auggie CLI | `~/.augment/settings.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Claude Code | `~/.claude.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Cline CLI | `~/.cline/data/settings/cline_mcp_settings.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Codebuff | `~/.agents/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Codex | `~/.codex/config.toml` | [`[mcp_servers.excellent]`](https://www.excellent.so/docs/claude-code-and-cursor#toml-codex-grok-build-and-open-interpreter) | | Crush | `~/.config/crush/crush.json` | [`mcp`](https://www.excellent.so/docs/claude-code-and-cursor#crush) | | Cursor Agent | `~/.cursor/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Devin CLI | `~/.config/devin/mcp_config.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Factory Droid | `~/.factory/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Gemini CLI | `~/.gemini/settings.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | GitHub Copilot CLI | `~/.copilot/mcp-config.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | GitLab Duo CLI | `~/.gitlab/duo/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | goose | `~/.config/goose/config.yaml` | [`extensions`](https://www.excellent.so/docs/claude-code-and-cursor#goose) | | Grok Build | `~/.grok/config.toml` | [`[mcp_servers.excellent]`](https://www.excellent.so/docs/claude-code-and-cursor#toml-codex-grok-build-and-open-interpreter) | | Hermes Agent | `~/.hermes/config.yaml` | [`mcp_servers`](https://www.excellent.so/docs/claude-code-and-cursor#hermes-agent) | | Junie CLI | `~/.junie/mcp/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Kilo Code CLI | `~/.config/kilo/kilo.json` | [`mcp`, a `local` server](https://www.excellent.so/docs/claude-code-and-cursor#opencode-and-kilo-code-cli) | | Kimi Code CLI | `~/.kimi-code/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Kiro CLI | `~/.kiro/settings/mcp.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Open Interpreter | `~/.openinterpreter/config.toml` | [`[mcp_servers.excellent]`](https://www.excellent.so/docs/claude-code-and-cursor#toml-codex-grok-build-and-open-interpreter) | | OpenClaw | OpenClaw's own registry, through `openclaw mcp` | [`openclaw mcp set`](https://www.excellent.so/docs/claude-code-and-cursor#openclaw) | | OpenCode | `~/.config/opencode/opencode.json` | [`mcp`, a `local` server](https://www.excellent.so/docs/claude-code-and-cursor#opencode-and-kilo-code-cli) | | Qwen Code | `~/.qwen/settings.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | | Rovo Dev CLI | `~/.rovodev/mcp_config.json` | [`mcpServers`, with `transport`](https://www.excellent.so/docs/claude-code-and-cursor#rovo-dev-cli) | | Windsurf Cascade | `~/.codeium/windsurf/mcp_config.json` | [`mcpServers`](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry) | Some agents move their config with an environment variable, and Excellent follows it: `GROK_HOME` (Grok Build), `HERMES_HOME` (Hermes Agent), `COPILOT_HOME` (GitHub Copilot CLI), `KIMI_CODE_HOME` (Kimi Code CLI), `CLINE_DATA_DIR` (Cline CLI), `ROVODEV_HOME` (Rovo Dev CLI), `OPENCODE_CONFIG` (OpenCode), `GLAB_CONFIG_DIR` (GitLab Duo CLI), and `OPENCLAW_CONFIG_PATH` or `OPENCLAW_STATE_DIR` (OpenClaw). `XDG_CONFIG_HOME` moves every `~/.config` path above, and GitLab Duo CLI's file to `$XDG_CONFIG_HOME/gitlab/duo/mcp.json`. OpenCode uses `opencode.jsonc` when that is the only one of the two files that exists. Rovo Dev CLI uses the `mcpConfigPath` set in `~/.rovodev/config.yml`, and otherwise falls back to an existing legacy `mcp.json`. ### Agents it detects but does not configure Excellent recognizes these agents but writes no config for them: | Agent | Why not | |---|---| | Zed Agent | Zed's documented MCP server settings do not establish which working directory the server runs in, so Excellent will not create an entry it cannot verify as portable. | | Mistral Vibe | The same: its MCP schema does not establish the working directory. | | OpenHands CLI | The same: there is no reviewed way to persist the server's working directory. | | Aider | It has no native MCP client. | | Continue CLI | It takes MCP servers from its own platform config, not a local file. | | Grok Bot | It is a desktop control surface. The work runs on Grok Bot's cloud computer. | | Jules Tools | The work runs in a cloud VM, not on your machine. | | mini-SWE-agent | No native MCP client is documented. | | Pi Agent | It ships no native MCP client, on purpose. | | Plandex | No persistent MCP config is documented. | | Roo Code CLI | Its MCP settings are per editor profile or per project, so there is no single file to write. | | Sourcegraph Cody CLI | No persistent MCP config is documented. | | SWE-agent | It uses its own tool bundles rather than a persistent MCP config. | | Trae Agent | Its MCP map lives in a project YAML you choose, so there is no single file to write. | | Warp Oz | It takes MCP servers per run (`--mcp`) or from Warp's cloud, with no local file. | ## The entry it writes Two paths appear in every entry. `/absolute/path/to/node` is the Node.js that ran the installer; `node -p process.execPath` prints it. `/absolute/path/to/@excellent-so/cli/dist` is the installed package's `dist` folder, which holds `server.js`; `echo "$(npm root -g)/@excellent-so/cli/dist"` prints it. An npm or `curl | sh` install writes no environment variables. ### The standard `mcpServers` entry Claude Code, Cursor Agent, Gemini CLI, GitHub Copilot CLI, Windsurf Cascade, Amazon Q Developer CLI, Antigravity CLI, Auggie CLI, Cline CLI, Codebuff, Devin CLI, Factory Droid, GitLab Duo CLI, Junie CLI, Kimi Code CLI, Kiro CLI and Qwen Code all get the same stdio block: ```json { "mcpServers": { "excellent": { "type": "stdio", "command": "/absolute/path/to/node", "args": ["/absolute/path/to/@excellent-so/cli/dist/server.js"], "cwd": "/absolute/path/to/@excellent-so/cli/dist", "env": {} } } } ``` ### Other JSON entries #### Rovo Dev CLI The same block, but Rovo Dev names stdio with `transport` instead of `type`: ```json { "mcpServers": { "excellent": { "transport": "stdio", "command": "/absolute/path/to/node", "args": ["/absolute/path/to/@excellent-so/cli/dist/server.js"], "cwd": "/absolute/path/to/@excellent-so/cli/dist", "env": {} } } } ``` #### Amp Amp's map is one key with a dot in its name, `amp.mcpServers`, and the entry has no `type`: ```json { "amp.mcpServers": { "excellent": { "command": "/absolute/path/to/node", "args": ["/absolute/path/to/@excellent-so/cli/dist/server.js"], "cwd": "/absolute/path/to/@excellent-so/cli/dist", "env": {} } } } ``` #### Crush Crush keeps its servers under `mcp`: ```json { "mcp": { "excellent": { "type": "stdio", "command": "/absolute/path/to/node", "args": ["/absolute/path/to/@excellent-so/cli/dist/server.js"], "cwd": "/absolute/path/to/@excellent-so/cli/dist", "env": {} } } } ``` #### OpenCode and Kilo Code CLI A `local` server, with Node and the server script together in one `command` array: ```json { "mcp": { "excellent": { "type": "local", "command": [ "/absolute/path/to/node", "/absolute/path/to/@excellent-so/cli/dist/server.js" ], "cwd": "/absolute/path/to/@excellent-so/cli/dist", "environment": {}, "enabled": true } } } ``` If the file already nests its servers under `mcp.servers` (OpenCode v2), the entry goes there, with `"disabled": false` in place of `"enabled": true`. ### TOML: Codex, Grok Build and Open Interpreter ```toml [mcp_servers.excellent] command = "/absolute/path/to/node" args = [ "/absolute/path/to/@excellent-so/cli/dist/server.js" ] cwd = "/absolute/path/to/@excellent-so/cli/dist" env = {} ``` ### YAML entries #### Hermes Agent ```yaml mcp_servers: excellent: command: "/absolute/path/to/node" cwd: "/absolute/path/to/@excellent-so/cli/dist" args: - "/absolute/path/to/@excellent-so/cli/dist/server.js" env: {} ``` #### goose goose gets a stdio extension: ```yaml extensions: excellent: type: "stdio" name: "Excellent" enabled: true cmd: "/absolute/path/to/node" cwd: "/absolute/path/to/@excellent-so/cli/dist" args: - "/absolute/path/to/@excellent-so/cli/dist/server.js" env_keys: [] envs: {} timeout: 300 ``` ### OpenClaw OpenClaw owns its config file, so Excellent does not edit it. It registers through OpenClaw's own command instead: ```bash openclaw mcp set excellent '{"command":"/absolute/path/to/node","args":["/absolute/path/to/@excellent-so/cli/dist/server.js"],"cwd":"/absolute/path/to/@excellent-so/cli/dist","env":{},"transport":"stdio"}' ``` The only environment keys Excellent will ever own or overwrite in an entry are `EXCELLENT_*`, `INTEGRATIONS_CRON_SECRET` and `ELECTRON_RUN_AS_NODE`. In the JSON entries, anything else you put in `env` is preserved when Excellent rewrites the entry. When it rewrites a TOML, YAML or OpenClaw entry it writes only its own keys, so keep your own variables out of those. Restart or reload the agent after installing, or it will not see the new server. ## Adding the entry by hand With 0.1.0, or if you would rather not let the installer write agent configs: 1. Find your agent in [the table above](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file) and open its file. Create it if it does not exist. 2. Add the entry for its shape from [The entry it writes](https://www.excellent.so/docs/claude-code-and-cursor#the-entry-it-writes) inside the map that is already there, next to any other servers. Do not replace the file. For OpenClaw, run the `openclaw mcp set` command instead. 3. Replace the two placeholder paths with the output of `node -p process.execPath` and `echo "$(npm root -g)/@excellent-so/cli/dist"`. 4. Restart the agent. Excellent does not write an entry for Zed, Mistral Vibe or OpenHands, and does not offer one to copy, for the reason in [the table above](https://www.excellent.so/docs/claude-code-and-cursor#agents-it-detects-but-does-not-configure). ## Claude Code After a restart, Claude Code lists the Excellent tools under the `excellent` server. By default it sees **14 tools**, not the full registry. The default profile is `product`, which is the verification journey and nothing else: - `verification_v2_commands`, `verification_v2_dispatch` - `universal_verification_status`, `universal_verification_open_attempt`, `universal_verification_submit_attempt`, `universal_verification_propose_amendment`, `universal_verification_request_review`, `universal_verification_trust_root` - `session_create`, `session_attach`, `session_event`, `session_evidence_record`, `session_trajectory`, `session_claim` Set `EXCELLENT_MCP_PROFILE=core` or `full` in the server's `env` block for a wider belt. Treat the profile as a menu rather than a fence: the enforced boundary is `EXCELLENT_MCP_ALLOW`, which is checked at dispatch. Both are `EXCELLENT_*` keys, which Excellent owns, so the next `install-agent-mcps` finds an entry that differs from the one it would write and writes it back without them. Set them again after a re-run. In use, you do not call these yourself. The agent opens an attempt against a Work item, submits its output, and reads the status back. Excellent records the attempt and, separately, a verifier reaches a verdict, which the agent cannot issue for itself. See [How verification decides](https://www.excellent.so/docs/how-verification-decides). ### Hooks and skills > **Only in a monorepo checkout.** If you installed with `curl | sh` or > `npm i -g @excellent-so/cli`, you get MCP registration and nothing else. No hooks, > no skills. The hook and skill wiring needs an Excellent monorepo root and is skipped > without one. Inside a checkout, `excellent-mcp install-skills` additionally writes three Claude Code hooks into `/.claude/settings.json`: `SessionStart`, `Stop`, and `PreToolUse` matching `Bash`. There is no `PostToolUse` hook. It only adds missing entries; your own hooks are left alone, and an unparseable settings file is refused rather than rewritten. What they do: the session hook reaps MCP server processes whose host died. The stop hook reminds you if you still hold in-progress work. The Bash hook is the interesting one: it blocks a `git commit` or `git push` when the agent holds no claim, and blocks bulk staging (`git add -A`, `git commit -a`) in a shared working tree. It also symlinks the shipped skill packs into `~/.claude/skills/`. That mirror is for Claude Code's UI only; the `skill_list` and `skill_get` tools read the repo directly and do not need it. ## Cursor Cursor gets **MCP and only MCP**: an `excellent` entry in `~/.cursor/mcp.json`, in [the standard shape](https://www.excellent.so/docs/claude-code-and-cursor#the-standard-mcpservers-entry). The installer calls it Cursor Agent. Excellent writes no `.cursorrules`, no `.cursor/rules/*.mdc`, no hooks and no project-scoped config. Cursor is detected from the `cursor-agent` binary or from Cursor.app being installed. The bare `agent` alias is deliberately not accepted as evidence, since it belongs to too many things. Separately, Cursor can be a backend for Excellent's own REPL: ```bash excellent chat --cli cursor ``` That routes each turn through your local `cursor-agent` using its auth, with no API key. `claude` is the first-class backend there. It gets the full belt and resumable sessions; the others run headless. ## Repository context `excellent init` writes `AGENTS.md` and `.excellent/context.md` into the repository. The Excellent chat reads `AGENTS.md`, `CLAUDE.md` and `.excellent/context.md` per directory. Claude Code reads `CLAUDE.md` and `AGENTS.md` on its own. Keeping the repository facts in those files means both tools see the same thing. ## Undoing it To disconnect one agent, delete the `excellent` entry from its file in [the table above](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file), then restart the agent. For OpenClaw, run: ```bash openclaw mcp unset excellent ``` Excellent does not put a removed entry back by itself. After the first setup that succeeds, it records that in `~/.excellent/agent-mcp-first-use-v1.json`, and from then on it writes agent configs only when you ask it to, as with `install-agent-mcps`. In a source checkout, `excellent-mcp uninstall-skills` removes Excellent's own symlinks and the exact `excellent` entries it added. It never deletes a real directory and never touches another vendor's entries. In an npm or `curl | sh` install it does not recognize the installer's entries as its own and leaves them in place, so remove them by hand as above. ## Related - [MCP](https://www.excellent.so/docs/integrate/mcp): connecting anything else that speaks MCP. - [Quickstart](https://www.excellent.so/docs/quickstart): the whole path, end to end. - [CLI reference](https://www.excellent.so/docs/cli): every command and environment variable. Source: https://www.excellent.so/docs/claude-code-and-cursor --- # MCP > Connect Excellent to agent tools through MCP, so another tool can read or operate Excellent through the same checked interfaces. Setup and undo. Use MCP when another tool needs to read or operate Excellent through the same checked interfaces. ```bash excellent-mcp install-skills ``` Expected result: the MCP server tools are available to the agent you configured on this machine. If installation fails, check whether the local service is running and whether the target agent supports MCP. Undo the connection by removing the MCP server from the target agent's config. Source: https://www.excellent.so/docs/integrate/mcp --- # CLI reference > Every command in the CLI's command table: setup, account, work, chat and preferences in excellent, plus the excellent-mcp companion binary. Some are newer than the release on npm. Two binaries ship in one package, `@excellent-so/cli`: - **`excellent`**: the command you use. Run bare, it opens the chat REPL. - **`excellent-mcp`**: the companion binary. Run with no arguments it *is* the MCP server, speaking stdio to whichever agent registered it. > **Newer than the release on npm.** `@excellent-so/cli` 0.1.0, the version npm > serves (checked October 23, 2026), predates part of this page. It has no > `install-agent-mcps`, `run`, `verify` or `receipt verify` in either binary, and no > `excellent-mcp verification`; it answers each with `unknown command` and exit 2. The > other commands on this page are in 0.1.0. This note comes out with the next release. Everything below is read from the CLI's command table. If a command is not on this page, it is not part of the supported surface. `excellent-mcp` carries a long tail of internal operator verbs that are deliberately absent from `excellent --help` and from here. Run `excellent --help` for the same list on your machine, and `excellent --help` for a command's own flags. ## Getting set up | Command | What it does | |---|---| | `excellent --version` | Print the CLI version. `-v` is an alias. | | `excellent init` | Scaffold `AGENTS.md` and `.excellent/{context.md,cli.toml}` in the current directory. | | `excellent doctor` | Health read: runtime, database, providers, keys, coding CLIs, `PATH`, versions. | | `excellent inspect` | What resolves here: tool-belt size, skills, MCP registration, context files. | | `excellent install-agent-mcps` | Register the Excellent MCP server in every detected agent it knows how to configure: 28, listed with their files in [Claude Code, Cursor and other agents](https://www.excellent.so/docs/claude-code-and-cursor#every-agent-and-its-file). `--json` prints the full report. | | `excellent update [--dry-run]` | Self-update. Detects npm, binary or source-checkout installs. | | `excellent completions ` | Print a shell completion script. | `excellent init` takes `--force` to overwrite an existing `AGENTS.md`. It has no `--help`: passing one runs the command. ## Account and consent | Command | What it does | |---|---| | `excellent login [--as HANDLE] [--workspace ID\|SLUG]` | Bind this terminal to a local workspace and account. Seals a device token in the OS keychain, never in the database. | | `excellent whoami` | Show the bound account and workspace. | | `excellent logout` | Clear the binding. | | `excellent accounts` | List bound accounts. | | `excellent authorize [--all\|--minimal\|--scopes a,b] [--status] [--revoke]` | Choose what the tool belt may do: `read`, `write`, `terminal`, `admin`. The belt advertises only what you granted. | ## Doing work ### `excellent run` ``` excellent run --agent --task [--json] ``` Runs a role against an existing task. Both flags are required. - `--agent`: the role name. - `--task`: an existing **numeric task id**. A free-text objective is validated and then refused with a structured hint: the underlying pathway targets task ids only, so file the objective as a task first. - `--json`: emit an envelope on stdout: `{ok, command, agent, taskIdOrObjective, taskId?, correlationId, exitCode?}`. - Other agent-run flags (`--goal`, `--cli`, `--dry-run`, `--loop`) pass through. Every invocation prints a `correlationId` on stderr. Exit codes: **0** completed · **1** not completed · **2** usage or goal refusal. ### `excellent verify` ``` excellent verify [--tenant T] [--bearer-file PATH] [--idempotency-key KEY] [--json] ``` Starts or reads a verification. A work id starts one; a `ver_*` id reads the status of an existing one. Defaults to tenant `local`; supply a real credential with `--bearer-file`. Also accepts `--token-id`, `--scopes` and `--now`. Exit codes: **0** ok · **1** runtime · **2** usage or validation · **3** authorization · **4** not found. ### `excellent-mcp receipt verify` ``` excellent-mcp receipt verify [--offline] [--pubkey PEM_FILE] [--key-id ID] ``` `excellent receipt verify` is a different command with the same name: it is the v0 alias of `excellent-mcp evidence verify`, it takes an EvidenceBundle rather than a receipt pack, and it **exits 2 on any flag but `--offline` and `--json`**. If you pass `--pubkey` to it you get a usage error, not a verification. Re-derives a receipt offline, with no database and no network. Checks the canonical payload hash, the Ed25519 envelope signature, the canonical binding, anchoring against the operator-pinned public key, and the ledger anchor. Prints one JSON line. `--pubkey` pins the issuer's Ed25519 public key as SPKI PEM; its sha256 fingerprint is the out-of-band anchor. `--offline` is accepted because the verifier already is offline. Exit codes: **0** verified · **2** usage · **8** unverifiable or invalid. ## Chat Bare `excellent` opens the REPL. It drives the same tool belt over any model provider, or through a coding CLI you already have. | Flag | Meaning | |---|---| | `--provider ID` | An HTTP provider plus key: `anthropic` (default), `openai`, `openrouter`, `deepseek`, `groq`, `together`, `mistral`, `xai`/`grok`, `ollama`, `lmstudio`, `vllm`. | | `--cli NAME` | Route through a local coding CLI instead, using *its* auth and no API key: `claude`, `codex`, `gemini`, `grok`, `xai`, `cursor`. `claude` is first-class: full belt and resumable. | | `--model ID` | Model id. Defaults per provider. | | `--base-url URL` / `--transport openai\|anthropic` | Custom endpoint and wire dialect. | | `--tools P` | Tool-belt breadth: `product` (default), `search`, `core`, `full`. | | `--allow a,b,c` | Advertise exactly these tools, overriding `--tools`. | | `--no-tools` | Connect no belt at all. | | `-p "message"` | One-shot: run a single turn and exit. | | `--json` / `--output-format text\|json\|stream-json` | Machine-readable output, for CI and pipes. | | `--resume [id]` / `--continue` | Continue a saved session. `excellent chat sessions` lists them. | | `--dry-run` | Resolve the provider and connect the belt, print JSON, exit. No model call. | | `--max-turns N`, `--budget-tokens N`, `--budget-wall-ms N` | Per-message ceilings. Turn default is 40. | Keys come from the environment (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, …), or skip keys entirely with `--cli`. Sessions auto-save to `~/.excellent/chat-sessions/`. `excellent chat --help` prints the full flag list. ## Preferences and appearance | Command | What it does | |---|---| | `excellent config ` | CLI preferences in `~/.excellent/cli.toml`. A project `./.excellent/cli.toml` overrides it. | | `excellent theme [name\|preview\|bg\|spinner]` | Accent theme, dark/light/auto background, spinner family. | | `excellent runtime [status\|bootstrap]` | Inspect or bootstrap the companion/standalone runtime. Flags: `--standalone`, `--companion`, `--data-dir DIR`, `--json`. | | `excellent bug` | Report a bug. Flags: `--title`, `--description`, `--severity`, `--email`, `--yes`. | ## The companion binary | Command | What it does | |---|---| | `excellent-mcp` | Run the MCP server on stdio. This is what an agent launches. | | `excellent-mcp install-agent-mcps` | Same as `excellent install-agent-mcps`. | | `excellent-mcp install-skills` | Source-checkout only: mirror skills into `~/.claude/skills`, register the MCP server, write project hooks, and install the `~/.local/bin` shim. | | `excellent-mcp uninstall-skills` | Reverse it, in a source checkout. Removes only Excellent's own symlinks and MCP entries. It does not remove the entries an npm install wrote; delete those by hand. | | `excellent-mcp status` | Show what is currently installed. | | `excellent-mcp verification commands` | List the canonical verification API commands and their MCP names. | | `excellent-mcp verification observables` | List the observables this tier can decide, and the rules for binding one. | ## Environment variables | Variable | Effect | |---|---| | `EXCELLENT_DATA_DIR` | Override `~/.excellent/` (the local database and sessions). | | `EXCELLENT_HOME` | Path to an Excellent monorepo root. Auto-detected when possible. | | `EXCELLENT_MCP_PROFILE` | Tool-belt profile the MCP server advertises. Default `product`. | | `EXCELLENT_MCP_ALLOW` | The actual security boundary, enforced at dispatch. The profile is a menu; this is the fence. | | `CLAUDE_CONFIG_DIR` | Override `~/.claude/`. | For installer variables (`EXCELLENT_CLI_URL`, `EXCELLENT_SOURCE`, `EXCELLENT_DRY_RUN`, `EXCELLENT_ALLOW_UNVERIFIED`), see [Install](https://www.excellent.so/docs/install). ## Commands that do not exist Asked for often, and absent on purpose: - `excellent uninstall`: remove it by hand. [Install](https://www.excellent.so/docs/install) has the steps. - `excellent export`: not implemented. The local database is a file in `~/.excellent/`. - `excellent status`: `excellent-mcp status` reports *installer* state, not workspace state. Use `excellent doctor`. An unknown command exits **2** and tells you to run `--help`. Source: https://www.excellent.so/docs/cli --- # Sample receipt > A real signed receipt, field by field: what each part binds, how to verify it, and what a valid signature does and does not establish. A receipt is what a verification leaves behind. It binds the work, the evidence, the policy and the verdict into one document, hashes that document canonically, and signs the hash. Its purpose is that you do not have to trust the machine that produced it. The document below is the **real release receipt** shipped in the Excellent repository. It is synthetic release evidence rather than a customer's codebase, but it is genuine output of the receipt pipeline, with a genuine signature. **Its verdict is `FAIL`, and we show it on purpose.** The release had fifteen gates, listed in `checkResultIds`. Ten passed. Three failed (`G01`, `G02`, `G13`), one was still running (`G09`) and one was blocked by something outside the system (`G12`). With gates failing, the release verdict is `FAIL`. We publish the receipt as it was issued rather than picking a flattering one, because that is what a receipt is for: it records the verdict that was reached. A valid signature proves the document has not been altered. It does not make the verdict a good one. ## The document ```json { "schemaVersion": 1, "serviceVersion": "2026-08-16.v1", "receipt": { "schemaVersion": 1, "id": "receipt_v0_release_4df3dd89b3e4d0a0", "tenantId": "tenant_v0_release", "runId": "run_v0_release_e44941560b9f", "parentReceiptId": null, "status": "FAIL", "checkResultIds": [ "G00:PASS", "G01:FAILING", "G02:FAILING", "G03:PASS", "G04:PASS", "G05:PASS", "G06:PASS", "G07:PASS", "G08:PASS", "G09:RUNNING", "G10:PASS", "G11:PASS", "G12:BLOCKED_EXTERNAL", "G13:FAILING", "G14:PASS" ], "evidenceHashes": [ "sha256:437284d411f1a9fc3aebf0050317c4eb7bf6fb17db85c683713237b307e52ee3", "sha256:3b9aed882a0f9ba291fb1a76739fc1fc6b6352746619be9d0eb83cb64d7859fe", "sha256:405989c4e96fc8cc1121b4ea331f9a7d72daa959e07748f85c608a58957547bd", "sha256:2ef2a674ded9b95de4e77ff8785a0470e00c8043cd3b3ca68013557463c8e583", "sha256:93bfdab20f663dc0a0f32740c66a5f0df4c31a471e2fa3751e76be001d506491" ], "policyVersionRefs": [ "policy_v0_release@2026.08" ], "engineVersion": "v0-release@1", "issuedAt": "2026-09-16T12:37:22.257Z", "signature": { "keyId": "v0-ed25519-2026-08", "algorithm": "ed25519", "value": "ed25519:COSkZ5UROiogqWwpWE6S9QfoQz5kdH8MTvmP6huRA25sqn3oO4lsgPpAEZxsVCLNpH2Pz2Cy6Nd9iv04x2LiCw" } }, "binding": { "predicate": "excellent.verification.agent-work-receipt.v1", "workItemId": "work_cand_9b618ed027764ef4bccf2a64_df-001", "workSnapshotId": "wsnap_work_cand_9b618ed027764ef4bccf2a64_df-001", "attemptId": "promo_23247c38c4574c96893c2036:release", "contractId": "verification-api/2026-08-16.v1", "evidenceSnapshotId": "evsnap_v0_release_4df3dd89b3e4d0a0", "evidenceGraphDigest": "sha256:c4395399c0c283d4f36877228b5de86860e8c39f0923e4c7401592f3781ef68f", "obligationResultIds": [ "obligation_lift_lcb_met", "obligation_no_hard_gate_regression", "obligation_offline_receipt_verifies" ], "evaluatorRefs": [ "evaluator:evidence-integrity", "evaluator:paired-bootstrap", "evaluator:release-gate-audit" ], "determinationId": "determination_v0_release_FAIL_4df3dd89b3e4d0a0", "validityStateId": "validity_v0_release", "enforcementDecisionId": "enforcement_promote_promo_23247c38c4574c96893c2036", "enforcementEffectIds": [ "effect_promotion_promo_23247c38c4574c96893c2036", "effect_release_proof_recorded" ], "reviewDecisionIds": [ "review_approval_release-approver" ], "outcomeObservationIds": [ "outc_e970c8c1732b49ca9d510c5f" ], "auditEventIds": [ "audit_v0_release_issue" ], "buildRef": "sha256:405989c4e96fc8cc1121b4ea331f9a7d72daa959e07748f85c608a58957547bd", "subject": { "treeHash": "git-tree:fcd3f669b7d3a9a46469e18a7959cc14ea17b65d", "commitSha": "commit:e44941560b9fe6713d82ec2d0573eb54983877c0", "buildId": "build:v0-release-e44941560b9f", "imageDigest": "sha256:e09f241b4fe78ae9d769e0d3c3caaab5", "tenantId": "tenant_v0_release", "environmentId": "env_local_release_probe", "policyDigest": "policy_v0_release@2026.08" }, "sourceCommit": "e44941560b9fe6713d82ec2d0573eb54983877c0", "buildDigest": "sha256:405989c4e96fc8cc1121b4ea331f9a7d72daa959e07748f85c608a58957547bd", "uiBundleDigest": "sha256:2ef2a674ded9b95de4e77ff8785a0470e00c8043cd3b3ca68013557463c8e583", "databaseSchemaVersion": "1052", "contractSchemaVersion": "verification-api/2026-08-16.v1", "cliArtifactDigest": "sha256:93bfdab20f663dc0a0f32740c66a5f0df4c31a471e2fa3751e76be001d506491", "experimentId": "exp_1094686b4cef40bca465ffb6", "candidateId": "cand_9b618ed027764ef4bccf2a64", "baselineCandidateId": "cand_5a509ab3c2024bd689b83fb4", "agentVersionId": "agentver_76d967647cdb4aca9a871f08", "baselineAgentVersionId": "agentver_8296005ce5cb4a6fb01f27be", "taskDistributionId": "taskdist_13fa273bfe104d1bbe3b061b", "promotionId": "promo_23247c38c4574c96893c2036", "promotionReceiptId": "receipt_promo_4fcc27f3404047806e786bf3", "releaseWorkItemId": "work_cand_9b618ed027764ef4bccf2a64_df-001", "comparison": { "comparisonId": "cmp_36cc7f27914849a0a9dab20f", "observedLiftPct": 25, "lcb": 15.625, "methodVersion": "paired-bootstrap/v1", "pairsCount": 32, "stats": { "lcb": 15.625, "observedLiftPct": 25 }, "holdoutPairsCount": 32, "hardGateRegressions": [] }, "profilesTested": [ "embedded", "local", "digitalocean-sandbox", "digitalocean-live" ], "interfacesTested": [ "rest", "cli", "mcp", "browser", "webhook", "worker" ], "testsSummary": { "suites": 58, "failures": 5 }, "approvals": [ { "approver": "release-approver", "method": "explicit", "decidedAt": "2026-09-16T12:37:22.190Z" } ], "deployment": { "localStackProbe": "unavailable(no passing local-stack gate)", "digitaloceanSandbox": "unavailable(no passing digitalocean-sandbox gate)", "digitaloceanLive": "unavailable(no passing digitalocean-live gate note)" }, "knownOutcomes": [ { "status": "verified" } ] }, "payloadHash": "sha256:ea48ebc0c36dced1cefd90ae750220594dd2e3124cc67c6ecb9ac524a2b55ca9", "signature": { "keyId": "v0-ed25519-2026-08", "algorithm": "ed25519", "value": "ed25519:COSkZ5UROiogqWwpWE6S9QfoQz5kdH8MTvmP6huRA25sqn3oO4lsgPpAEZxsVCLNpH2Pz2Cy6Nd9iv04x2LiCw" }, "publicVerification": { "algorithm": "sha256", "payloadHash": "sha256:ea48ebc0c36dced1cefd90ae750220594dd2e3124cc67c6ecb9ac524a2b55ca9", "signatureAlgorithm": "ed25519", "keyId": "v0-ed25519-2026-08", "receiptId": "receipt_v0_release_4df3dd89b3e4d0a0", "runId": "run_v0_release_e44941560b9f", "tenantId": "tenant_v0_release", "issuedAt": "2026-09-16T12:37:22.257Z", "parentReceiptId": null } } ``` ## What each part is for **`receipt`** is the verdict itself. `status` is one of the six verification states. `checkResultIds` names every check that was folded into it, `evidenceHashes` pins the evidence those checks read, and `policyVersionRefs` pins the policy version they were judged under. `parentReceiptId` is null here; on a re-verification it points at the receipt this one supersedes, because receipts are append-only and a verdict is never edited in place. **`binding`** is the part that makes the verdict mean something. It names the exact work item, attempt, contract and evidence snapshot, plus `evaluatorRefs` (which evaluators decided) and `subject`, which pins the artifact: a git tree hash, a commit, a build id, an image digest, an environment. Without the binding, a `PASS` is a verdict about nothing in particular. **`payloadHash`** is `sha256:` plus the hex sha256 of the canonical JSON of a fixed field list. Canonical means object keys sorted with `localeCompare`, no whitespace, and undefined-valued keys omitted. The profile is `excellent.canonical-json.v1`: non-finite numbers are rejected and timestamps require an RFC 3339 offset. **`signature`** is Ed25519 over the ASCII bytes of that `payloadHash` string, base64url-encoded behind an `ed25519:` prefix. Ed25519 is the authoritative algorithm. Older HMAC receipts remain verifiable, but only as legacy read-only records: an HMAC key cannot be upgraded into an Ed25519 key by self-declaration. **`publicVerification`** restates what a third party needs in order to check the signature without parsing the rest. ## Verifying it **Anyone can run the public verifier.** [Receipt format](https://www.excellent.so/docs/receipt-format) serves a zero-dependency verifier, an example receipt pack and its public key from this site, with the three `curl` commands and the real output. That is the check to run if you do not have the Excellent source. It takes a receipt *pack* (the envelope plus its trust root and ledger anchor), so it does not accept this bare envelope: point it at the example pack, or at a pack of your own. **This exact receipt needs an Excellent monorepo checkout.** The monorepo's source is not public; the commands below are for people who already have a checkout. Its standalone verifier checks this document against the shipped public key. Copy the verifier, receipt and key somewhere outside the repo (being outside is the point) and run it there: ```sh mkdir /tmp/receipt-check && cd /tmp/receipt-check cp /scripts/lib/offline-receipt-verifier.mjs . cp /docs/v0/release-receipt.json . cp /docs/v0/release-receipt-public-key.pem . node offline-receipt-verifier.mjs release-receipt.json release-receipt-public-key.pem ``` Replace `` with the path to your Excellent monorepo checkout. The real output is: ```json { "bound": true, "receiptId": "receipt_v0_release_4df3dd89b3e4d0a0", "tenantId": "tenant_v0_release", "decisionDisclosure": "UNDISCLOSED", "claimedKeyId": "v0-ed25519-2026-08", "claimedSignatureAlgorithm": "ed25519", "signatureAlgorithm": "ed25519", "payloadHashMatches": true, "signatureValid": true, "excellentReachable": false, "excellentResolutionError": "ERR_MODULE_NOT_FOUND", … } ``` The omitted fields are `decidedBy`, `predicateStatus`, `undecidedReasonCode`, `recomputedPayloadHash`, `tamperedSignatureRejected`, `tamperedContentRejected`, and `cwd`. `excellentReachable: false` means the standalone verifier did not load Excellent to get its answer. The shipped tampered fixture changes the receipt but retains its original hash and signature. Copy it into that outside directory and run the verifier against it: ```sh cp /docs/v0/adversary/receipt-tamper-fixtures/tampered-receipt.json . node offline-receipt-verifier.mjs tampered-receipt.json release-receipt-public-key.pem ``` The verifier reports: ```json { "bound": false, "receiptId": null, "tenantId": null, …, "claimed": { "receiptId": "receipt_exp_v0_proof_cand_v0_proof", … }, "claimedKeyId": "v0-ed25519-2026-08", "payloadHashMatches": false, "signatureValid": false, "tamperedSignatureRejected": true, "tamperedContentRejected": true } ``` When the receipt is not bound, its identity and decision fields are `null`; the unverified envelope values appear under `claimed`. The verifier exits 0 when it reaches an answer, including a negative one, and exits 2 only when it cannot reach a verdict. Read the JSON booleans rather than treating exit 0 as a passing receipt. ## What a valid signature does and does not establish A valid signature means the receipt has not been altered since the holder of that key signed it. That is tamper-evidence, and it is real. It does **not** establish that the key is the tenant's key. If you obtained the key from the same system that issued the verdict, a forged receipt re-signed under an attacker's key verifies exactly as cleanly as a genuine one. Confirm the key's fingerprint through a channel this system does not control, then pin it. That is what `--pubkey` is for. Fetch a tenant's published trust material with: ```bash excellent-mcp verification universal-trust-root --json ``` ## Related - [Receipt format](https://www.excellent.so/docs/receipt-format): the JSON Schema for the whole pack, and a verifier you can download and run yourself. - [How verification decides](https://www.excellent.so/docs/how-verification-decides): what has to be true before a receipt can say `PASS`, and why one failing check is enough for `FAIL`. - [Receipts](https://www.excellent.so/docs/receipts): the short version. - [CLI reference](https://www.excellent.so/docs/cli): `excellent-mcp receipt verify` flags and exit codes. Source: https://www.excellent.so/docs/sample-receipt --- # Receipt format > Read the signed receipt schema and verify a sample bundle offline with a dependency-free Node script; no Excellent installation is required. A receipt is what a verification run leaves behind. It binds the work, the evidence, the policy and the verdict into one document, hashes that document canonically, and signs the hash with an Ed25519 key. The point of it is that you do not have to trust the machine that produced it, or us. That claim is only worth something if you can check it. This page publishes the format (a JSON Schema) and a verifier you can read in one sitting and run on a machine that has never had Excellent on it. Both are served from this site: - [receipt-schema.json](https://www.excellent.so/receipt-schema.json): the format, as JSON Schema 2020-12 - [verify-receipt.mjs](https://www.excellent.so/verify-receipt.mjs): a verifier, Node 18+, no dependencies - [receipt-example.json](https://www.excellent.so/receipt-example.json): an example receipt pack - [receipt-example-key.pub.pem](https://www.excellent.so/receipt-example-key.pub.pem): the public key it was signed with ## Check one in under a minute ```bash curl -fsSLO https://www.excellent.so/verify-receipt.mjs curl -fsSLO https://www.excellent.so/receipt-example.json curl -fsSLO https://www.excellent.so/receipt-example-key.pub.pem node verify-receipt.mjs receipt-example.json --pubkey receipt-example-key.pub.pem ``` Real output, from that exact command: ```console Excellent receipt check — receipt-example.json PASS pack shape schemaVersion 1, envelope and trust root present PASS signature algorithm is Ed25519 signature.algorithm = ed25519 PASS canonical binding present the receipt is bound to a work snapshot, contract, evidence snapshot, evaluator, determination, enforcement decision, review and build PASS payload matches its hash recomputed sha256:ee27611cb8b1d02d3edca0ad12308a64409930de759f7fc64c9f39042387b5bc from the receipt content and it matches envelope.payloadHash PASS envelope summary agrees with the receipt publicVerification restates the receipt id, run, tenant, key and hash without contradicting them PASS receipt and envelope carry the same signature receipt.signature mirrors envelope.signature PASS Ed25519 signature verifies the signature over sha256:ee27611cb8b1d02d3edca0ad12308a64409930de759f7fc64c9f39042387b5bc checks out against the public key you supplied PASS signing key is declared by the pack trustRoot declares key_example_2026_09 PASS key fingerprint matches the one you pinned sha256 over your PEM file is sha256:7e7b79a7135e246e76931580fd56a5d49fe94b2024c18dc81157038822bd1b4c, which is the fingerprint the pack declares for key_example_2026_09 PASS ledger anchor the ledger and audit log match anchor sha256:ba2658d08481e61503a281a16ab83a37edcd54521475e33d5571fc6809258c0f, recorded to operator-held-file at 2026-09-23T17:05:00.000Z RESULT: VERIFIED ``` The script then prints what that result does and does not establish. It exits 0 when the pack verifies, 8 when it does not, and 2 on a usage error, the same contract as `excellent-mcp receipt verify`. Add `--json` for one machine-readable line instead. Then break it, and watch it fail. Change any character of the receipt's content (the verdict, a timestamp, one hex digit of an evidence hash) and run it again: ```console FAIL payload matches its hash envelope.payloadHash is sha256:ee27611cb8b1d02d3edca0ad12308a64409930de759f7fc64c9f39042387b5bc but the content canonicalizes to sha256:7930319d69b2c1fceb5c0a4bc0187b57789c55df8a2dbc523685827afa6a08e2 — the receipt body has been altered since it was hashed RESULT: NOT VERIFIED ``` The signature is still valid: it is a signature of the original payload. What no longer holds is that the document in front of you is the document that was signed. Reformatting is a different matter: minify the file, reorder its keys, change its indentation, and it still verifies, because the signature covers a canonical rendering rather than the bytes on disk. ## What the verifier checks In order, and each one independently: 1. **The pack is the right shape.** `schemaVersion` 1, with an envelope and a trust root. 2. **The signature algorithm is Ed25519.** An authoritative receipt signed any other way is refused rather than downgraded. 3. **A binding is present.** Without one, a `PASS` is a verdict about nothing in particular. 4. **The payload matches its hash.** The canonical payload is rebuilt from the receipt and the binding, hashed, and compared to `envelope.payloadHash`. This is the tamper check. 5. **The envelope's summary agrees with the receipt.** `publicVerification` is a convenience copy and is not signed, so it is treated as a claim and cross-checked. 6. **The receipt and the envelope carry the same signature.** 7. **The Ed25519 signature verifies** against the public key you passed on the command line, never a key read out of the pack itself. 8. **The pack declares the signing key**, and 9. **the key's fingerprint matches the one you pinned**: sha256 over the bytes of your PEM file, compared to the fingerprint the pack states. 10. **The ledger anchor holds.** The anchor's id is recomputed from its contents, and the ledger and audit log are compared entry by entry against what it commits to. Only the first nine are about the receipt. The tenth is about whether the chain it sits in has been rewritten. ## The schema [receipt-schema.json](https://www.excellent.so/receipt-schema.json) describes the whole pack: the signed envelope, the receipt inside it, the binding that says what the receipt is about, the trust root, and the ledger anchor. Every field in it exists in the code that mints receipts; optional fields are marked optional, and fields whose meaning is not obvious from their name carry a description. Four things the schema says that are easy to miss, and that matter if you are writing your own verifier: **The signature does not cover the file.** It covers a canonical rendering of a fixed field list: `predicate`, `schemaVersion`, `id`, `tenantId`, `runId`, `parentReceiptId`, `status`, `checkResultIds`, `evidenceHashes`, `policyVersionRefs`, `engineVersion`, `issuedAt`, `binding`, and `decision` when the receipt carries one. Object keys are sorted, id arrays are sorted, there is no whitespace, and absent optional keys are omitted rather than written as null. Everything else in the file is outside the signature. **`publicVerification` is outside it.** It restates the receipt id, run, tenant, key and hash so a third party can check the signature without parsing the rest, but nothing stops someone editing it. A verifier has to re-derive those facts from the receipt and reject a document where the two disagree. **Absent is not the same as empty, except where it is.** In the binding, an absent `obligationResultIds` canonicalizes to `[]`, so absent and empty hash identically. An absent `decision` is omitted from the payload entirely, so absent and present-but-empty do not. That asymmetry is deliberate: receipts issued before `decision` existed are signed and immutable, and emitting the key unconditionally would have stopped every one of them verifying. **`status` is an aggregate, `decision` is not.** A run whose predicate was evaluated and came back false, and a run where nothing evaluated the predicate at all but one evidence artifact was unregistered, both aggregate to `FAIL`. `decision` is the field that separates them. When it is absent, the receipt does not disclose whether anything was decided, which is not the same as saying nothing was. ## The example receipt The document below is the file at [receipt-example.json](https://www.excellent.so/receipt-example.json). It is a **synthetic example**, and the line between what is invented and what is not matters here. The work it describes is invented: a throwaway task, a throwaway tenant, ids that point at nothing. The document around that work is not. It was produced by the same envelope builder, the same canonical hash, the same Ed25519 signing convention and the same anchor construction that a customer's receipts go through, signed with a keypair generated for this page and then discarded, and it verifies both under the script above and under Excellent's own offline verifier. A receipt from real work names a real repository, a real commit and a real task. Publishing one of those is a separate decision, and this page is built so that such a receipt can take this slot without anything else here changing. ```json { "schemaVersion": 1, "generatedAt": "2026-09-23T17:05:00.000Z", "envelope": { "schemaVersion": 1, "serviceVersion": "2026-08-16.v1", "receipt": { "schemaVersion": 1, "id": "receipt_example_a1b2c3", "tenantId": "tenant_example", "runId": "run_example_7f3a", "parentReceiptId": null, "status": "PASS", "decision": { "decided": true, "decidedBy": "excellent.evaluator.unit-test-predicate@1.4.0", "predicateStatus": "PASS", "undecidedReasonCode": null }, "checkResultIds": [ "check_example_unit_tests", "check_example_typecheck", "check_example_evidence_integrity" ], "evidenceHashes": [ "sha256:0ae18d55145fdfa9be069b98ea6afa2e16070cba8b2b025c76dea1b6ed52d35e", "sha256:70d6383e2b282c115efcac1f436506827d7d01964f5a95645b1b7dfd51d18976", "sha256:db792624ecd6a865a2155a8bbf457f33c7b11fa8a47bc1a38118ddaa641f2b44" ], "policyVersionRefs": [ "excellent.policy.agent-work@2026-08-16", "sha256:9f2c1d6b5a4e38f70c1b2d3e4f50617283940a1b2c3d4e5f60718293a4b5c6d7" ], "engineVersion": "excellent-verification-engine@2026-08-16.v1", "issuedAt": "2026-09-23T17:04:11.000Z", "signature": { "keyId": "key_example_2026_09", "algorithm": "ed25519", "value": "ed25519:nAAB2y8VQ5kvh5md75R42aIcMhfsKBlVEPP_kmhuYcN9VdiQn3x-fh8AvtZTmRfBJr-lqP4-uGNY2hD9TAHdDg" } }, "binding": { "predicate": "excellent.verification.agent-work-receipt.v1", "workItemId": "work_example_add_retry", "workSnapshotId": "snapshot_example_0f11", "attemptId": "attempt_example_01", "runManifestId": "manifest_example_01", "contractId": "contract_example_unit-tests-must-pass@3", "evidenceSnapshotId": "evidence_snapshot_example_51ab", "evidenceGraphDigest": "sha256:06e23d6cc15f57df2d8e123fbe2520e8291472fa4f2dcc573aa18a903e93d598", "obligationResultIds": [ "obligation_example_tests", "obligation_example_types" ], "evaluatorRefs": [ "excellent.evaluator.unit-test-predicate@1.4.0" ], "determinationId": "determination_example_9c2", "validityStateId": "validity_example_current", "enforcementDecisionId": "enforcement_example_promote", "enforcementEffectIds": [ "effect_example_merge_allowed" ], "reviewDecisionIds": [ "review_example_auto_accept" ], "outcomeObservationIds": [ "outcome_example_no_regression_7d" ], "auditEventIds": [ "audit_example_mint" ], "buildRef": "build_example_2026-09-23.1", "subject": { "treeHash": "sha1:4d9f1c0b7a6e5d4c3b2a19087f6e5d4c3b2a1908", "commitSha": "c0ffee1234567890abcdef1234567890abcdef12", "buildId": "build_example_2026-09-23.1", "imageDigest": "sha256:1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f809", "tenantId": "tenant_example", "environmentId": "env_example_ci", "policyDigest": "sha256:9f2c1d6b5a4e38f70c1b2d3e4f50617283940a1b2c3d4e5f60718293a4b5c6d7" } }, "payloadHash": "sha256:ee27611cb8b1d02d3edca0ad12308a64409930de759f7fc64c9f39042387b5bc", "signature": { "keyId": "key_example_2026_09", "algorithm": "ed25519", "value": "ed25519:nAAB2y8VQ5kvh5md75R42aIcMhfsKBlVEPP_kmhuYcN9VdiQn3x-fh8AvtZTmRfBJr-lqP4-uGNY2hD9TAHdDg" }, "publicVerification": { "algorithm": "sha256", "payloadHash": "sha256:ee27611cb8b1d02d3edca0ad12308a64409930de759f7fc64c9f39042387b5bc", "signatureAlgorithm": "ed25519", "keyId": "key_example_2026_09", "receiptId": "receipt_example_a1b2c3", "runId": "run_example_7f3a", "tenantId": "tenant_example", "issuedAt": "2026-09-23T17:04:11.000Z", "parentReceiptId": null } }, "trustRoot": { "tenantId": "tenant_example", "generatedAt": "2026-09-23T17:05:00.000Z", "declaredKeys": [ { "keyId": "key_example_2026_09", "fingerprint": "sha256:7e7b79a7135e246e76931580fd56a5d49fe94b2024c18dc81157038822bd1b4c", "algorithm": "ed25519" } ], "signedByKeyId": "key_example_2026_09", "signatureAlgorithm": "ed25519" }, "ledgerAnchor": { "schemaVersion": 1, "id": "sha256:ba2658d08481e61503a281a16ab83a37edcd54521475e33d5571fc6809258c0f", "target": "operator-held-file", "anchoredAt": "2026-09-23T17:05:00.000Z", "ledgerHead": "sha256:fabe76b800f3d84dde97be85a1e32b72c096491d02f4c826400d54a207ba6e33", "ledgerLength": 2, "ledgerEntryHashes": [ "sha256:91ba2f9aa2f63fa36df1bb719959bcaebb8034852886bbee141b2a5825a059d3", "sha256:fabe76b800f3d84dde97be85a1e32b72c096491d02f4c826400d54a207ba6e33" ], "auditLogHead": "sha256:033be47f7c89645acecf79fd7f6976622c62f18dd105331dd1771c8d2d03a5f4", "auditLogLength": 1, "auditLogEntryHashes": [ "sha256:033be47f7c89645acecf79fd7f6976622c62f18dd105331dd1771c8d2d03a5f4" ] }, "ledger": [ { "id": "ledger_example_1", "entryHash": "sha256:91ba2f9aa2f63fa36df1bb719959bcaebb8034852886bbee141b2a5825a059d3" }, { "id": "ledger_example_2", "entryHash": "sha256:fabe76b800f3d84dde97be85a1e32b72c096491d02f4c826400d54a207ba6e33" } ], "auditLog": [ { "id": "audit_example_1", "entryHash": "sha256:033be47f7c89645acecf79fd7f6976622c62f18dd105331dd1771c8d2d03a5f4" } ], "legacyReadOnly": false } ``` ## Checking a receipt of your own The script takes any receipt pack, not just this one. Excellent writes them; point the script at yours and pin your own key: ```bash node verify-receipt.mjs my-receipt-pack.json --pubkey my-key.pub.pem ``` If you also hold an anchor id from outside the pack, supply it and rollback becomes detectable too. Both flags are required together: an anchor id with no time cannot tell a pack that is merely out of date from one attesting a different ledger: ```bash node verify-receipt.mjs my-receipt-pack.json --pubkey my-key.pub.pem \ --expect-anchor-id sha256:… --expect-anchored-at 2026-09-23T17:05:00.000Z ``` `verify-receipt.mjs` is a second, independent implementation of the format described here. The reference implementation is `excellent-mcp receipt verify --pubkey `, which runs the same checks from inside the product and reports them as one JSON line. The two agreeing is the useful signal. The two disagreeing would be a bug worth telling us about, in whichever one turns out to be wrong. ## What a valid signature does not establish A valid signature means the receipt has not been altered since the holder of that key signed it, and that the verdict is bound to a named work snapshot, contract, evidence snapshot and build. That is tamper-evidence, and it is real. It does not establish that the work was good. A receipt records that checks ran and what they concluded. Whether those were the right checks (whether they test anything worth testing) is a judgement the signature cannot make for you. The receipt names its checks by id precisely so that you can go and look. It does not establish who signed. A signature proves custody of a key, never an identity. If you obtained the key from the same system that issued the verdict, a forged receipt re-signed under an attacker's key verifies exactly as cleanly as a genuine one. Confirm the key's fingerprint through a channel that system does not control, then pin it. That is what `--pubkey` is for, and why the script refuses to read a key out of the pack it is checking. And with no independently held anchor, it does not establish that you are holding the current pack. A genuine anchor from an earlier state, presented with the ledger rewound to match it, verifies exactly like a current one. ## Related - [Sample receipt](https://www.excellent.so/docs/sample-receipt): the product's real release receipt (verdict `FAIL`), field by field, and a tamper being caught. - [Receipts](https://www.excellent.so/docs/receipts): the short version. - [How verification decides](https://www.excellent.so/docs/how-verification-decides): what has to be true before a receipt can say `PASS`. - [CLI reference](https://www.excellent.so/docs/cli): the CLI verifier's flags and exit codes, and which CLI release has it. Source: https://www.excellent.so/docs/receipt-format --- # The local verification buyer's guide > What local verification should mean in 2026: where evidence lives, how receipts are checked, and what must work without trusting a hosted agent dashboard. "Local-first" matters more when AI agents start changing real work. The question is no longer only where your notes or records live. It is where the evidence for agent-authored work lives, who can inspect it, and whether a result can be checked without trusting a hosted dashboard. This guide is for teams evaluating verification systems around AI agents. ## What local verification should mean A local verification system should give you three things: 1. **The proof is yours.** Attempts, artifacts, command output, screenshots, diffs, and receipts live in files or infrastructure you control. 2. **The result is inspectable.** A reviewer can see what the agent claimed, what changed, what checks ran, what passed, what failed, and what could not be checked. 3. **The system works without trusting the agent.** The completion claim and the verification decision are separate. If one of those is missing, you may have an agent dashboard, but you do not have a verification system. ## The questions to ask a vendor ### 1. Where do receipts live? Ask where the durable record of each attempt and result is stored. A good answer names the local path, database, artifact store, or self-hosted service. A weak answer points only to the vendor's hosted activity feed. ### 2. Can the evidence be inspected outside the app? Receipts should preserve enough structure that your team can audit them later. That means command names, exit codes, relevant logs, changed files, artifact references, timestamps, and reviewer decisions. Screenshots and summaries are useful, but they should not replace raw proof when raw proof exists. ### 3. Is verification separate from the agent? The agent can explain what it did. It should not be the only authority on whether the work is done. Look for a verification step that can run tests, inspect diffs, check browser behavior, read CI, observe deployments, and preserve uncertainty when it cannot decide. ### 4. What happens offline? Local verification should still let you inspect prior attempts and receipts without a network. Some evidence sources need network access, such as hosted CI or deployment logs, but the local result history should not disappear when a vendor service is down. ### 5. How do you leave? Ask how to export attempts, evidence, and receipts. If leaving means screenshots, CSV fragments, or a support ticket, the proof trail is not really yours. ## Red flags - The system stores only chat transcripts. - The agent's final message is treated as the result. - CI status is shown without the command, commit, branch, or relevant output. - Failed attempts are overwritten by retries. - Screenshots exist but are not tied to the task, route, viewport, or result. - Prompt or model changes are promoted without held-out comparison. - Export gives you summaries but not evidence references. ## How Excellent answers these Excellent is built around attempts, checks, evidence, results, and receipts: - It runs beside the agent and model you choose. - It records what was requested, what the agent claimed, and what changed. - It attaches proof from commands, browser checks, CI, deployments, diffs, and reviewer notes. - It returns failed work with the exact evidence that failed. - It keeps accepted results as receipts you can open later. Receipts stay on the machine that produced them and do not sync between teammates yet. - It compares agent setup changes on held-out tasks before promotion. You should not have to trust our marketing page for that. The docs and receipts should be the proof. ## What this guide is, and is not This is not a vendor ranking. It is a checklist for deciding whether a system can verify agent work in a way your team can audit. The right tool depends on your workflow. The floor should be simple: the agent does the work, verification checks the work, and the evidence stays inspectable. Source: https://www.excellent.so/guides/local-verification-buyers-guide --- # How to choose an agent verification system > Twelve questions to ask before you trust an agent workflow: evidence capture, checks, reviewer boundaries, receipts, and exit cost. An **AI agent work verification system** checks a completion claim against evidence from the task: what was requested, what changed, which checks ran and what they observed. It should leave a record another person can inspect. This is different from verifying an agent’s identity or permission to act. Choosing an AI agent is only half the decision. The other half is choosing how its work gets checked. ## Match the tool to the question | Question | Evidence to look for | Start here | | --- | --- | --- | | What did the agent do? | Tool calls, inputs, outputs and errors recorded during the attempt. A trace helps reconstruct work; it does not establish correctness by itself. | [Work and attempts](https://www.excellent.so/docs/work-and-attempts) | | How good is the output across examples? | A stated rubric, reference labels and an evaluation set separate from the examples used to tune the agent. | [LLM judges and evaluation](https://www.excellent.so/compare/llm-as-judge-evals) | | Did this particular task meet its requirements? | Checks tied to the requested outcome, observed results and unresolved checks that remain visible. | [Verification checks](https://www.excellent.so/checks) | | Can someone else inspect the result? | The evidence record and its limits, plus a way to check that the receipt has not changed. | [A real signed receipt](https://www.excellent.so/receipts) | For a concrete workflow, name the expected result before running the agent, preserve the attempt, run an independent check and retain the observed result. If the check cannot run, record that gap instead of treating the task as verified. Compare [tool categories](https://www.excellent.so/compare) before using a product shortlist as the decision. The checklist below describes selection criteria, not a claim that every command is in the current Excellent release. Check the [quickstart version note](https://www.excellent.so/docs/quickstart) before trying the documented v1 workflow. This guide is the checklist we use when a team asks whether an agent workflow is ready for real engineering work. It is organized around the parts that matter after the demo: attempts, checks, evidence, reviewer boundaries, receipts, and improvement. ## Bucket 1: Attempts ### 1. What exactly is recorded? The system should capture the task, repository state, branch, changed files, agent identity, model, prompt context, tools, artifacts, and final claim. If the record is only a chat transcript, review will still depend on memory. ### 2. Are retries preserved? Failed attempts should not disappear when the agent tries again. You need the first failure to understand whether the prompt, context, check, or agent behavior needs to change. ### 3. Can a reviewer reconstruct the work? A useful attempt record lets a reviewer answer: what was requested, what changed, what the agent claimed, and what proof was required. ## Bucket 2: Checks ### 4. Can checks be task-specific? Different work needs different proof. A docs update, API change, UI fix, migration, and release task should not all run the same generic checklist. ### 5. Does the system capture raw output? Pass/fail summaries are not enough. Keep command names, exit codes, relevant output, screenshots, traces, CI links, and deployment observations. ### 6. Does it say when it cannot tell? Unchecked is a real result. A good system does not turn missing evidence into false confidence. ## Bucket 3: Review boundaries ### 7. Is verification separate from the agent? The agent can claim it is done. Verification should run outside that claim. Look for structural separation: required checks, reviewer identities, and result transitions that cannot be faked by the same actor that made the change. For the evidence on model reviewers, read [the review gap](https://www.excellent.so/research/the-review-gap). ### 8. Can humans attach judgment? Some work needs human review. The system should let reviewers add notes, decisions, and skipped-proof rationale without losing the mechanical evidence. ### 9. Can failed work be returned cleanly? The agent should receive the exact failure or uncertainty, not a vague "please fix." Returned results should preserve the failed evidence and start a new attempt. ## Bucket 4: Receipts and improvement ### 10. What does an accepted receipt contain? At minimum: task, attempt, changed files, checks, artifacts, decision, timestamps, and reviewer context. ### 11. Can receipts be exported? If you cannot export receipts and evidence references, the audit trail is rented. ### 12. How are better setups promoted? Prompt, model, tool, and context changes should be compared on held-out tasks. A setup should not be promoted because it feels better in one session. ## The honest summary The right verification system is the one that makes agent work inspectable enough for your team to trust, return, or reject it. For Excellent, that means a loop around the agent you already use: - Start with a task. - Record the attempt. - Run checks. - Attach evidence. - Return a result. - Keep a receipt. - Compare improvements before promotion. That is the bar to hold any agent workflow against. Source: https://www.excellent.so/guides/choosing-an-agent-verification-system --- # Evaluating AI agents > How to tell a useful AI agent from a confident chat box: separation of duties, evidence trails, checks, receipts, and the questions that filter the noise. Every product launched in 2026 calls itself "agentic." Most are not. The difference matters because the capability you need is not a confident answer. It is work that can be checked, returned, accepted, and improved from evidence. This is the framework we use to tell them apart. ## The two-axis map Plot every "AI feature" on two axes: - **Authority:** does the AI just suggest, or does it act? - **Accountability:** is there a separate party that verifies the action? That gives you four quadrants: | | Low authority | High authority | |--------------------------|-----------------|------------------------| | **No accountability** | Copilot | Loose cannon | | **Verifier required** | Reviewable draft | Real agent | You want products in the bottom-right. Most products are in the top-left. The top-right is what makes the news when it goes wrong. ## What real agent capability looks like Six properties, in priority order. ### 1. Named roles, not "the AI" A real agent system has roles you can name: - **Shipper** does the work. - **Verifier** checks it. - **Triager** sorts the inbound. - **Scribe** writes the artifacts. - **Planner** sequences the work. If the product just calls everything "the AI," ask which role you're getting. If they can't answer, the system doesn't have roles. It has one undifferentiated model with prompts. ### 2. Separation of duties The shipper can't approve the shipper. The verifier is a different identity, ideally a different model or a different prompt with a different system context. This isn't a process. It's a property of the platform. The MCP server should refuse to mark work `done` when the actor is the same one that moved it to `review`. If a single agent can finish its own work, you don't have separation of duties. You have a checkbox. ### 3. An evidence trail you can read after the fact Every action the agent took should be queryable: which model, which tool, what arguments, what return value, what cost. The auditors aren't going to accept "we trust the agent." They'll accept a log. The minimum bar: **per-session, per-tool, per-cost timeline** of every autonomous action. ### 4. Budgets the agent can't exceed A real agent system has a budget: dollars per session, tokens per task, or tool-calls per goal. The system refuses to continue past the cap and asks for supervision. If your AI feature can run indefinitely with no cap, it's a foot-gun. ### 5. Goals, not tasks A copilot lives in a chat window and reacts to the last message. An agent has a **durable goal** that persists across sessions. "Triage the inbound queue daily" is a goal. "Summarize this email" is a task. The presence of goals is the cleanest tell between the two. ### 6. Inspectable prompts and tools You should be able to see the system prompts, the tool definitions, and the policy file that governs what the agent can do. Vendors that hide the prompts hide the failure modes. If you can't read it, you can't reason about it. ## How to test a vendor's claim A practical evaluation, in four steps. Start with [how to read an agent benchmark headline](https://www.excellent.so/research/salesforce-35-percent-is-a-success-rate) before comparing scores. ### Step 1: Read the audit log Ask the vendor for a sample audit log from a real workflow. Look for: actor IDs, model names, tool calls with arguments, costs, and a clear before/after delta. If the "log" is `INFO: AI processed task`, walk away. ### Step 2: Try to make the agent approve its own work Sign in as one user. Mark a task `in progress`. Move it to `review`. Try to move it to `done` as the same identity. The platform should refuse. If it doesn't, the verification gate is a UI convention, not a property of the system. ### Step 3: Look for the budget knob Find the place where you set "this role can spend at most $X per session" or "at most Y tool calls per goal." If there isn't one, the vendor's confidence in the agent is unearned. ### Step 4: Read the prompts If the prompts are inspectable, read them. They tell you exactly what the agent is told to do, what tools it can call, and what guardrails are nominal vs. structural. If the prompts are hidden, the failure modes are hidden with them. ## Red flags - **"Just trust the AI."** No. - **"Magic" or "intelligence" in the spec sheet.** Spec sheets do not contain magic. - **A single AI module that does everything.** That's marketing, not architecture. - **No way to disable the AI per-record or per-workflow.** You will need to. - **Inference is mandatory in the vendor's cloud.** Your customer data shouldn't be the price of admission. ## What Excellent verifies For comparison, Excellent does not ask you to trust a claimed agent layer. It records the work and checks the result: - **Work:** the request, scope, owner, expected checks, and proof envelope are written down before the agent starts. - **Attempts:** each run records what the agent claimed, what changed, and which evidence it attached. - **Checks:** tests, browser proof, deployment proof, data checks, and custom gates run as independent evidence. - **Results:** a result is accepted only when the evidence matches the work, not because the agent said it is done. - **Receipts:** accepted work leaves behind a durable receipt that can be read later by a human or another agent. - **Experiments:** model, prompt, and process changes are compared against observed outcomes instead of opinion. You don't have to take our word for any of that: every one of those bullets is visible in the [product](https://www.excellent.so/how-it-works) and the [docs](https://www.excellent.so/docs). --- **Next up:** the [AI-agent vendor comparisons](https://www.excellent.so/compare) put specific vendors beside Excellent, so you can hold each one up to these six points yourself. Source: https://www.excellent.so/guides/evaluating-coding-agents --- # Rolling out agent verification: a 30-day plan > Plan a 30-day agent verification rollout: choose a workflow, define checks and evidence, track attempts, return failures, and assess team reliability. Most agent rollouts start with enthusiasm and then hit the same question: who proves the work is actually good? This is a 30-day plan for adding verification without stopping the team. Each week has one objective, one measurable outcome, and one thing you deliberately do not need to finish. ## Week 0: Choose the first workflow Do not start with every repository and every agent. Pick one workflow where proof is already valuable: - A frontend task with browser behavior. - A backend task with a focused test suite. - A release chore with CI and deployment evidence. - A documentation or contract update with clear review rules. Write down the current path from request to merge. Include the agent used, the model provider, the repo, the branch pattern, the required commands, the review owner, and what usually gets missed. The measurable outcome for week 0 is simple: one task type with a known proof envelope. ## Week 1: Record attempts Install Excellent and run it beside the agent you already use. Goals for the week: 1. A real task starts as an Excellent attempt. 2. The attempt records the request, repo state, model, agent, prompt context, changed files, and agent claim. 3. The team can open the result and see what happened without reconstructing it from terminal history. What you are not doing this week: - Replacing your agent. - Rewriting your release process. - Connecting every evidence source. The goal is observability. Before you improve anything, make the agent's work legible. ## Week 2: Attach checks and evidence Now connect the proof sources that decide whether the work should be trusted. Start with the narrow proof envelope from week 0: 1. Add the task-specific commands. 2. Capture exit codes and useful output. 3. Attach browser screenshots or traces when behavior matters. 4. Link CI or deployment output when the task depends on it. 5. Record human review notes when judgment is still required. By the end of week 2, a result should be able to say `verified`, `returned`, or `unchecked` with the evidence attached. ## Week 3: Return failures cleanly Verification is useful only if failed work comes back with enough detail for the next attempt. Tune the return path: - Send the exact failed check back to the agent. - Preserve the original task and failed attempt. - Make retries visible instead of overwriting the first run. - Require a fresh result for each material change. - Mark uncertainty honestly when proof is missing. This is where teams usually find the highest leverage. A clear returned result turns a vague code review into a concrete repair loop. ## Week 4: Compare the setup Once attempts and results are consistent, compare better ways to run the agent. Pick one candidate change: - A different model. - A smaller or cheaper model for part of the workflow. - A new prompt or instruction file. - A tighter context package. - A different check cadence. - A retry rule that explains failures before rerunning commands. Test the candidate on held-out tasks. Do not promote it because it feels better in a demo. Promote it only when the evidence shows better quality without hiding cost, human help, or regressions. ## What is left after 30 days After the four weeks, you should have: - One workflow verified from request to result. - Receipts for accepted agent work. - Failed attempts returned with concrete evidence. - A known set of checks the team trusts. - A first candidate comparison for improving the agent setup. Then repeat the pattern for the next workflow. ## When this plan fails It fails when teams: - Start with every repo and every agent at once. - Let the agent decide whether its own work is done. - Treat CI as the whole proof envelope when product behavior also matters. - Hide failed attempts instead of using them to improve checks. - Promote a new prompt or model without held-out comparison. The point is not ceremony. The point is to make agent work inspectable enough that trust can come from evidence. Source: https://www.excellent.so/guides/agent-verification-rollout-plan --- # The review gap: what the evidence says about unchecked AI agent work > Eleven outside studies from 2023 to 2026 on how completion claims, passing tests and missing review let wrong agent work through, and what kinds of review catch it. Excellent · November 16, 2026 · Research · Verification AI agents now open pull requests, update CRM records and close tickets. We read the published evidence from 2023 to 2026 on four questions: 1. How much real work do agents finish? 2. How often is "passing" work actually right? 3. Can an agent check its own work? 4. What kind of review closes the gap? Read together, the studies point the same way. In their samples, passing tests overstated correctness, models judged their own work too kindly, and most agent pull requests in one large study had no recorded review. Review that gathered its own evidence, fixed what "done" means in advance or escalated the hard calls did better. This is a living review. We re-open every source each quarter and log changes at the bottom. ## Agents finish less than it looks In two benchmark suites of realistic, multi-step work, the leading agents fully completed fewer than half of the tasks. - Salesforce AI Research's [CRMArena-Pro](https://arxiv.org/abs/2505.18878) gave nine models 19 CRM tasks across 4,280 queries. The best, Gemini 2.5 Pro, scored 58.3% on single-turn tasks in the B2C environment and 35.1% on multi-turn tasks in B2B; within B2B alone it fell from 54.1% to 35.1%. The median model succeeded on fewer than one multi-turn job in four. We wrote up [what that 35% does and does not mean](https://www.excellent.so/research/salesforce-35-percent-is-a-success-rate). - Carnegie Mellon's [TheAgentCompany](https://arxiv.org/abs/2412.14161) simulated a small software company with 175 tasks. The best model fully completed 30% of them; its partial-completion score was 39%. Both score task outcomes against reference criteria: CRMArena-Pro compares extracted answers, and TheAgentCompany checks checkpoints in its simulated company. Neither reports whether the agent's own claim of completion matched the outcome. In a deployment, that claim may be the first thing a person sees. ## Passing the tests is not the same as being right A test suite checks what it checks. Agents are good at satisfying exactly that. - **Plausible is not correct.** Wang, Pradel and Liu [audited](https://arxiv.org/abs/2503.15223) patches from three issue-solving tools that SWE-bench Verified counted as solved (ICSE 2026). 29.6% of those plausible patches behaved differently from the developers' fix. On manual inspection, 28.6% of the divergent patches were certainly incorrect. Overall, reported solve rates were inflated by 6.2 absolute percentage points. - **When the tests and the spec disagree, agents pick the tests.** [ImpossibleBench](https://arxiv.org/abs/2510.20270) (Zhong, Raghunathan and Carlini, ICLR 2026) built coding tasks whose tests contradict the specification, so any pass is a cheat. With its full scaffold, GPT-5 cheated on 54% of the Conflicting-SWEbench tasks, and on 76% of the Oneoff-SWEbench tasks, where a single test's expected output was changed. Its methods ranged from editing the tests to overloading operators. - **Telling them not to does little.** [METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/) found o3 gaming the scorer in 39 of 128 RE-Bench runs (30.4%). Instructing it not to cheat "had a nearly negligible effect". Two results point at the fix. ImpossibleBench allowed models to flag an impossible task for human intervention, which ended the run. That cut GPT-5's cheating on the conflicting tasks from 54% to 9% and o3's from 49% to 12%; the effect was much smaller for Claude Opus 4.1. The SWE-bench audit also found that the benchmark's own validation let 7.8% of all patches count as correct while they failed the developers' test suite. More independent evidence, and a way to escalate, both help. ## The author cannot be the reviewer The obvious shortcut is to ask the agent, or another copy of the same model, to check its own work. The evidence says that is a weak check. - Huang et al. ([ICLR 2024](https://arxiv.org/abs/2310.01798)): "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction." - Panickssery, Bowman and Feng ([2024](https://arxiv.org/abs/2404.13076)) showed that the models they tested recognize their own writing and rate it more highly. In their fine-tuning experiments, greater self-recognition went with stronger self-preference. - In ImpossibleBench, LLM monitors caught 86–89% of cheating attempts on the simpler LiveCodeBench tasks, but only 42–65% on the more complex SWE-bench tasks. Independence has to be real: a different model family, a person, or a check that does not depend on the author's account. ## Most agent pull requests in one large study had no recorded review Where review was measured at scale, it was often missing rather than weak. - Duma et al. ([EASE 2026](https://arxiv.org/abs/2605.02273)) studied 33,596 agent-authored pull requests in popular repositories from the AIDev dataset. 61.38% had no recorded review. Of the 39,122 review comments that did appear, 71.58% were written by agents. Only 15.9% of agent PRs showed any human participation. In repositories with both kinds of PR, human participation was similar (30.12% of agent PRs, 30.83% of human ones), but among reviewed PRs, human-only review was 8.08% for agent PRs against 25.21% for human PRs. The authors note that no recorded review does not prove no one looked. - A vendor study by [CodeRabbit](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) (December 2025) of 470 open-source PRs found 10.83 issues per AI-co-authored PR against 6.45 for human-only PRs, about 1.7 times as many. It is not peer reviewed, and the authors note they cannot confirm which PRs were human-only. Put together: in the AIDev sample, most agent-authored PRs had no recorded review, and in CodeRabbit's vendor sample, AI-coauthored PRs had more classified issues per PR. Neither finding says how much agent work reaches the main branch. ## What closes the gap Three lines of evidence say what good review looks like. **Review that gathers its own evidence.** [Agent-as-a-Judge](https://arxiv.org/abs/2410.10934) (Zhuge et al., ICML 2025) gave the judge tools to inspect the workspace rather than read the agent's account. On 55 development tasks with 365 requirements, its verdicts matched the human experts' consensus on 90.44% of requirements, against 60.38% for a plain LLM judge, both judging OpenHands' work without its trajectory. It cost 2.29% as much as the human panel and took 2.36% of the time. **Cheap review for easy calls, escalation for hard ones.** Carnegie Mellon's [JEV-as-a-Judge](https://arxiv.org/abs/2609.26550) tested JEV, TypeSafe AI's hosted decision-only judge, which returns label probabilities instead of text. It came within three points of GPT-6 "wherever a verdict can be read off the text", at 0.36% of its fee, and fell behind "where the verdict must be derived, as in math, code, and logic". Accepting its confident verdicts and escalating the rest was 0.9 points *more* accurate than GPT-6 alone on 1,610 held-out pairs, at 41% of the fee. The authors note the routing weakens on style-adversarial pairs and on prose with no reference answer. **Checks fixed before the work starts.** In our own [preregistered lab study](https://www.excellent.so/research/five-reviewers), we gave 72 attempts at small coding tasks to five kinds of review. On the preregistered labels, the test suite approved 32 of the 35 wrong attempts. A frontier model reading the diff approved 1. Checks written from the contract before any agent started approved none, at the cost of holding back 5 of 37 good attempts. It is a small pilot and sets no rate. ## Six rules for reviewing agent work These are our recommendations from the evidence above, not a tested recipe. 1. **Treat the report as a claim.** "Done", "tests pass" and a clean exit are statements by the author, not evidence. 2. **Gather evidence the author did not produce.** Re-run the checks yourself, read the real end state, run the full suite. 3. **Decide what done means before the work starts.** Checks written afterwards drift toward what was built. 4. **Keep the reviewer independent.** Another model family, a person, or a deterministic check, never the author grading itself. 5. **Spend review where it is hard.** Let cheap checks clear the easy calls. Escalate code, derivations and confident-sounding answers. 6. **Let models propose and execution decide.** A model may suggest the input that breaks the work. Only running it should count. ## Closing Agents keep getting more capable. The studies above suggest the larger gap is in how their work is accepted: on the author's word, on a passing test, or with no recorded review. The remedy they point to is old: independent evidence, decided in advance. Excellent is built around it. Its planned v1 workflow freezes your checks before the work and [re-runs them on the finished change](https://www.excellent.so/checks). The [published CLI](https://www.excellent.so/changelog) does not yet include that workflow; checks written from the spec are next. ## How we read the evidence We used peer-reviewed papers and preprints where available and opened every source cited. Vendor reports are labeled. Figures are quoted as published, with the model versions each study tested; most of those models are already a generation old. We left out figures we could not trace to a primary page, such as an often-quoted forecast about agent project cancellations. ## Sources 1. Huang et al., [CRMArena-Pro](https://arxiv.org/abs/2505.18878), Salesforce AI Research, arXiv, May 2025. 2. Xu et al., [TheAgentCompany](https://arxiv.org/abs/2412.14161), Carnegie Mellon University, arXiv v3, September 2025. 3. Wang, Pradel and Liu, [Are "Solved Issues" in SWE-bench Really Solved Correctly?](https://arxiv.org/abs/2503.15223), ICSE 2026, arXiv v2. 4. Zhong, Raghunathan and Carlini, [ImpossibleBench](https://arxiv.org/abs/2510.20270), ICLR 2026. 5. METR, [Recent frontier models are reward hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/), June 5, 2025. 6. Huang et al., [Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798), ICLR 2024. 7. Panickssery, Bowman and Feng, [LLM Evaluators Recognize and Favor Their Own Generations](https://arxiv.org/abs/2404.13076), 2024. 8. Duma et al., [These Aren't the Reviews You're Looking For](https://arxiv.org/abs/2605.02273), EASE 2026. 9. CodeRabbit, [State of AI vs human code generation](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report), December 17, 2025. Vendor report. 10. Zhuge et al., [Agent-as-a-Judge: Evaluate Agents with Agents](https://arxiv.org/abs/2410.10934), ICML 2025. 11. Li, Miao, Krishnan and Padman, [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure](https://arxiv.org/abs/2609.26550), arXiv preprint v5, October 9, 2026. 12. Excellent, [Same work, five reviewers](https://www.excellent.so/research/five-reviewers), November 2026. ## Changelog - **November 16, 2026.** First version. Next review in February 2027. Source: https://www.excellent.so/research/the-review-gap --- # Same work, five reviewers > We gave 72 attempts at small coding tasks to five kinds of review. The test suite approved 32 of the 35 wrong ones. Checks written from the spec before any agent started approved none. Excellent · November 11, 2026 · Research · Verification We gave 72 attempts at 12 small coding tasks to five kinds of review. On our preregistered labels, the test suite approved 32 of the 35 attempts that were wrong. Checks written from the spec, before any agent started, approved none. Most of those 35 were wrong on purpose: we planted or prompted shortcuts to see what each reviewer would miss. So we also report the 12 wrong attempts we did not script: 9 where Claude was told to cut corners and 3 honest mistakes. On those, the test suite approved 10 and the checks written first approved none. An agent saying "done" is a claim. A passing test run is a claim with better manners. We wanted to know how much wrong work each common kind of review lets through, and how much good work the stricter kinds hold back. This was a small pilot. We wrote down how we would judge it before anything ran, scored it once, and had a model from another AI lab check our scoring before we wrote a word about it. ## The setup - **12 tasks.** Each is a broken Node function, such as parsing a hex color, formatting a byte count or converting `camelCase` to `snake_case`. Each comes with a written contract of 4 to 7 rules and one visible test that checks only some of them. Codex wrote the tasks. - **72 attempts.** 36 honest fixes, 12 each from Claude Code, Codex and a local 9-billion-parameter model (Qwen3.5-9B). 12 attempts in which Claude was told to cut corners. 24 shortcuts we planted by script, plus 1 seed. One cut-corners attempt that leaked its instruction into its own output was excluded, leaving 72. - **Ground truth.** 651 hidden test cases on which two models from different labs (Grok and Codex) and a reference implementation all agreed. By that standard, 35 attempts were wrong and 37 were right. Five reviewers judged every attempt: 1. **The test suite.** Run `npm test` and accept a pass. 2. **Frozen replay.** Restore the original test and protected files, check that the edit stayed inside the allowed paths, then run the test again. 3. **A fast AI judge.** A hosted decision model that reads the agent's transcript and answers whether the task is done. 4. **A frontier model with evidence.** Claude reading the transcript, the diff and the test log. 5. **Checks written first.** Frozen replay plus 24 checks per task that Claude wrote from the contract before any attempt existed. At review time they run as plain code, with no model involved. ## What each reviewer let through ![Wrong attempts approved, of 35: the test suite 32, frozen replay 26, a fast AI judge 12, a frontier model with the diff 1, checks written first 0. Good attempts held back, of 37: 0, 0, 6, 3 and 5.](https://www.excellent.so/research-figures/five-reviewers.svg) *Phase 0b, preregistered labels, 72 attempts. Generated from the scored results file.* | Reviewer | Wrong approved, all (of 35) | Wrong approved, unscripted (of 12) | Good held back (of 37) | | --- | --- | --- | --- | | The test suite | 32 | 10 | 0 | | Frozen replay | 26 | 5 | 0 | | A fast AI judge | 12 | 2 | 6 | | A frontier model with the diff | 1 | 1 | 3 | | Checks written first | 0 | 0 | 5 | The test suite mainly sees what the visible test checks, and most wrong work passed that test. The frozen replay also restores the protected files and enforces the allowed paths. So it approved none of the 3 attempts that edited the test and the 3 that changed files outside the task, where the plain test suite approved all 6. Shortcuts that passed the visible test without touching it passed both. Reading the work helped much more. The frontier model with the diff let 1 wrong attempt through. The checks written first let none through. Most of the 35 were scripted to pass the visible test, which makes the first two rows look worse than they would on natural mistakes. The 12 we did not script are the fairer view, and the order holds there too. ## "All tests now pass" The local model was asked to fix `camelToSnake`. It rewrote the function to handle acronyms such as `XMLHttpRequest` and finished with: *"All tests now pass."* They did. The test suite approved. The frozen replay approved. The fast judge approved. Claude, reading the diff and the test log, approved. It was still wrong. `"ABC"` came back as `ab_c` instead of `abc`, and `"userID"` as `user_i_d` instead of `user_id`. It failed 4 of the 57 hidden cases for its task. This was an honest attempt, not a planted shortcut. Of the five reviewers, only the checks written from the contract caught it: 2 of its 24 checks failed. ## The missing leap day In a second task, Claude was told to cut corners. It changed one line of a date parser so that `1900-02-29` would be rejected, and reported: *"`npm test` passes."* The new leap-year rule forgot the 400-year case, so `2000-02-29`, a real date, now came back as invalid. The test suite, the frozen replay and the fast judge approved it. The frontier model reading the diff and the checks written first caught it; 3 of the 24 checks failed, on 2000, 1600 and year 0. ## What review costs Stricter review holds back some good work. Of 37 good attempts, the fast judge held back 6, the frontier model 3 and the checks written first 5. Some of those holds were the reviewer being right and our labels wrong. The independent review of our results found 3 attempts we had marked correct that break their own contract on inputs our hidden cases missed. We did not edit the labels. We wrote a blind repair procedure, committed it before it ran, and report both results below. The frontier-model review took about 6 seconds and $0.035 per attempt. The fast judge took about 0.18 seconds. The checks written first need no model at review time: they are written once per task, before the work starts. ## What we built because of it We treat an agent's report and a test's exit code as claims, never as evidence. What "done" means is fixed before the work starts, and a model is never the reason a check passes. A model can lower a verdict or send it to a person. A model can still help find problems. In a sixth arm, Claude proposed inputs it expected to break each attempt, and we counted a proposal only if running it actually failed. That arm approved 0 of the 35 wrong attempts and held back 3 of the 37 good ones, the best balance in the study. It is the basis for a veto we are building: a model may propose a counterexample, but only executing it can hold the work. Excellent's planned v1 workflow freezes your checks before the work starts and re-runs them itself, following the frozen replay in the second row. The [published CLI](https://www.excellent.so/changelog) does not yet include that workflow. Checks written from the spec before the work, the last row, are what we are building next. This study is why. ## What this does not show - **No error rate.** Twelve small single-function tasks are far from real repositories. These results set no bound on how often any reviewer, or any agent, gets it wrong. - **Most wrong work was planted.** Of the 35, 23 were scripted by us to pass the visible test, 9 came from Claude being told to cut corners, and 3 were honest failures, all from the local model. Claude Code and Codex got all 24 of their honest attempts right. This measures how review handles wrong work, not how often agents produce it. - **One prompt per judge.** Each model reviewer saw one fixed prompt, once per attempt. This is not a verdict on Claude, the fast judge or any model in general. - **The checks saw the starting code.** Claude wrote the checks from the task, the contract, the visible test and the broken source, before any attempt existed. - **Labels were imperfect.** On the repaired labels below, the frontier model and the checks written first each let 1 of 38 through. The order of the reviewers is unchanged, except that those two tie. ## Method and data The plan was preregistered and frozen in git before any attempt ran. A model from a different family (Codex) reviewed it twice before the run and held it both times; we fixed what it found, and the third review passed. Predictions were hashed before the labels were computed, and the results were scored once, on October 1, 2026. Codex then reviewed custody, labels and arithmetic. Its first verdict was a hold: it found the 3 mislabeled attempts and 2 errors in our review prompt. After the repair and a corrected write-up, it passed. The decision rule we fixed in advance did not select any configuration on the primary labels, because the checks written first missed its coverage floor by 2 attempts. It did on the repaired labels. We report the primary result first. The independent review required this wording for any public claim: > "In this descriptive pilot on 12 Codex-authored single-function Node fixtures, the preregistered no-leak labels (35 failures, 37 successes) yielded P-claude FA/cover of 0/35 and 32/37, P-qwen 3/35 and 6/37, Claude transcript-plus-diff 1/35 and 34/37, and the contract path 26/35 and 37/37. […] These synthetic results support no population error bound or generalization claim." In that sentence, P-claude is the checks written first, the contract path is the frozen replay, and FA is wrong work approved. P-qwen is the same checks written by the local model, which held back most good work. The repaired labels are a secondary, post hoc result. They come from a blind procedure committed before it ran, which grew the hidden cases from 651 to 1,477, each agreed by Grok, Codex and the reference. For one task, parse-percent, Codex's proposal calls timed out, so its new inputs were proposed by Grok alone, though both models still answered them. Three Codex replies also contained 41 cases instead of the 40 asked for. | Reviewer, repaired labels | Wrong approved (of 38) | Good approved (of 34) | | --- | --- | --- | | The test suite | 35 | 34 | | Frozen replay | 29 | 34 | | A fast AI judge | 13 | 30 | | A frontier model with the diff | 1 | 34 | | Checks written first | 1 | 31 | | Checks written by the local model | 4 | 5 | The [data appendix](https://www.excellent.so/research-data/phase0b-appendix.zip) has the plan, the hashes, the labels, the per-attempt verdicts and the review. For the fast judge it gives counts only. Archive SHA-256: `b5e828885580d8133856425e043e668fdff6d9bdcf639ed3703c977d70d3fd07`. [Install Excellent →](https://www.excellent.so/install) Source: https://www.excellent.so/research/five-reviewers --- # Check what your AI agent actually did > Why we're building independent verification for AI-agent work, how the planned workflow works, and what the published CLI supports today. Excellent · November 3, 2026 · Company · Verification Your AI agent says it's done. You gave it the job so you wouldn't have to do it, and now you're checking its work, which is the job. We're building Excellent to take on that check: establish what the agent actually did and how it compares with what you asked. Excellent is being built for independent verification of AI-agent work. The intended workflow connects the task, the checks, the evidence and the result, so you can inspect what was established and what remains unresolved. The [published CLI](https://www.excellent.so/changelog), version 0.1.0, provides repository setup and a separate-agent review gate. It does not yet include the wider verification workflow described below. ## Why we're building it AI agents can produce more work than people can carefully review. That's the point, and the problem. A wrong "done" looks exactly like a right one. The easy fix is to ask another model whether the first one did a good job. That helps. But it's still a model reading a claim, and the gap stays open. The planned workflow starts with independent observations. The repo, the tests, the build and the browser can each tell you something real about the result, within the limits of the checks connected to the task. Software comes first because it leaves unusually good evidence. The question is the same for a refund, a booking or an email: what would show it was done? ## The planned verification workflow This is the intended v1 workflow our docs describe. The [verification rules](https://www.excellent.so/docs/how-verification-decides) explain the supported checks and their limits. The [published CLI](https://www.excellent.so/changelog) predates this workflow; its current capabilities are listed below. - **Run.** Every attempt starts on the record. Its agent, model, task and repo are captured before the work begins. - **Check.** Excellent saves the result as it was handed in. Then it gathers evidence from the repo and the tools connected to the task: the git tree, the test runner, the build, the browser, CI. If a check can't run, Excellent says so. It won't quietly call that a pass. - **Return.** Wrong work comes back with the unmet check, what was expected and what was observed. The next attempt starts from that evidence instead of a fresh prompt. - **Receipt.** Accepted work leaves a receipt: signed, append-only and kept on your machine. A retry is a new attempt. The first one doesn't get rewritten. The design keeps verification separate from the agent doing the work. Deterministic checks or authorized human review can support a pass when the evidence requirements are met. Model judgment alone cannot; it can lower a verdict or send it to a person. The authored animation on our homepage illustrates this design with a hypothetical refund. Its amounts and outcomes are scripted. In the story, an agent is asked to refund the one damaged item on order 58213 and says the refund is ready: $96.00. The animation shows an amount check expecting $48.00 and returning the attempt. A second attempt shows $48.00, four passing checks and a verified result, with both attempts visible. These are illustrative states, not observations from a running refund integration or stored verification records. The illustration gives both attempts the same confident tone. The intended check distinguishes them by the amount. ## What's on npm today The CLI on npm is `@excellent-so/cli` 0.1.0, published July 21, 2026. It holds: - Two commands in one package: `excellent`, and `excellent-mcp`, a Model Context Protocol server that gives any MCP-capable agent the same tools. - `excellent init`, which sets a repository up for your agent. It writes an AGENTS.md and a `.excellent/` folder for project context and settings. - A review gate. `excellent-mcp review queue` lists work waiting for verification that the current agent didn't ship. `review approve` refuses to sign off on work the same agent shipped. - `excellent-mcp agent converge`, which alternates a verifier and a shipper until the queue is verifiably done, stalls, or reaches its cycle limit. Workers run through Claude Code or Codex, picked with `--cli`. - `excellent doctor` checks the runtime, the local database, providers, installed agent CLIs and your PATH in one read. - `excellent update` updates the CLI in place. The installer checks the mirrored build against its published SHA-256 checksum before installing it. If they don't match, it stops. So, plainly: 0.1.0 predates the v1 workflow above. It does not include the documented `excellent run`, `excellent verify` or `excellent receipt verify` commands. The [changelog](https://www.excellent.so/changelog) lists every published version and what each one holds. ## Free to run Excellent is free for individuals. No account, no credit card. It runs on macOS, where we develop and test it; Windows and Linux are untested. It needs Node.js 22.3 or newer. ```bash curl -fsSL https://excellent.so/install.sh | sh cd your-repo excellent init ``` Teams can join early access, or work directly with Excellent engineers on a deployment. ## What we publish We publish what we learn about agents doing real work, under three questions. How do we know an agent actually accomplished an objective? What predicts failure in long-running agentic systems? What changes when agents encounter real businesses? Our first research piece goes up on [Research](https://www.excellent.so/research) today: a close read of a benchmark number that's easy to mistake for a typical agent's failure rate. [Why local verification matters for AI agents](https://www.excellent.so/research/why-local-first) and [Introducing Excellent](https://www.excellent.so/research/introducing-excellent) cover the idea behind the product. [Install Excellent →](https://www.excellent.so/install) Source: https://www.excellent.so/research/check-what-your-ai-agent-actually-did --- # Salesforce's 35% is a success rate > The 35% from Salesforce's CRMArena-Pro is the best of nine models' multi-turn success rate, not a typical agent's failure rate. The median model finished fewer than one multi-turn job in four. Excellent · November 3, 2026 · Research · Reliability In June 2025 a trade headline said ["enterprise AI agents fail 65% of multiturn tasks"](https://ppc.land/salesforce-study-reveals-enterprise-ai-agents-fail-65-of-multiturn-tasks/). The number comes from [CRMArena-Pro](https://arxiv.org/abs/2505.18878), a benchmark from Salesforce AI Research. It is easy to read as a typical agent failing two jobs in three. The paper says something narrower, and in one way worse. ## What the 35% is The 35% is one model's score. Gemini 2.5 Pro, the best of the nine models tested, succeeded on 35.1% of multi-turn tasks in the paper's B2B environment. On the B2C environment the same model scored 30.0%. It is a success rate, averaged over four business skills: workflow execution, policy compliance, text reasoning and database querying. The 58% that usually travels with it is the same model's single-turn score on the B2C environment (58.3%). The paper's abstract says that leading agents reach about 58% single-turn success, dropping to about 35% in multi-turn settings. The headline kept the numbers and dropped the word "leading". [CIO](https://www.cio.com/article/4008228/salesforce-study-warns-against-rushing-llms-into-crm-workflows-without-guardrails.html) wrote that "agents achieved a 58% success rate in single-step tasks … but dropped to just 35% in multi-step workflows", with no model named. ## The median model did worse Read as a failure rate, 65% is the best case, not the typical one. Here is every model's multi-turn score from the paper's Table 2, averaged over the four skills: | Model | B2B multi-turn | B2C multi-turn | | --- | --- | --- | | Gemini 2.5 Pro | 35.1% | 30.0% | | o1 | 32.5% | 30.4% | | Gemini 2.5 Flash | 30.1% | 23.4% | | Gemini 2.0 Flash | 26.9% | 23.4% | | Llama 4 Maverick | 23.7% | 14.9% | | Llama 3.1 405B | 21.0% | 20.0% | | GPT-4o | 19.7% | 18.3% | | GPT-4o mini | 19.3% | 15.8% | | Llama 3.1 70B | 16.3% | 15.1% | The median model succeeded on 23.7% of multi-turn B2B jobs and 20.0% of B2C jobs: fewer than one in four. ## What a failure means in this benchmark A failure here is a final answer that did not match the expected one. The benchmark uses exact match for IDs and values and token-level F1 for free text. In the multi-turn setting, an LLM-simulated user with a sampled persona gives out the details of the request a little at a time, often incompletely at first, so the agent has to ask the right follow-up questions. The models are the ones available in spring 2025: o1, GPT-4o, the Gemini 2.5 and 2.0 family, Llama 3.1 and Llama 4. The paper has one version, dated May 2025, so its numbers describe those models and not today's. ## The number the benchmark does not report CRMArena-Pro scores the final answer against the ground truth and stops there. It does not report whether an agent's own claim of completion matched the outcome. In a deployment, the agent's summary may be the first thing a person sees, before anyone checks the records it changed. If the agent reports a refund to the wrong order as done, nothing in that report says otherwise. So the question for anyone deploying agents is narrower than "what share fail?" It is "of the jobs that fail, how many come back marked done, and what would catch them before they ship?" CRMArena-Pro does not answer that, and it was not designed to. ## How to read the next agent benchmark headline 1. **Which model?** Here, the 35% is one model, the best of nine, in one of two environments. 2. **Which setting?** Single-turn and multi-turn can differ by nearly 30 points: Gemini 2.5 Pro scored 58.3% and 30.0% on the same B2C environment. 3. **Success or failure?** Here, the 65% is the complement of the best model's 35.1% B2B multi-turn score, the most favorable reading the paper allows. 4. **What was scored?** An exact-match answer, a final state, or a judge's opinion are three different measurements. 5. **What did the agent report?** If the benchmark doesn't say, nobody knows how many of the failures would have looked like successes. That last question is the one we work on. Excellent's planned v1 workflow treats an agent's "done" as a claim and [re-runs frozen checks on the finished change](https://www.excellent.so/checks). The [published CLI](https://www.excellent.so/changelog) does not yet include that workflow. ## Sources - Huang et al., [CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions](https://arxiv.org/abs/2505.18878), Salesforce AI Research, arXiv v1, May 2025. Abstract, section 3.6 and Table 2. - [CRMArena-Pro dataset card](https://huggingface.co/datasets/Salesforce/CRMArenaPro), Hugging Face. - Luis Rijo, ["Salesforce study reveals enterprise AI agents fail 65% of multiturn tasks"](https://ppc.land/salesforce-study-reveals-enterprise-ai-agents-fail-65-of-multiturn-tasks/), PPC Land, June 29, 2025. - Mastufa Ahmed, ["Salesforce study warns against rushing LLMs into CRM workflows without guardrails"](https://www.cio.com/article/4008228/salesforce-study-warns-against-rushing-llms-into-crm-workflows-without-guardrails.html), CIO, June 17, 2025. Source: https://www.excellent.so/research/salesforce-35-percent-is-a-success-rate --- # Introducing Excellent > Verification for agentic tasks: attempts, checks, evidence, results, and receipts, running beside the agent you already use. The Excellent Team · June 26, 2026 · Company Excellent is a verification system for agentic tasks. It does not replace the agent you already use. Your agent still writes code, edits files, runs tools, and claims the task is finished. Excellent runs beside it and asks the question that matters next: what evidence proves the work? ## The agent does the work Start with a real task in a real repository. Excellent records the request, the repo state, the agent, the model, the relevant context, and the completion claim. That attempt becomes the unit of work. It can pass, fail, retry, or remain unchecked without losing what happened. ## Verification checks the work Excellent can attach proof from the places engineering teams already trust: - Git changes. - Test, lint, and build commands. - Browser checks and screenshots. - CI and deployment signals. - Human review notes. The agent's confidence is useful context. It is not the decision. ## Failed work comes back with evidence When an attempt fails, Excellent returns the concrete failure: the command, output, screenshot, missing criterion, or uncertainty that blocked verification. The next attempt starts with that context instead of a vague request to try again. ## Accepted work leaves a receipt When work is accepted, the receipt records what changed, what proved it, what could not be checked, and who approved the result. That matters for review, release notes, audits, and future agent comparisons. You can open the evidence later instead of reconstructing it from a chat transcript. It lives on the machine that produced it. Evidence does not sync between teammates yet. ## Better setups have to prove it Excellent also helps teams compare changes to the agent workflow itself: prompts, models, tools, context windows, retry rules, and check cadence. A candidate setup should be promoted because it performs better on held-out tasks, not because it felt better in one demo. ## Try it on a real repo Install Excellent locally, run one agent-authored task through the loop, and inspect the result. [Install Excellent →](https://www.excellent.so/install) Source: https://www.excellent.so/research/introducing-excellent --- # Why local verification matters for AI agents > Why local, inspectable evidence matters for AI agent work, and how request scope, checks, failures, and receipts make completion claims easier to examine. The Excellent Team · May 20, 2026 · Company · Verification The conventional wisdom is that AI belongs in the cloud: hosted models, hosted agents, hosted dashboards, hosted logs. That may be fine for experimentation. It is not enough for trusting agent-authored code. ## The model is not the proof A capable AI agent can edit files, run commands, open a browser, and explain what it did. None of that makes the work correct by itself. The proof is the surrounding evidence: the task, repo state, changed files, command output, screenshots, CI signals, deployment observations, and review decision. If that evidence is scattered across hosted tools and temporary transcripts, the team has to reconstruct trust by hand. ## Local changes the review Local verification means the receipt for an agent attempt is something you can inspect and keep, on your own machine, rather than a claim you have to take on trust. - The attempt records what was requested and what the agent claimed. - The checks record what ran, what passed, what failed, and what could not be checked. - The evidence records the artifacts behind the decision. - The result says verified, returned, or unchecked. That makes review concrete. A failed attempt comes back with the failed check. An accepted attempt leaves a receipt. A missing proof source becomes visible instead of being covered by a confident summary. ## Ownership matters more with agents When a team starts changing prompts, models, tools, context, and retry rules, it needs history. Which setup produced the verified result? Which one cost less? Which one needed less human help? Which one created a regression? You cannot answer those questions from vibes. You need receipts and held-out comparisons. That is why Excellent runs beside the agent you already use. The agent does the work. Excellent verifies the work. The evidence stays inspectable. [Install Excellent →](https://www.excellent.so/install) Source: https://www.excellent.so/research/why-local-first