AI Coding Agent Says Tests Pass: Verify the Work

When an AI coding agent says "done" or "tests pass," treat that as a claim, not evidence. Verify the exact diff, run the documented test command yourself or in CI, confirm the intended tests actually executed, test the changed behavior directly, and check any external state with a read-only query. A green sentence in a chat is not a green build.

This guide gives you a repeatable verification loop for Claude Code, Codex, and other coding agents. It includes exact commands, expected results, a copyable evidence contract, and the failure modes behind false confidence.

What Counts as Proof?

A useful completion report connects each claim to an independent observation:

Do not ask the same agent to summarize its own summary. Verification should read the repository, test runner, browser, CI service, or provider that owns the state.

Prerequisites

Before you verify the work, collect:

• the task's acceptance criteria;

• the repository's documented test and lint commands;

• the expected list of changed files;

• a disposable branch, worktree, or CI job for controlled negative tests;

• read-only credentials for any external system the change affects.

Do not give a verification agent permission to merge, deploy, delete, or mutate production data. Its job is to observe and report.

Step 1: Freeze the Agent's Claims

Ask for a structured completion report before you run anything else:

List every behavior you changed. For each claim, provide:
1. the changed files and relevant diff;
2. the exact verification command you ran;
3. its exit code, test count, failures, and skips;
4. the expected result you observed;
5. anything you did not verify.
Do not rerun or modify anything while producing this report

Expected result: you have a finite list of claims that can be checked. Phrases such as "should work," "looks good," or "all set" are not verification results.

Step 2: Inspect the Repository State

Start with the machine-readable working-tree summary, then review the complete diff:

git status --porcelain=v1
git diff --stat
git diff

If the changes are staged, also run:

git diff --cached --stat
git diff --cached

Expected result: every changed file serves the requested task, generated files are intentional, and no credentials, unrelated refactors, debug logs, or disabled tests appear.

Git documents --porcelain as a stable format for scripts. That makes an empty result a stronger automation check than parsing the human-readable status output. It does not prove correctness, but it proves what the worktree currently contains.

For a stricter scope check, list the files changed from the branch point:

git diff --name-only "$(git merge-base HEAD origin/main)"...HEAD

Compare that list to the agent's report. An unmentioned migration, lockfile, snapshot, or configuration change deserves review before testing.

Step 3: Run the Exact Test Command and Capture the Exit Code

Use the repository's own documented command. For a pnpm project, a targeted example might look like this:

pnpm --dir apps/web test --testPathPattern=path/to/changed.test.ts
test_status=$?
printf 'test_exit_code=%s\n' "$test_status"
exit "$test_status"

Replace the path and command with the owning package's real test command. Do not substitute a faster command unless the repository says it is equivalent.

Expected result:

• exit code is 0;

• the intended test file or suite appears in the output;

• at least one relevant test ran;

• no unexpected tests were skipped, filtered out, or marked todo;

• the process reached its final summary instead of timing out or remaining in watch mode.

GitHub Actions maps exit code 0 to success and a nonzero exit code to failure. A log line containing the word "passed" cannot override the process status.

Step 4: Prove the Test Detects the Bug

A passing test is useful only if it can fail for the right reason. In a disposable branch or worktree:

1. temporarily remove or invert the smallest part of the fix;

2. rerun only the regression test;

3. confirm it fails with the expected assertion;

4. restore the fix;

5. rerun the same command and confirm it passes.

Expected result: one controlled code change moves the test from red to green. If the test passes both times, it does not cover the claimed regression.

Do this only in a disposable environment. Never use broad cleanup commands to restore a working tree that contains someone else's changes.

Step 5: Verify the Behavior, Not Just the Function

Tests can be green while the user-visible path remains broken. Exercise the narrowest real workflow that crosses the changed boundary.

For a web change, prefer assertions over screenshots alone:

await page.goto('http://localhost:3000/example');
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();

Expected result: the assertion observes the final state a user depends on. Playwright's web assertions retry until the expected state appears or the timeout expires, which avoids replacing evidence with an arbitrary sleep.

For an API change, send one valid request and one deliberately invalid request. Record the status code and response body for both. For a migration, query the expanded schema with old and new application contracts before removing compatibility.

Step 6: Verify External State Independently

If the task changes a pull request, issue, document, CRM record, payment object, or deployment, local tests cannot prove the final provider state. Query the system that owns it with a read-only operation.

With Aident Loadout, first confirm the account and Vault connection:

aident account auth status
aident vault vault --action status

Then search for the relevant read-only Action and inspect its current schema before execution:

aident capabilities search \
  --query "get pull request checks" \
  --types '["action"]' \
  --targetEnv staging

aident capabilities get --name "<exact Action name from search>"

Run the exact Action once with the repository and pull request from your task. Expected result: the provider reports checks for the same head commit you reviewed, and each required check is complete and successful.

Do not copy a provider token into the prompt. Aident Vault keeps the credential outside the agent's context while the read-only Action returns the evidence you need.

Step 7: Produce a Claim-to-Evidence Report

End with a table that a reviewer can audit without replaying the entire session:

If any row is unknown, say unknown. Do not convert missing access or a timed-out command into a passing result.

Common Reasons "Tests Pass" Is Misleading

The Agent Ran Tests From the Wrong Directory

The command succeeds but discovers a different package or no tests. Check the working directory, test-path output, and final test count.

The Test Filter Matched Nothing

Some runners exit successfully when no tests match, depending on configuration. Confirm the named test file and relevant case appear in the output.

A Shell Wrapper Swallowed the Exit Code

Pipes, background processes, and commands followed by unconditional success messages can hide failure. Capture and propagate the real test process status.

The Test Proves the Mock, Not the Integration

A mocked provider can validate application branching, but it cannot prove current OAuth scopes, remote schemas, or production state. Add a harmless read-only provider check.

The UI Screenshot Is Stale or Ambiguous

A screenshot can show the right pixels without proving the click, URL, saved record, or accessibility state. Pair it with DOM, network, or provider assertions.

The Fix Has No Negative Control

If the regression test never fails without the fix, it may exercise the wrong path. Run the controlled red-then-green check in a disposable environment.

CI Passed for a Different Commit

Status checks belong to a commit. Compare the checked head SHA to the commit you reviewed before treating the PR as verified.

A Copyable Verification Prompt

Verify this coding task without changing files or external state.

1. Translate the task into explicit claims.
2. Inspect git status and the complete diff.
3. Run the repository's documented targeted tests and owning suite.
4. Record exact commands, exit codes, test counts, failures, and skips.
5. In a disposable worktree, prove the regression test fails without the fix
   and passes with it.
6. Exercise the changed user or API path with explicit assertions.
7. Use connected read-only Actions to verify any external state against the
   exact commit or object ID.
8. Return a claim-to-evidence table. Mark anything unverified as unknown.

Do not merge, deploy, delete, or perform provider writes

Why This Verification Loop Works

The loop separates authorship from evidence. The diff proves scope. The exit code and test count prove what ran. A red-then-green check proves the test can detect the regression. A direct assertion proves the changed behavior. A read-only provider query proves external state.

This is the same useful structure seen in Aident's Ollama networking guide: name the exact problem, answer it immediately, give reproducible steps, state expected results, and diagnose concrete failures. It does not assume that one popular topic will transfer to another.

For related safeguards, define repository-specific review rules in AGENTS.md, learn how to investigate security-review false positives, and set boundaries that prevent a coding agent from deleting files.

Use Aident Loadout to verify one real external claim with a schema-inspected, read-only Action. Measure success as one provider result tied to the exact commit or object ID you reviewed, without copying a provider credential into the prompt.

Sources

• Git status documentation, Git, accessed July 25, 2026

• Git diff documentation, Git, accessed July 25, 2026

• About status checks, GitHub, accessed July 25, 2026

• Setting exit codes for actions, GitHub, accessed July 25, 2026

• Playwright assertions, Playwright, accessed July 25, 2026

• Writing Playwright tests, Playwright, accessed July 25, 2026

Refresh this guide when your repository changes its test commands, CI provider, protected-branch rules, or external verification contract.