What 4,187 Production AI Agent Tool Calls Reveal

Across 4,187 unique production AI agent Action executions, 3,277 ended successfully and 910 ended in failure. That is a 78.3 percent terminal success rate.

That number is useful, but it is not a universal score for AI agents. It is a measured cohort from one production execution layer: 133 Actions across 32 integrations and two execution backends, observed over 13 complete UTC days from August 4 through August 16, 2026. It measures whether an external Action reached a successful terminal state. It does not measure whether the agent chose the right Action, satisfied the user's full task, or produced a correct final answer.

The more important finding is what sits behind the aggregate. Most recorded failures were provider or Action errors, but 14.4 percent of failures were explicit setup, funding, or approval gates. Successful executions also had a long latency tail: 2.5 seconds at the median and 18.4 seconds at the 95th percentile.

The Production Snapshot

Daily terminal success ranged from 52.2 to 94.5 percent, with a median daily rate of 78.7 percent. The lowest-rate days also had lower volume, so the range should not be treated as a stable service-level objective. It is evidence that one blended monthly percentage can hide meaningful daily variation.

How the Cohort Was Measured

The analysis used terminal Aident Loadout execution events in PostHog. It:

1. included only production traffic;

2. excluded internal users;

3. required a non-empty execution ID;

4. grouped events by that execution ID;

5. kept the latest terminal outcome for each execution; and

6. calculated duration percentiles from the terminal event.

The query window began on July 20, but eligible terminal telemetry in this sample began on August 4. The reported cohort therefore covers 13 complete observed days, not 28 days.

This method prevents a retry or duplicate terminal event from being counted as a separate execution. It also creates boundaries. Executions with no terminal event are absent. The data does not identify whether a user ultimately completed a larger task, and it does not compare models, providers, or Actions under controlled conditions. No prompts, provider payloads, user identities, or private Action inputs were used in the analysis.

A Failure Rate Is Not a Retry Policy

The 910 terminal failures divided into five recorded categories:

The broad Action error category needs deeper provider-specific classification before it can support a remediation decision. It can include transient transport errors, deterministic input errors, provider rejections, or ambiguous completion. Those cases should not share one automatic retry rule.

The remaining categories are even clearer. Missing credits, a disconnected integration, and required risk acknowledgement do not improve when the same request is replayed. Together they account for 131 executions, or 14.4 percent of failures. They need funding, connection, or approval state to change first.

Use the AI agent tool failure and retry matrix to separate transient failures from contract, permission, budget, state, and ambiguous-side-effect failures before adding retries.

Tail Latency Matters More Than the Median

A successful Action completed in 2.5 seconds at p50, but the p95 was 18.4 seconds. The p95 was more than seven times the median. Failed calls were faster at both percentiles, partly because deterministic rejection can happen before a provider finishes useful work.

That produces three practical rules:

• Measure p50, p95, and p99 by Action, not only a blended average.

• Separate provider execution time from total agent task time.

• Track retry time and approval wait time independently, because neither is visible in a single terminal duration.

An agent with acceptable median latency can still feel unreliable when one in twenty calls takes much longer. A workflow with several sequential Actions compounds that tail even if each individual Action is healthy.

Tool Success Is Only One Layer of Agent Reliability

A successful terminal Action proves that the execution layer returned success. It does not prove that the agent:

• selected the correct Action;

• supplied the semantically correct arguments;

• respected the user's authority and risk boundary;

• interpreted the result correctly;

• avoided an unnecessary or duplicate call; or

• completed the user's full objective.

This distinction also appears in controlled research. The original tau-bench paper evaluates whether tool-using agents reach the correct final database state across a conversation, not merely whether individual calls parse. Its historical experiments found that leading function-calling agents completed fewer than half of the tasks and were inconsistent across repeated trials.

A 2026 paper, Towards a Science of AI Agent Reliability, separates reliability into consistency, robustness, predictability, and safety. Its twelve proposed metrics are independent of raw accuracy. That reinforces the need to report repeatability, response to perturbations, calibration, and failure severity alongside a terminal execution rate.

Microsoft's ThinkingBox-Bench makes the repeatability denominator concrete. Across 507 tasks, 12 models, and 20 trials per task, it distinguishes whether any trial succeeds from whether every trial succeeds. Its trace review attributed 77.5 percent of failed runs to tool usage, including unrecovered tool errors and failed lookups. That taxonomy is not directly comparable with Aident's production event categories, but both analyses show why teams must inspect trajectories and final state instead of inferring completion from one successful call.

More recent guidance reaches the same measurement conclusion. Anthropic's agent eval guide recommends evaluating the full harness, tool calls, environment, and multi-turn trajectory. OpenAI's evaluation guidance argues for contextual evals built around the specific workflow rather than relying only on frontier benchmarks. OpenAI's third-party evaluation playbook also calls out harness choice, tool access, scoring, and environment validity as factors that can materially change a result.

Build a Reliability Scorecard in Five Layers

Use one scorecard with five separate denominators:

Do not collapse these layers into one "agent success rate." A higher execution success rate can coexist with worse task quality if the agent calls the wrong Action more confidently. A lower execution rate can also reflect healthy policy gates that correctly stop unauthorized work.

For each layer, preserve the cohort definition, time window, sample size, version, and missing-event policy. Then compare like with like after a model, tool catalog, routing rule, or provider contract changes.

Benchmark Your Own Workflow

Start with a narrow workflow and a fixed test set:

1. Define the intended final state and unacceptable side effects.

2. Include ordinary, ambiguous, invalid-input, permission-denied, rate-limited, and timeout cases.

3. Run repeated trials against the exact production tool schema and harness.

4. Record decision, contract, execution, interpretation, and task outcomes separately.

5. Review the longest successful calls and every ambiguous write before setting retry rules.

Then compare the controlled eval with live aggregate telemetry. The eval explains what should happen under known conditions. Production telemetry shows what actually happens under real traffic, provider latency, account state, and changing inputs. You need both.

The Aident Loadout Action usage audit provides a bounded way to inspect Action identity, caller source, terminal status, duration, and credits without collecting prompts or provider payloads.

Use This Number as a Baseline, Not a Promise

The 78.3 percent terminal success rate is a baseline for this observed cohort. It should not be marketed as an industry-wide failure rate, a model leaderboard, or a service-level promise.

Its value is diagnostic. It shows why teams need more than one reliability number: a deduplicated terminal outcome, a failure taxonomy, latency percentiles, and an end-to-end task assertion. With those layers in place, improvements can target the actual constraint instead of masking it with retries.

Follow https://aident.ai/SETUP.md, then benchmark one workflow. Start read-only, preserve unique execution IDs, and require explicit approval before any consequential test Action.

Sources

• Aident Loadout production terminal execution telemetry, August 4-16, 2026

• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

• Towards a Science of AI Agent Reliability

• ThinkingBox: Measuring whether agents finish the job, Microsoft

• Demystifying evals for AI agents, Anthropic

• How evals drive the next chapter in AI for businesses, OpenAI

• A shared playbook for trustworthy third party evaluations, OpenAI

Refresh this benchmark when the production telemetry contract changes, at least 30 complete observed days are available, a materially larger Action or integration cohort is available, or a controlled task-level eval can be joined without exposing private data.