Benchmark

Ratel's numbers come from an open harness, ratel-ai/ratel-bench, and the published results live at ratel.sh/benchmark.

Three arms

Every scenario runs three ways on the same model:

• control-baseline: the entire candidate tool pool is exposed to the agent on every turn.
• control-oracle: only the gold tool is exposed; the accuracy upper bound.
• ratel-full: the model sees a handful of targeted tools plus the capability tools, regardless of how large the underlying pool is.

The gap between baseline and ratel-full is the token saving; the gap to oracle is the accuracy headroom.

Two suites

• BFCL: the Berkeley Function-Calling Leaderboard, simple and multiple splits, with pool sizes swept across realistic MCP setups (30 → 600 tools).
• Skills suite: agent tasks over six datasets (bigcodebench, champ, logicbench, medcalcbench, theoremqa, toolqa), measuring retrieval and completion on multi-step work.

What gets measured

Per model, arm, and split: task completion, tool selection accuracy, recall, mean total tokens, and latency. A separate retriever evaluation sweeps the pool sizes (precision, recall, MRR, nDCG).

Where results live

ratel.sh/benchmark renders the reports as interactive charts and tables, methodology included. The site regenerates at build time from ratel-bench's published reports, so it always tracks the source of truth.

For the headline numbers and what they mean in your agent, see Efficiency.