Ratel Benchmark

Ratel, measured

No corporate fluff. Just raw data and sharp claws.

Evaluated with golden industry standards, measured with BFCL v3.

Token Spend by Model

Mean total tokens per task, one row per model. Oracle = only the gold tools in context (the floor); With Ratel = the search_tools gateway; Baseline = the full tool pool. The dumbbell shows how far Ratel pulls each model back toward the oracle floor. Most recently benchmarked models are on top.

Nemotron Super 3 120bNemotron Super 3 120b36.2k1.2k3.8kGPT Oss 120b 1.0GPT Oss 120b 1.015.4k1.1k2.9kGPT 6.1 SolGPT 6.1 Sol14.8k4682.0kClaude Haiku 5.5Claude Haiku 5.540.7k1.6k4.4kGPT 6 LunaGPT 6 Luna15.1k5202.6kClaude Opus 5.5Claude Opus 5.538.5k1.6k4.2kGPT 6 AstraGPT 6 Astra14.8k4772.0kMistral Large 3 675b InstructMistral Large 3 675b…27.9k6592.7kGPT Oss 20b 1.0GPT Oss 20b 1.015.9k1.1k3.9kGemma 4 26b A4bGemma 4 26b A4b26.3k2.2k3.0kGemma 4 31bGemma 4 31b25.3k5442.4kQwen3 Coder NextQwen3 Coder Next36.2k1.1k3.8kClaude Sonnet 5.5Claude Sonnet 5.538.4k1.5k4.1kClaude Fable 5.1Claude Fable 5.138.3k1.5k4.1k
Oraclegold tools onlyWith Ratelsearch_tools gatewayBaselinefull tool pool

How Ratel effect agent accuracy and token cost

Pick a model. The bars compare mean total tokens per task; the radar overlays all five metrics for With Ratel (the search_tools gateway) vs Without Ratel (the full tool pool). On Token efficiency the axis shows tokens spent from the center out, so Without Ratel spikes there. Hover any plot for exact values.

Mean total tokens per task

−89% tokens
Without RatelClaude Sonnet 5.5 · full tool pool
0
With RatelClaude Sonnet 5.5 · search_tools gateway
0
Anthropic · direct

Claude Sonnet 5.5

Task completionTool selectionRecallToken efficiencySpeed
Task completion
92.3%
Without Ratel 94.5%
Tool selection
97.5%
Without Ratel 98.5%
Recall
95.1%
Without Ratel 96.6%
Token cost
4,098
Without Ratel 38,391
−89%
p50 latency
3,568 ms
Without Ratel 3,465 ms

Tool Selection by LLM

Pooled over the simple + multiple splits. Without Ratel = full tool pool in context; With Ratel = search_tools gateway; Oracle = only the gold tools in context (the ceiling).

(scenarios = 599, tool-pool size = 100)

ModelArmTask completionTool selectionRecallMean total tokens
Nemotron Super 3 120bWithout Ratel92.0%97.7%95.4%36,180
Oracle92.0%99.0%96.3%1,191
With Ratel83.8%89.5%86.9%3,815
GPT Oss 120b 1.0Without Ratel85.8%94.7%90.7%15,429
Oracle84.6%92.7%89.7%1,089
With Ratel82.0%92.0%87.6%2,888
GPT 6.1 SolWithout Ratel80.3%94.8%90.0%14,806
Oracle83.8%97.2%92.5%468
With Ratel80.5%94.5%89.7%1,961
Claude Haiku 5.5Without Ratel93.2%99.0%96.4%40,666
Oracle92.8%99.0%96.3%1,597
With Ratel88.5%95.3%92.3%4,443
GPT 6 LunaWithout Ratel85.3%98.0%93.6%15,087
Oracle84.1%97.7%93.3%520
With Ratel81.1%95.8%90.8%2,579
Claude Opus 5.5Without Ratel96.2%98.8%98.0%38,497
Oracle94.5%99.3%97.4%1,608
With Ratel94.3%98.2%96.6%4,227
GPT 6 AstraWithout Ratel80.6%94.2%89.5%14,786
Oracle82.3%95.2%90.6%477
With Ratel79.1%92.5%87.7%2,013
Mistral Large 3 675b InstructWithout Ratel88.1%96.2%93.0%27,944
Oracle91.8%99.3%96.0%659
With Ratel88.5%94.7%91.8%2,654
GPT Oss 20b 1.0Without Ratel81.6%90.7%87.1%15,910
Oracle84.0%96.3%91.8%1,077
With Ratel79.0%87.5%83.9%3,895
Gemma 4 26b A4bWithout Ratel85.5%97.5%91.0%26,262
Oracle78.3%99.7%88.2%2,219
With Ratel84.8%96.8%91.0%3,025
Gemma 4 31bWithout Ratel87.8%97.8%93.3%25,270
Oracle87.6%99.7%94.6%544
With Ratel86.8%97.5%93.2%2,405
Qwen3 Coder NextWithout Ratel91.5%98.2%95.2%36,207
Oracle94.0%99.3%97.3%1,095
With Ratel89.5%96.0%93.4%3,806
Claude Sonnet 5.5Without Ratel94.5%98.5%96.6%38,391
Oracle94.2%99.8%97.4%1,526
With Ratel92.3%97.5%95.1%4,098
Claude Fable 5.1Without Ratel96.2%98.5%97.5%38,310
Oracle94.8%99.3%97.8%1,451
With Ratel93.5%97.2%95.7%4,054

Retrieval evaluation

Evaluation of retriever engine. Pick a pool size — the bars and tables show the mean ranking metrics at top k = 1,3,5. Accuracy = a gold tool in the top-K; complete = every gold tool in the top-K; gold = share of queries whose gold tool was retrievable.

Tool pool size

Simple (single gold tool)

100%98%96%94%92%90%
Accuracy
Complete
Recall
MRR
nDCG
K=1
Accuracy
Complete
Recall
MRR
nDCG
K=3
Accuracy
Complete
Recall
MRR
nDCG
K=5
KaccuracycompleterecallMRRnDCGgold sim.
197.0%97.0%0.9700.9700.97097.0%
399.0%99.0%0.9900.9790.98299.0%
599.5%99.5%0.9950.9800.98499.5%

Multiple (several gold tools)

100%98%96%94%92%90%
Accuracy
Complete
Recall
MRR
nDCG
K=1
Accuracy
Complete
Recall
MRR
nDCG
K=3
Accuracy
Complete
Recall
MRR
nDCG
K=5
KaccuracycompleterecallMRRnDCGgold sim.
196.5%96.5%0.9650.9650.96596.5%
3100.0%100.0%1.0000.9810.986100.0%
5100.0%100.0%1.0000.9810.986100.0%