Agent Vendor Verifier is a tool for validating the fidelity and reliability of tool-calling LLMs across different vendors.

Agent Vendor Verifier

Agent Vendor Verifier is a benchmarking framework for evaluating Agent tool-call effectiveness.

The benchmark aggregates these dimensions into a single, comparable fusion score (IRF), enabling fair cross-vendor comparison for agent-style tool usage.

Agent Vendor Verifier is built upon K2-Vendor-Verifier.

Why Agent Vendor Verifier?

• Multi-dimensional metrics: Go beyond “was a tool called” by measuring:
  • correctness,
  • schema compliance,
  • request success and stability,
  • latency and throughput.
• Comparable fusion score: combine heterogeneous metrics into a single score for ranking and model/vendor selection.

Metrics

For each sample, the benchmark records the finish_reason (e.g. tool_calls, stop, others) and optional tool-call validation results.

F1 Score

F1 score measures whether a model triggers tool calls on the correct samples, compared against a designated baseline vendor for the same model.

• A higher F1 indicates closer alignment with the baseline on when to issue tool calls.