An agent calls a data tool. The call returns HTTP 200, valid JSON, the schema the agent expected. The agent reads it, writes a confident summary, and moves on. The work_email field was null. Nothing threw. Nothing logged an error. The agent just built the next three steps on a hole in the data and never noticed.
This is the failure that agent evaluation usually misses. Most eval writing grades the final answer: was the summary good, did the model follow instructions. That catches the model being wrong. It does not catch the tool being empty. And empty is common, because a single enrichment provider tends to return a complete contact for only 35 to 50 percent of a given list. A 200 response is an infrastructure fact, not a semantic one. The connection worked. Whether the agent got good data is a separate question, and it needs its own test.
Grounding a model in real tool data helps, but it does not close the gap. A Stanford study of leading AI legal research tools, all of them retrieval-grounded, found they hallucinate between 17 and 33 percent of the time, against 43 percent for ungrounded GPT-4. Grounding roughly halved the error and left one in six to one in three answers wrong. The lesson for anyone wiring a tool into an agent: the tool is where you get the most return, and it is still not self-checking. You evaluate it or you ship the leftover error rate.
This post builds a small test rig for that. It assumes you already have an agent that calls a tool. The prospect research agent post covers building one; this is what you wrap around it before you trust it. Code is in Python and JavaScript.
What "evaluate a tool call" actually means
Tool-call evaluation has two layers, and they fail independently.
The first is the trajectory: did the agent pick the right tool, pass the right arguments, in the right order, and use the result faithfully. Anthropic's guidance on writing tools for agents recommends collecting the total number of tool calls, tool errors, runtime, and token consumption, because patterns there are diagnostic. Lots of redundant calls suggest a pagination or limit problem. Lots of invalid-parameter errors suggest the tool description needs work.
The second layer is the one the trajectory hides: the data quality of what came back. The agent can pick the perfect tool, pass perfect arguments, and receive a record that is half empty. That call scores fine on trajectory and fails the task. Reliability over many steps is hard to guarantee. When the tau-bench benchmark introduced its consistency metric for multi-turn tool tasks in 2024, leading function-calling agents passed the same task on all eight retries less than a quarter of the time. Today's best models score higher, but multi-turn tool reliability is still not a given, and feeding an agent quietly-empty data is how you make it worse.
So the rig checks both. Trajectory metrics, then output assertions on the data itself.
Capturing the call
Wrap the tool so every call records what happened. This is the raw material for both layers.
import os
import time
from dataclasses import dataclass, field
import requests
@dataclass
class ToolCall:
args: dict
ok: bool
latency_ms: float
payload: dict = field(default_factory=dict)
error: str | None = None
def enrich(email: str) -> ToolCall:
start = time.perf_counter()
try:
resp = requests.post(
"https://api.datalegion.ai/person/enrich",
headers={"API-Key": os.environ["DATALEGION_API_KEY"]},
json={"email": email},
timeout=5,
)
except requests.RequestException as exc:
return ToolCall({"email": email}, False, _ms(start), error=str(exc))
elapsed = _ms(start)
if not resp.ok:
return ToolCall({"email": email}, False, elapsed, error=f"http {resp.status_code}")
return ToolCall({"email": email}, True, elapsed, payload=resp.json())
def _ms(start: float) -> float:
return round((time.perf_counter() - start) * 1000, 1)function ms(start) {
return Math.round((performance.now() - start) * 10) / 10;
}
async function enrich(email) {
const start = performance.now();
let resp;
try {
resp = await fetch("https://api.datalegion.ai/person/enrich", {
method: "POST",
headers: {
"API-Key": process.env.DATALEGION_API_KEY,
"Content-Type": "application/json",
},
body: JSON.stringify({ email }),
signal: AbortSignal.timeout(5000),
});
} catch (err) {
return { args: { email }, ok: false, latencyMs: ms(start), payload: {}, error: String(err) };
}
const latencyMs = ms(start);
if (!resp.ok) {
return { args: { email }, ok: false, latencyMs, payload: {}, error: `http ${resp.status}` };
}
return { args: { email }, ok: true, latencyMs, payload: await resp.json(), error: null };
}Now the success flag and latency give you the trajectory metrics for free, and the payload is what the assertions run against.
Asserting on the output
This is the part that catches the 200-OK-but-empty failure. A schema check confirms the response parsed and has the right shape. A fill-rate check confirms the fields you actually need are populated, not just present. They are different tests. A field can exist in the schema and still be null.
REQUIRED_FIELDS = ["job_title", "company_name", "work_email"]
def assert_output(call: ToolCall) -> list[str]:
issues: list[str] = []
if not call.ok:
return [f"call failed: {call.error}"]
matches = call.payload.get("matches")
if not isinstance(matches, list):
return ["malformed: no matches array"]
if not matches:
return ["no match found"]
person = matches[0].get("person", {})
# The fill-rate check: present in schema is not the same as populated.
for f in REQUIRED_FIELDS:
if not person.get(f):
issues.append(f"required field empty: {f}")
return issuesconst REQUIRED_FIELDS = ["job_title", "company_name", "work_email"];
function assertOutput(call) {
if (!call.ok) return [`call failed: ${call.error}`];
const matches = call.payload.matches;
if (!Array.isArray(matches)) return ["malformed: no matches array"];
if (matches.length === 0) return ["no match found"];
const person = matches[0].person || {};
const issues = [];
// The fill-rate check: present in schema is not the same as populated.
for (const f of REQUIRED_FIELDS) {
if (!person[f]) issues.push(`required field empty: ${f}`);
}
return issues;
}Run that against a record where the API matched the person but had no work email, and the check returns ["required field empty: work_email"] where a status-code check would have returned nothing. That one assertion is most of the value.
Using the confidence the data already carries
Good data tools return their own confidence signals, and an eval should read them instead of trusting every value equally. The Person Enrich response tags emails, phones, and locations with a confidence level and a num_sources count: how many independent sources agreed on that value. Treat agreement across sources as a built-in cross-check, and let the eval set a floor.
def assert_confidence(call: ToolCall, min_sources: int = 2) -> list[str]:
issues: list[str] = []
person = call.payload.get("matches", [{}])[0].get("person", {})
for email_obj in person.get("emails", []):
if email_obj.get("current") and email_obj.get("type") == "professional":
if email_obj.get("confidence") == "low":
issues.append(f"low-confidence email: {email_obj['address']}")
if email_obj.get("num_sources", 0) < min_sources:
issues.append(f"single-source email: {email_obj['address']}")
return issuesfunction assertConfidence(call, minSources = 2) {
const issues = [];
const person = call.payload.matches?.[0]?.person || {};
for (const emailObj of person.emails || []) {
if (emailObj.current && emailObj.type === "professional") {
if (emailObj.confidence === "low") issues.push(`low-confidence email: ${emailObj.address}`);
if ((emailObj.num_sources || 0) < minSources) issues.push(`single-source email: ${emailObj.address}`);
}
}
return issues;
}A value confirmed by one source is not the same as a value three sources agree on, and an agent that treats them identically is overconfident by construction. Cross-source agreement as a verification technique is a recommended practice rather than something with a published benchmark, so tune the floor to how much risk the downstream step carries. A draft email can run on a single-source address; a billing decision should not.
Judging faithfulness
The last gap is the agent inventing detail the tool never returned. Schema and fill-rate checks confirm the data arrived. They don't confirm the agent used only that data. For that, compare the agent's claim against the tool payload and ask whether the payload supports it. This is where an LLM judge earns its place, because "is this claim grounded in this JSON" is hard to encode as a rule.
import anthropic
client = anthropic.Anthropic()
JUDGE = """You are grading whether a claim is supported by source data.
SOURCE (the only facts that exist):
{source}
CLAIM (what the agent wrote):
{claim}
Answer with exactly one word: SUPPORTED if every fact in the claim
appears in the source, or UNSUPPORTED if the claim adds, changes, or
infers anything not in the source."""
def judge_faithfulness(claim: str, source: dict) -> bool:
msg = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=5,
messages=[{
"role": "user",
"content": JUDGE.format(source=source, claim=claim),
}],
)
return msg.content[0].text.strip().upper().startswith("SUPPORTED")import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
const JUDGE = `You are grading whether a claim is supported by source data.
SOURCE (the only facts that exist):
{source}
CLAIM (what the agent wrote):
{claim}
Answer with exactly one word: SUPPORTED if every fact in the claim
appears in the source, or UNSUPPORTED if the claim adds, changes, or
infers anything not in the source.`;
async function judgeFaithfulness(claim, source) {
const msg = await client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 5,
messages: [{
role: "user",
content: JUDGE.replace("{source}", JSON.stringify(source)).replace("{claim}", claim),
}],
});
return msg.content[0].text.trim().toUpperCase().startsWith("SUPPORTED");
}A prompt this strict turns hallucination into a failing grade. If the agent wrote "VP of Engineering, based in Austin" and the payload has the title but no location, the claim is UNSUPPORTED, and you caught an inference before it reached a customer. This is the same split the established eval frameworks use. DeepEval separates tool correctness from argument correctness and can validate the tool's returned output; LangSmith scores the full trajectory with code graders plus an LLM judge against a rubric. The rig here is the hand-rolled version of the same idea.
Running it as a suite
Assertions are only an eval when you run them against known cases on every change. Anthropic suggests starting small, with 5 to 10 real or synthetic examples, each paired with a verifiable outcome. A golden set for a data tool is a handful of inputs where you know what the answer should be.
GOLDEN = [
{"email": "known.good@example.com", "expect_match": True},
{"email": "no.such.person@example.com", "expect_match": False},
]
def run_suite(cases: list[dict]) -> None:
failures = 0
for case in cases:
call = enrich(case["email"])
issues = assert_output(call) + assert_confidence(call)
matched = call.ok and bool(call.payload.get("matches"))
if matched != case["expect_match"]:
issues.append(f"expected match={case['expect_match']}, got {matched}")
status = "PASS" if not issues else "FAIL"
if issues:
failures += 1
print(f"[{status}] {case['email']} ({call.latency_ms}ms) {issues}")
print(f"\n{len(cases) - failures}/{len(cases)} passed")const GOLDEN = [
{ email: "known.good@example.com", expectMatch: true },
{ email: "no.such.person@example.com", expectMatch: false },
];
async function runSuite(cases) {
let failures = 0;
for (const c of cases) {
const call = await enrich(c.email);
const issues = [...assertOutput(call), ...assertConfidence(call)];
const matched = call.ok && Boolean(call.payload.matches?.length);
if (matched !== c.expectMatch) {
issues.push(`expected match=${c.expectMatch}, got ${matched}`);
}
const status = issues.length === 0 ? "PASS" : "FAIL";
if (issues.length) failures++;
console.log(`[${status}] ${c.email} (${call.latencyMs}ms)`, issues);
}
console.log(`\n${cases.length - failures}/${cases.length} passed`);
}Run this in CI against your golden set, and a provider change, a schema drift, or a coverage drop in a field your agent depends on shows up as a failing test instead of a confident, wrong answer in production. Track the trajectory metrics over time too: rising latency or a climbing empty-field rate is the early signal that the tool, not the model, is what degraded.
The reason this matters more for agents than for a human-driven workflow is that the human was the check. A person reading an enriched record notices an empty field and goes looking. An agent at scale removes that pause, which is the whole argument for the data layer under any AI system. The rig puts the check back. The full Person Enrich schema, including the confidence and source fields the assertions read, is at www.datalegion.ai/docs/person-data/schema. Wrap your tool, write five golden cases, and make the empty field fail loudly.
Build it with a coding agent
To stand up the evaluation suite quickly, paste this into Claude Code, Codex, or Cursor:
Build an evaluation suite for an agent tool that calls the Data Legion Person
Enrich API.
- POST https://api.datalegion.ai/person/enrich
- Header: API-Key: $DATALEGION_API_KEY
- Body: {"email": "<address>"}
- Read matches[0].person. Email objects carry confidence
("high"/"moderate"/"low"), validation_status, and num_sources.
References for context (read these first):
- The walkthrough this is based on:
https://www.datalegion.ai/blog/post/evaluating-agent-tool-calls
- Person Enrich API: https://www.datalegion.ai/docs/api-reference/person-enrichment
- Person field schema: https://www.datalegion.ai/docs/person-data/schema
- How confidence and sources are scored:
https://www.datalegion.ai/docs/data-quality/quality-scoring
Requirements:
- Wrap the tool call so each run records arguments, latency, success, and the
returned payload.
- Output assertions: required fields are present (fill rate), and flagged
values (low-confidence or single-source emails) are caught.
- An LLM-as-judge faithfulness check using the Anthropic SDK and model
claude-sonnet-4-6, comparing the tool output against the source record.
- A suite runner over a small golden set that fails loudly in CI.
Before you start, ask me which language and framework to use. Sketch a short
plan, then build it with tests and a short README. When you finish, check the
result against each requirement above and tell me what is covered.