Right-Sizing Tool Descriptions to Drive Enterprise-Agent Accuracy
What we learned about tool descriptions from running an enterprise agent on a hard benchmark.
Introduction
While building Surelock, our agent for enterprise operations, we kept hitting the same surprise: a tool's description affects task completion more than we expected. On a hard benchmark, with the agent and the model held fixed, we rewrote nothing but the descriptions and task success went up on every model we tried (Opus, Sonnet, and Haiku). The effect has a ceiling, though. Past a certain point, piling on more text makes results worse.
This post covers what worked, what backfired, and the description guidelines we settled on.
The benchmark
EnterpriseOps-Gym (EOG) is a ServiceNow benchmark of 512 tools across 8 business domains: CSM, ITSM, HR, Email, Calendar, Teams, Drive, and a cross-domain Hybrid set. Unlike static datasets, its tasks execute against live MCP servers and are graded by SQL verifiers that check the final state of the database rather than the sequence of actions. A task passes only if every verifier passes.
EOG tools ship with their own descriptions, averaging roughly 500 characters (about 75 words) per tool. The rich domains carry verbose API dumps: Calendar averages around 1,700 characters per tool and Teams around 1,100. Others are sparse, with Email under 100 characters per tool and CSM around 145.
Surelock, and how we ran it
Surelock is our ReAct-based agent. The full stack adds retrieval-first tool selection with an LLM reranker, planning, context compaction, and model flexibility: it can pick the underlying model and switch it at runtime, failing over between AWS Bedrock and Azure in either direction.
For this study we ran it deliberately stripped down, as plain ReAct in oracle mode (the task's tools pre-loaded, matching the benchmark's own protocol). With retrieval and reranking largely inactive, the numbers isolate a single variable: the quality of the tool descriptions. We ran on Claude Opus 4.6, Sonnet 4.6, and Haiku 4.5 via AWS Bedrock, and evaluated on the benchmark's seven single-domain splits of 561 tasks across CSM, ITSM, HR, Email, Calendar, Teams, and Drive.
The baseline throughout is that same agent on the tools' stock descriptions. Every gain below is measured against it, with the agent and model held fixed.
What worked: right-sizing, not maximizing
Tightening descriptions reliably flipped consistent failures into consistent passes. But "tightening" meant different things in different domains. Two real before/after cases make the point.
1. Strip the boilerplate the agent can't use (Teams).Task: "Archive the External Partners Hub channel, then create a private channel 'Backend Q2 2026 Planning.'" With the stock description the agent looped, repeatedly listing teams, channels, and users without converging, and it failed consistently across runs.
Stock archive_channel (890 chars), an HTTP/permissions spec:
Ours (554 chars), how to actually use it for the task:
Same tool. We removed the API trivia and told the agent how to identify the right channel. After that it passed consistently.
2. Add the interface where it's genuinely thin (Email).The stock update_vacation_settings description is a single sentence with no parameters, so the agent had to guess the argument format. It kept getting the time format wrong.
Stock update_vacation_settings (144 chars):
Ours (644 chars), the actual contract:
Once the enum values and the epoch-ms format were spelled out, the guessing stopped.
The guidelines we followed
These edits weren't ad-hoc. Prior work already establishes that description quality drives agent behavior. An empirical study of 856 MCP tools coined the term "tool description smells" and showed that such defects measurably misguide agents (Hasan et al., 2026), and tool calls are known to be brittle to small argument or format mistakes (ToolACE, 2024). Building on that, we wrote every description to a fixed set of guidelines. Beyond the usual clean-up, we added material the raw descriptions lacked: prerequisite steps (which lookups to call first), follow-up steps, usage guidance, and worked example values. The rules:
Write in the user's voice, not the API's. Describe how to accomplish the task rather than the HTTP contract. Strip endpoints, permission tables, and SDK trivia the model can't act on.
Document the interface precisely. Required params, exact enum values, formats such as epoch ms, and a concrete example value wherever a format is easy to get wrong. This is the argument-level precision that tool calls are most brittle to.
State prerequisites and follow-ups explicitly. Which tools to call before (to fetch IDs) and after (to complete the workflow).
Correct, don't coach. Only fix information that is genuinely missing or wrong for any caller. Don't add workarounds for a server's quirks.
Keep the environment out of it. Describe the tool, not the system behind it. No database tables, no grader logic, no vendor deployment notes.
Right-size, don't maximize. Aim for the shortest description that removes ambiguity. Adding more is not free, as the next section shows.
Not all of these additions carried equal weight. The prerequisite guidance clearly landed: the agent read it and sequenced its lookups correctly. The follow-up guidance was followed far less reliably. A meaningful share of our remaining failures are tasks where the agent nails the main action but skips a required next step. In principle the model should read the follow-up guidance and act on it, and where it doesn't, lightweight scaffolding such as explicit task tracking can help close the gap. We pick that thread up separately. Here the agent stays fixed and the focus stays on the descriptions.
Applied consistently, these guidelines lifted every model above its own stock-description baseline (Surelock, oracle mode, 3-run average):
Model | Stock descriptions | Enriched descriptions |
|---|---|---|
Claude Opus 4.6 | 44.1% | 48.1% |
Claude Sonnet 4.6 | 41.5% | 43.2% |
Claude Haiku 4.5 | 27.5% | 28.4% |
(3-run averages, controlled ablation: raw MCP descriptions vs enriched, oracle mode. Each figure is the mean of the seven per-domain success rates, as the benchmark reports them. The "Stock descriptions" column is the same agent on the raw MCP descriptions, our baseline. Only the descriptions change between the two columns.)
Reading the failures honestly
Across all 561 tasks a more complicated picture appears, and it's worth looking at from a few angles: how much a single run wobbles, where enrichment stops helping, which failing tasks are genuinely hard, and which are impossible as written.
Consistency
Consider the pass-any rate, the share of tasks solved on at least one of the three runs, next to the 3-run average:
Model | Stock (avg → pass-any) | Enriched (avg → pass-any) |
|---|---|---|
Claude Opus 4.6 | 44.1% → 59.8% | 48.1% → 65.8% |
Claude Sonnet 4.6 | 41.5% → 57.9% | 43.2% → 62.2% |
Claude Haiku 4.5 | 27.5% → 43.6% | 28.4% → 47.0% |
The gap is large and consistent for every model. Our strongest configuration, enriched Opus, solves nearly two-thirds of tasks on some run but only about 48% on average. That gap is the finding. These models can find a correct solution far more often than they can reproduce it. Reliable multi-step execution remains an open problem even for frontier models, which is why we report the average rather than the best run, and won't pretend the number is a hard floor.
Descriptions have a sweet spot
Enrichment helped until it didn't. As the easy wins ran out, the remaining failures changed character: the agent was choosing the right tools but mishandling advanced fields and multi-step sequencing. So we did the obvious thing and added more coverage: edge-case parameters, richer field specs, explicit handling notes. Performance started to fall.
Here is a representative case. A Calendar task that created and color-coded calendars passed consistently with a concise create_event description, then failed consistently after a later, more comprehensive revision. We hadn't touched the tool, only its description, growing it from 1,754 to 3,362 characters (the stock version, for reference, is 3,389). The additions:
Every addition was individually defensible. Together they buried the two fields this task actually needed, a title and a color_id, under detailed instructions for advanced options the task never touched. The agent began tripping over the conference-data warnings instead of just creating the event.
The calendar case is the clean one, a single tool with a regression you can point to. Just as often the effect was diffuse: extra verbosity spread across the whole catalog dragged performance down with no single culprit. Either way the lesson is the same. Enrichment isn't monotonic. Past the point where a description removes ambiguity, more text stops adding signal and starts subtracting it, and the gains plateau. We might expect a model to use whatever extra detail we hand it, but in practice it doesn't, which is why right-size, don't maximize is a rule rather than an afterthought.
Genuinely hard problems
Some tasks no model ever passes, and not because they're unfair. One Teams task restructures a team after staffing changes: archive a retired channel, remove two departed members, stand up a new private channel and add everyone who remains, create and publish a training webinar, and set up an audio call between two members. That is eight separate checks. Across every run and every model we tried, the agent clears most of them but never all eight. The parts it drops are the compound ones, such as the webinar that must be created and then published, or the call that has to reach exactly the right two people, where completing the first step but not the second still fails the check. Nothing is wrong with the task. In oracle mode the agent had every tool and field it needed and still dropped a step. That's an execution gap, not a knowledge gap.
Ungradable-as-written tasks
Others are impossible by construction. Two Teams tasks check the same create_call field against opposite expected values, one wanting source.identity.user.id and the othersource.identity.applicationInstance.id. No single tool description can satisfy both. Others do literal string matching on a value the agent produces correctly but with different casing.
That leaves a caveat and a to-do list. The caveat is that some of the "failures" are ungradable tasks, where the agent often does the work right and the grader can't credit it, and those we should discount. The rest is real work rather than hidden capability: the run-to-run wobble, the over-enrichment we introduced, and the multi-step goals the agent leaves half-finished are all gaps to close. Reading the failures honestly means telling those two apart.
Anti-patterns
The "dos" are the guidelines above. These are the failure modes to avoid. Each is a general trap, with the specific place we hit it.
Comprehensiveness crowds out the essentials. Documenting every advanced field buries the handful a given task actually needs. The
create_eventregression above is exactly this trap.Prescribing procedure overrides the model's own planning. Baking in rigid "always do X, then Y" sequencing removes the model's freedom to adapt, and its own plan is often better. Encode constraints such as valid values and prerequisites, not scripts.
Tool descriptions are now production code. A wording change to a description is a change to a live system's behavior. Revising one can silently alter or drop a field that was working. We replaced a simple
attendees: list of emailswith a heavier object form and broke the tasks that relied on it. Version them, review them, and diff each change against the tasks it touches, exactly as you would code.Vendor and API trivia is noise. Endpoints, permission matrices, SDK notes, and cloud-availability tables describe the system, not how the agent should use the tool. They dilute the signal and can mislead. The
archive_channelwin was mostly a matter of deleting this.
Finding the right tool
Everything above ran in oracle mode, with the task's tools handed to the agent up front, so the numbers would isolate description quality. In production nothing hands you the tools. The agent has to find the right one among hundreds first. That retrieval step is its own problem, and we evaluated Surelock's approach to it in isolation. Its reranker, an LLM scoring each candidate against the query, reaches 99% top-1 on BFCL and 78% top-1 / 96% top-3 on a hard 370-tool pool. That is 9 to 16 points over the two stages of the standard RAG retrieval pipeline (bi-encoder dense retrieval and a cross-encoder reranker) run on the identical setup. Precise descriptions and precise retrieval are two versions of the same bet, made at two points in the pipeline: give the model exactly what it needs to pick and call the right tool.
Takeaway
Model choice is a real lever, but it isn't the only one, and it's rarely the cheapest. Swap Opus for Haiku and you save several-fold on cost. Give that Haiku three attempts and it clears 47% of tasks, right where Opus lands on a single run at 48%. The other thing that moves the number is the system you wrap around the model: descriptions right-sized to what the agent needs, retrieval that surfaces the right tool, and orchestration that carries a multi-step goal to the end. Those levers are cheap, and they compound. Choose a capable model, then give the harness the effort it deserves rather than the afterthought it usually gets. That harness is what Surelock is built to be.
Model: Claude Opus 4.6 / Sonnet 4.6 / Haiku 4.5 via AWS Bedrock. Benchmarks: EnterpriseOps-Gym (ServiceNow), BFCL, τ-bench. EOG figures are 3-run averages across seven of the benchmark's eight domains (561 tasks) from a controlled ablation (raw MCP vs enriched descriptions, oracle mode, temp 0.2). Description statistics measured across all 512 EOG tools.