AI agents fail in ways that look minor in a demo and expensive in production. A model can answer the right business question, then call the wrong tool, call the right tool in the wrong order, or pass arguments that are technically valid but operationally dangerous. Those failures are easy to miss if your test suite only checks final text output.

If you want to test AI agent tool selection failures before they reach users, you need to treat tool routing as a first-class behavior. That means testing the agent’s decision path, not just the end result. It also means building cases for browser workflows, API workflows, and chained actions where the failure is not obvious until a later step breaks.

This tutorial focuses on three common failure modes:

  1. The agent picks the wrong tool.
  2. The agent picks the right tool, but in the wrong order.
  3. The agent picks the right tool and order, but uses the wrong parameters.

The goal is not to make agents perfect. The goal is to make failures visible, repeatable, and cheap to catch in CI.

Why tool selection is its own test surface

Most teams start with output assertions. For a chat agent, that might mean checking whether the final response contains the right answer. For a workflow agent, it might mean checking whether a support ticket was updated or a browser form was submitted.

That is necessary, but incomplete.

Tool-using agents introduce a new layer of behavior between the prompt and the result. The model may have to decide whether to use search, database lookup, browser interaction, a calculator, a calendar API, or a custom action. This is a routing problem, and routing problems deserve routing tests.

A passing final answer can hide a broken tool decision, especially when a fallback or retry masks the original mistake.

A few examples:

  • The agent should query an orders API, but instead opens the browser and scrapes stale UI.
  • The agent should send a refund request before updating the ticket, but it updates the ticket first and then fails on the refund.
  • The agent should call search_products(query, locale, limit), but it passes the wrong locale and silently changes ranking.
  • The agent should fetch browser state before clicking a button, but it clicks based on a cached DOM snapshot and misses a modal.

These are not just LLM quality issues. They are integration issues, contract issues, and workflow design issues.

Define the failure modes you want to catch

Before writing tests, define the kinds of mistakes you want to observe. Good agent testing starts with a failure taxonomy.

1. Wrong tool selection

The agent chooses an incorrect tool for the task. Examples:

  • Uses browser automation instead of an internal API.
  • Calls a write action when read-only was required.
  • Uses a general search tool when a structured database query would be safer.

This often happens when tool descriptions are too similar, when names overlap, or when the prompt under-specifies the task.

2. Wrong order of tools

The agent calls the correct tools, but the sequence is invalid. Examples:

  • Tries to submit a form before authenticating.
  • Fetches an account before verifying the user identity.
  • Creates a record before checking whether it already exists.

Order failures matter because many workflows are stateful. A single out-of-order call can invalidate the rest of the run.

3. Wrong parameters

The agent routes correctly, but uses a bad input value. Examples:

  • Passes the wrong customer ID.
  • Uses a date in the wrong timezone.
  • Sends a limit of 1000 when the tool contract expects 100 max.
  • Converts a browser selector into a brittle text match.

Parameter failures are especially dangerous because they can look like tool success while still producing the wrong outcome.

4. Ambiguous fallback behavior

The agent makes a wrong or weak first choice, then recovers in a way that hides the issue. Example:

  • It calls a failing tool, retries with a different one, and produces the right result.
  • It answers from memory after a tool timeout.
  • It falls back from API to browser and completes the task, but with lower reliability.

If you only assert success, you may miss the degraded path.

Model the tool contract first

You cannot test tool selection well if tool definitions are vague. Each tool should have a clear contract:

  • Purpose, what it is for.
  • Preconditions, when it may be used.
  • Inputs, required and optional parameters.
  • Side effects, what it changes.
  • Failure modes, what errors it may raise.
  • Expected ordering constraints, if any.

For example, a search tool and a database lookup tool should not just differ in name. Their affordances should be distinct enough that the agent has a reason to choose one over the other.

A useful way to write tool contracts is to state them like testable rules:

  • Use get_customer_by_email when the input is a verified email and the record must be exact.
  • Use search_customers when the input may be partial or approximate.
  • Never call create_ticket before validate_account_status.
  • Never pass browser selectors that are based on dynamic text when a data attribute exists.

These rules become the backbone of your assertions.

Instrument the agent so you can inspect decision paths

If the agent is a black box, your tests will stay shallow. Instrument it so every run records:

  • The tools considered, if available.
  • The tool chosen.
  • Tool arguments.
  • Tool response status.
  • Retries and fallback attempts.
  • Final outcome.

At minimum, store an execution trace as structured JSON. For example:

{ “run_id”: “agent-run-1842”, “steps”: [ { “tool”: “browser_click”, “args”: { “selector”: “button:has-text(Submit)” }, “status”: “failed”, “error”: “element not found” }, { “tool”: “api_submit_order”, “args”: { “order_id”: “ord_123” }, “status”: “success” } ] }

That trace lets you assert not only that the task completed, but that it completed through an acceptable path.

Build a test matrix around decision pressure

Tool selection failures usually appear when the task is under pressure: ambiguity, missing context, timing issues, or conflicting incentives. Build test cases that create those conditions.

Ambiguous prompts

Use prompts that could legitimately map to more than one tool, then verify the preferred choice.

Examples:

  • “Check whether the customer paid.”
  • “Update the order and notify the user.”
  • “Find the latest version of the policy.”

These tasks force the agent to choose a route based on context, not keyword matching.

Similar tools

Create tools with overlapping semantics to see whether the agent can discriminate.

Examples:

  • get_invoice vs get_payment_status
  • browser_read_dom vs browser_click
  • search_catalog vs search_catalog_by_sku

If your agent repeatedly picks the wrong one, your tool naming or descriptions may be too close.

State-dependent tasks

Test sequences where the correct tool depends on previous steps.

Examples:

  • Authenticate, then fetch account data.
  • Validate input, then mutate state.
  • Fetch browser context, then click an element.

Parameter-sensitive tasks

Test inputs that are valid enough to pass schema checks, but wrong enough to cause business errors.

Examples:

  • Customer identifiers that resemble each other.
  • Time ranges that cross midnight or timezone boundaries.
  • Pagination limits near system thresholds.

Negative space

Test what happens when the preferred tool is unavailable.

  • Does the agent choose a safe fallback?
  • Does it ask for clarification?
  • Does it fabricate a result?

This is especially important in autonomous agent testing, where a tool outage can change the entire decision path.

Test wrong-tool selection with tool injection and assertions

A practical way to test tool routing errors is to make the right tool highly observable and the wrong tool detectable.

For example, suppose an agent can either call a structured API or scrape a browser page. Your test should assert that the API is used when it should be.

In a Playwright-based harness, you can watch for network requests while the agent runs:

import { test, expect } from '@playwright/test';

test('uses API instead of browser scraping for order lookup', async ({ page }) => {
  const apiCalls: string[] = [];
  page.on('request', req => {
    if (req.url().includes('/api/orders')) apiCalls.push(req.url());
  });

await page.goto(‘https://app.example.test’); await page.getByRole(‘button’, { name: ‘Ask Agent’ }).click(); await page.getByLabel(‘Prompt’).fill(‘Check order status for ORD-123’); await page.getByRole(‘button’, { name: ‘Run’ }).click();

await expect(page.getByText(‘Order shipped’)).toBeVisible(); expect(apiCalls.length).toBeGreaterThan(0); });

This is not just checking the answer. It is checking the route.

You can also make the browser path intentionally fragile during testing, so if the agent uses it incorrectly, the failure becomes obvious. For instance, hide a button behind a modal or change a non-essential label. If the agent is supposed to use the API, any browser scraping should stand out in logs or fail deterministically.

Test wrong-order failures with stateful guards

Wrong order is harder to catch because the final effect can be partially correct. The best approach is to make each prerequisite explicit and observable.

A simple pattern is to enforce tool preconditions in your test double. For example, if submit_refund should only occur after validate_payment, the mock can reject out-of-order calls.

from unittest.mock import Mock

state = {“validated”: False}

def validate_payment(order_id): state[“validated”] = True return {“ok”: True}

def submit_refund(order_id): if not state[“validated”]: raise Exception(“precondition failed: validate_payment first”) return {“refund_id”: “r_123”}

That pattern turns ordering into a deterministic test assertion. The agent either respects the workflow or it fails loudly.

For browser workflows, the same idea applies. If a form requires login, make the login state explicit in the test and assert that protected actions fail before authentication. You can use storage state in Playwright or a seeded session in your test backend.

If a workflow depends on invisible state, write tests that make that state visible.

Test parameter mistakes with contract-level assertions

Schema validation alone is not enough. A parameter can pass validation and still be semantically wrong.

For instance, this tool schema may accept both customer_id and email, but the task might require exact lookup by customer ID:

{ “customer_id”: “string”, “email”: “string”, “include_history”: “boolean” }

To test parameter correctness, assert the values passed into the tool, not just the final output. In API tests, use a mock or a test server that records requests:

import { expect, test } from '@playwright/test';

test('passes exact customer id into lookup tool', async ({ page }) => {
  const payloads: any[] = [];

await page.route(‘**/lookup-customer’, async route => { payloads.push(await route.request().postDataJSON()); await route.fulfill({ json: { name: ‘Ada Lovelace’ } }); });

await page.goto(‘https://app.example.test’); await page.getByLabel(‘Prompt’).fill(‘Find customer 42 and show account summary’); await page.getByRole(‘button’, { name: ‘Run’ }).click();

expect(payloads[0].customer_id).toBe(‘42’); expect(payloads[0].include_history).toBe(false); });

This kind of test catches:

  • Coercion mistakes, such as passing a display name instead of an ID.
  • Default-value mistakes, such as enabling expensive flags unnecessarily.
  • Formatting mistakes, such as ISO dates with the wrong timezone.

Browser workflows need different failure traps than API workflows

Tool-selection problems show up differently in browser automation than in direct API integrations.

Browser workflow traps

Browser-based agents can fail by choosing the right page but the wrong action, or the right action with the wrong locator. To test these, make the page intentionally rich in near-matches:

  • Two buttons with similar labels.
  • A disabled submit button until a field is valid.
  • A modal that appears only after a delay.
  • A table row with similar text but different hidden metadata.

Then assert the agent used the intended element and sequence.

For example, if the agent should click a data attribute instead of brittle text, prefer selectors like this:

typescript

await page.getByTestId('confirm-payment').click();

If your browser agent is choosing the wrong control, you need logs showing which locator it resolved and what alternative elements existed. Otherwise, your test will only show that “something failed.”

API workflow traps

API-based agents usually fail in cleaner, but more dangerous, ways. The request succeeds, but the payload is wrong. To catch this, use:

  • Request recording.
  • Response fixtures with edge-case payloads.
  • Contract assertions on headers, query parameters, and body fields.
  • Mock servers that return specific error codes for invalid order.

In API workflows, the main question is not whether the agent can call the endpoint. It is whether it can call the endpoint with the exact semantic intent you expect.

Design tests for fallback and recovery behavior

A robust agent should recover when a tool fails, but recovery should be controlled.

You want to know:

  • Which fallback was chosen?
  • Was the fallback acceptable for this task?
  • Did the agent explain the limitation?
  • Did it stop after a safe failure, or keep guessing?

Create tests that fail one tool on purpose. Then assert that the agent either:

  • Uses the defined fallback path, or
  • Asks for clarification, or
  • Stops with a clear error.

For example, if the order lookup API returns a 503, the agent may be allowed to use cached read-only data, but it should not fabricate a fresh status. That distinction belongs in tests.

Use behavioral assertions, not just output assertions

A good agent test suite checks multiple layers:

  1. Final user-visible result.
  2. Tools used.
  3. Tool order.
  4. Tool parameters.
  5. Error handling and fallback.

A single test can assert all five if the harness exposes the trace.

Example checklist for a critical workflow:

  • Did the agent choose create_ticket instead of update_ticket?
  • Did it call validate_customer before mutating state?
  • Did it pass the correct account ID?
  • Did it avoid browser interaction when the API was available?
  • Did it stop after a tool contract violation?

This approach is more durable than asserting fragile text responses.

Practical harness pattern for autonomous agent testing

A useful pattern is to split your tests into three layers:

1. Tool contract tests

Test each tool in isolation, like any other service boundary.

  • Input validation.
  • Error responses.
  • Output shape.
  • Side effects.

2. Decision-path tests

Test whether the agent chooses the expected tool route for a task.

  • Single-step routing.
  • Multi-step ordering.
  • Fallback handling.

3. End-to-end workflow tests

Test the complete browser or API flow with the agent in the loop.

  • Form submission.
  • Record creation.
  • Confirmation state.

This layered approach keeps failures easier to diagnose. If a workflow test fails, you can tell whether the problem is in the tool itself, the agent decision, or the integration.

Add CI gates that fail on suspicious routing

Tool-selection bugs should block merges when they affect critical paths. In continuous integration, make routing assertions part of the same pipeline as unit and integration tests.

A simple GitHub Actions job might run your agent tests on every pull request:

name: agent-tests

on: pull_request: push: branches: [main]

jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - run: npm ci - run: npm test – –grep “agent routing”

For teams with higher risk workflows, add a separate gate for tool-selection tests. That lets you distinguish ordinary UI breakage from agent routing regressions.

Common mistakes when testing tool selection

Only testing happy paths

If your tests always present the exact same phrasing, the agent may appear reliable until a user asks the question in a slightly different way.

Letting fallbacks hide mistakes

A fallback that completes the task may still indicate a bad primary route. Record the first tool choice, not just the final success.

Ignoring parameter semantics

A valid JSON object is not the same as a correct business operation.

Overfitting to prompts

If your tests depend on a single wording, they will not survive normal prompt variation. Prefer task intent over exact phrasing.

Using one tool per task too often

If a workflow never has to choose between alternatives, you are not really testing selection. You are testing execution only.

A concrete test plan you can adapt

If you are starting from scratch, use this sequence:

  1. Inventory every tool the agent can call.
  2. Mark which tools are mutually exclusive, order-sensitive, or parameter-sensitive.
  3. Write a contract for each tool.
  4. Add tracing so every decision path is observable.
  5. Create ambiguous prompts for routing tests.
  6. Add stateful tests for ordering.
  7. Add payload assertions for parameter correctness.
  8. Break one tool at a time and verify fallback behavior.
  9. Run the suite in CI and treat routing regressions as failures.

That gives you a practical baseline for catching tool routing errors before they reach production.

When a failure is a prompt problem, and when it is a system problem

Not every wrong tool choice means the model is “bad.” Sometimes the system design caused the error.

Ask these questions:

  • Are the tool names too similar?
  • Are the descriptions precise enough?
  • Does the agent have too many overlapping choices?
  • Are you missing a precondition check before tool invocation?
  • Does the runtime expose enough state for the agent to choose safely?

If the answer is yes to any of these, fix the system before you tune prompts.

Prompt edits can help, but prompt-only fixes tend to be fragile if the underlying tool contract is unclear.

Closing thought

The best way to test AI agent tool use is to assume the model will sometimes be confident and wrong. Build tests that make those mistakes visible at the level where they matter: tool choice, tool order, and tool parameters.

If you can observe the route, you can debug the route. If you can assert the route in CI, you can keep routing regressions out of production. That is the difference between an agent that merely seems smart and one that is safe enough to rely on.

For a broader background on the discipline, it helps to revisit the basics of software testing and test automation, then apply those principles to autonomous systems with a much stricter view of observability, state, and side effects.