When an AI test agent takes the wrong recovery path, the failure is usually not the first thing that went wrong. The real issue is often a missing piece of evidence: the agent saw a stale locator, a transient modal, a timing glitch, or an ambiguous page state, then chose a branch that looked plausible but was wrong. By the time the test fails, you need more than a stack trace. You need a forensic trail of what the agent observed, what it considered, what it discarded, and why it committed to a specific recovery move.

This article is a debugging checklist for teams building agentic QA workflows, especially when tests can retry, branch, repair locators, re-run steps, or self-heal. The goal is not to make every agent decision explainable in a philosophical sense. The goal is to collect enough structured evidence that a QA engineer, SRE, or test platform owner can reproduce the decision and determine whether the agent, the app, the environment, or the recovery policy was at fault.

The wrong recovery path is rarely a single bad choice. It is usually a sequence of reasonable choices made under incomplete observability.

Why wrong recovery paths are hard to debug

Traditional test failures are usually linear. A click did not land, a selector was missing, a request timed out, or an assertion failed. Agentic systems add a second layer. The agent can attempt alternative locators, repeat steps, dismiss dialogs, refresh the page, backtrack, or use a different route altogether. That flexibility is useful, but it also creates a new failure class:

  • The agent recovers from the wrong symptom.
  • The agent resolves the symptom in the wrong context.
  • The agent mutates state in a way that hides the original issue.
  • The agent succeeds locally, then fails later because the recovery changed the test trajectory.

For example, if a login page shows a cookie consent banner, an agent might interpret the blocked sign-in button as a locator issue and switch to a fallback selector. If the real problem is that the banner intercepts clicks, the fallback selector may still be correct, but the chosen recovery path wastes time and may create a misleading trail. Another common case is a page that rerenders after an async API call. A naive retry may click a button twice, causing duplicate state changes that make the rest of the flow unreliable.

This is why agentic recovery observability matters. You are not only debugging the failing step, you are debugging the decision process.

What to log at the moment the agent begins recovery

The most useful logs are the ones captured before the agent explores a different path. Once the agent starts branching, state changes quickly and the original context is easy to lose.

1) The triggering symptom

Log the exact event that caused recovery to start.

Capture:

  • Step name or action label
  • Timestamp with monotonic and wall-clock time if available
  • Failure type, for example timeout, locator miss, assertion mismatch, click intercepted
  • Error message or browser exception
  • Whether the failure came from the app, the browser, the runner, or the agent policy

Example categories matter because a timeout and a stale element reference can lead to very different recovery policies. A good recovery engine should distinguish between symptoms that imply transient instability and symptoms that imply a bad test design.

2) The original execution context

Log the state of the test right before recovery started.

Include:

  • Current test step index
  • Current branch or scenario path
  • URL and route
  • Active frame or iframe context
  • Active window or tab
  • Authentication state, if relevant
  • Any environment flags, feature flags, or tenant identifiers
  • Prior retries already attempted for this step

A recovery path can only be evaluated relative to the context in which it was chosen. If the agent switched tabs, navigated to a new route, or lost an iframe, the same recovery action can become invalid.

3) The last known good interaction

Record the last action that definitely succeeded, not just the action immediately preceding the failure.

Useful fields:

  • Action type, such as click, type, select, wait, navigate
  • Target locator or selector
  • Observed UI text or accessible name
  • DOM snapshot hash, if available
  • Network event or API response that correlated with success

This helps narrow whether the failure happened because of a specific action, a race between UI and network, or a broader environment issue. The farther the agent gets from a known good state, the more valuable this becomes.

What to log about the candidate recovery paths

If your agent evaluates more than one recovery option, log the full set of candidates, not just the winner. This is often the difference between a five-minute triage and a week of speculation.

4) The candidate list

For each recovery option, record:

  • Candidate ID or rank
  • Recovery type, such as retry click, alternate locator, dismiss overlay, refresh page, go back, re-run assertion, wait for state, restart flow
  • Triggering heuristic or rule
  • Confidence score or priority, if your agent uses one
  • Expected effect of the recovery

If the agent uses an LLM or policy model, capture the prompt inputs that influenced the decision, or a sanitized version of them. If it uses deterministic rules, capture the rule name and version.

5) The evidence used to rank candidates

This is one of the most important pieces of AI test debugging. The agent should not just record the chosen branch, it should record the observations that made it think that branch was viable.

Examples:

  • A button had similar text to a previous one
  • A modal overlay matched a known pattern
  • The element count changed between polls
  • The URL included a redirect marker
  • The page title became unstable
  • Accessibility tree labels suggested the target had moved

If the wrong path was chosen, you need to know whether the agent misread the evidence or whether the evidence itself was ambiguous.

6) The rejected alternatives

Most teams forget to log what the agent declined. This is a mistake. The rejected alternatives reveal whether the agent was overconfident, underinformed, or operating under a bad policy.

Capture for each rejected path:

  • Why it was rejected
  • Which signals disqualified it
  • Whether any signal was missing or stale
  • Whether the rejection was due to a policy guardrail or a model judgment

If the agent chose to refresh the page instead of waiting for a locator to appear, that may be a sensible choice. If it refused to dismiss a banner because the banner was not in the screenshot but was present in the DOM, that points to a visibility mismatch you can fix in observability.

What to log about the UI and DOM state

Recovery mistakes often happen because the agent is looking at the wrong representation of the page. Screenshots, DOM, accessibility tree, and network state can disagree.

7) Screenshot or viewport evidence

Log the visual state at the moment of failure and at the moment the recovery decision was made.

Useful details:

  • Screenshot timestamp
  • Viewport size and device profile
  • Scroll position
  • Whether the page was partially loaded, loading, or fully rendered
  • Overlay and modal presence

A screenshot is not always enough by itself, but it helps answer questions like, “Was the button actually visible?” and “Did a toast or modal cover the target?”

8) DOM and locator snapshots

Log the element that the agent expected, plus nearby candidate elements.

Capture:

  • Selector or locator strategy used
  • Inner text or accessible name
  • Stable attributes, such as data-testid, role, aria-label
  • Parent and sibling context for nearby ambiguity
  • Whether the element was detached, hidden, disabled, or covered

A lot of wrong recovery paths come from locator ambiguity. The agent sees multiple elements that look equally valid, chooses one, then the test drifts. If you log nearby candidates, you can quickly tell whether the selector was too broad or whether the app introduced an unexpected duplicate.

9) Accessibility tree state

If your agent uses accessibility data to reason about the page, log it separately from the DOM and screenshot.

Why this matters:

  • The accessible name may differ from visible text
  • Hidden labels can still influence target matching
  • Roles and relationships can reveal dialog state or navigation structure

For debugging recovery decisions, the accessibility tree is often the clearest sign of whether the app is in a modal, form, or error state.

What to log about timing and retries

Recovery path errors are frequently timing errors in disguise. The agent may be right about what it should do, but wrong about when it should do it.

10) Poll intervals and wait strategy

Record:

  • Wait type, explicit, implicit, fixed sleep, event-driven, custom polling
  • Poll interval and timeout
  • Number of polls performed
  • Whether the condition changed during polling
  • Whether the agent escalated from wait to alternate recovery

If the agent waits too little, it may branch prematurely. If it waits too long, it may mask a genuine regression. The log should show the exact threshold at which it gave up.

11) Retry count and retry reason

For every retry, log:

  • Retry number
  • Reason for retry
  • What changed since the previous attempt
  • Whether the retry reused the same locator or changed strategy
  • Any side effects already caused by earlier attempts

This matters because a retry is not harmless if the action is not idempotent. Clicking “Submit” twice can create duplicate orders, duplicate signups, or duplicate jobs. An agent that retries the wrong action without logging side effects can create misleading pass results.

12) State transitions between attempts

If the test state changed, record the transition explicitly.

Examples:

  • Logged out to logged in
  • Form pristine to form dirty
  • Modal open to modal dismissed
  • Navigation pending to navigation complete
  • API error to recovery dialog visible

These transitions are the bridge between “I tried something” and “the app changed.” They also help you decide whether to make the recovery policy stricter or to add a better wait condition.

What to log about external dependencies

Sometimes the recovery path was wrong because the page was not the issue. The real problem lived in an API, a queue, a feature flag, a CDN, or a flaky test environment.

13) Network requests and responses

Log network evidence near the failure point:

  • Request URL and method
  • Status code
  • Response timing
  • Correlation ID or trace ID
  • Key response fields that affect UI state
  • Retries at the HTTP layer

If the UI is waiting on an API that returns 202 or a delayed 200, the agent might choose a UI recovery when the right answer is to wait for backend completion. Network logs make that distinction much easier.

14) Browser and runner health

Capture environment factors that can bias recovery selection:

  • Browser version
  • Runner version
  • Headless versus headed mode
  • CPU or memory pressure, if available
  • Container or VM image version
  • WebDriver or Playwright protocol errors

Environment instability often causes the first failure, but the wrong recovery path compounds the problem. A stale browser session can make the agent chase phantom UI issues that are really session or runtime faults.

15) Feature flags and config

If the UI behavior depends on config, log the exact configuration state.

Include:

  • Feature flag values
  • Tenant or workspace settings
  • Locale and timezone
  • A/B test variant, if applicable
  • Mock versus live integration mode

A recovery path can look wrong when it is actually responding to a different product variant. Without config logs, you can spend hours debugging a test that was simply running against the wrong UI contract.

A practical logging checklist for agentic recovery

Use this as a minimum viable evidence set when an AI test agent branches or self-recovers.

Minimum fields to keep

  • Test run ID
  • Step ID
  • Recovery attempt ID
  • Error type and message
  • URL, frame, and window context
  • Screenshot or viewport snapshot
  • Locator, target text, and nearby candidates
  • Candidate recovery options and ranking rationale
  • Retry count and timeout values
  • Network request or trace correlation
  • Environment and feature-flag state
  • Final branch taken and final outcome

Fields to add when debugging complex failures

  • Accessibility tree snapshot
  • DOM diff from last successful state
  • Model prompt, policy version, or ruleset version
  • Rejected alternatives with rejection reasons
  • Action side effects already observed
  • Session or auth token lifecycle state
  • Correlated backend logs, if available

If you can only afford one extra thing beyond screenshots, log the candidate recovery set. That usually reveals whether the agent misread the page or simply had a bad policy.

How to structure logs so humans can review them

A flat stream of logs is not enough. Make recovery evidence reviewable.

Prefer a decision record model

Instead of logging only events, log a decision record for each recovery branch. A useful structure is:

{ “run_id”: “run_10492”, “step_id”: “checkout_submit_click”, “failure”: { “type”: “click_intercepted”, “message”: “Element covered by overlay” }, “context”: { “url”: “https://app.example.com/checkout”, “frame”: “main”, “viewport”: “1440x900” }, “candidates”: [ { “id”: “dismiss_overlay”, “confidence”: 0.91, “evidence”: [“overlay visible”, “close button in DOM”] }, { “id”: “retry_click_same_target”, “confidence”: 0.37, “evidence”: [“target stable”, “overlay may clear”] } ], “chosen”: “retry_click_same_target”, “reason”: “policy favored minimal intervention” }

This makes it obvious where the wrong decision happened and why. It also gives you a stable schema for querying failures across many runs.

Store artifacts alongside structured logs

Structured logs tell you what happened. Artifacts show you what the agent saw.

Keep links to:

  • Screenshot files
  • DOM dumps
  • Network traces
  • Video clips, when available
  • Browser console logs
  • Model or policy traces, sanitized as needed

A good practice is to make every recovery decision clickable from run history. That is where Endtest’s AI Test Creation Agent is relevant in a broader sense. Endtest uses an agentic AI workflow to create editable tests inside the platform, and that kind of run history plus reviewable artifacts is exactly what helps teams inspect what the agent did, then adjust the test rather than guess. For teams already exploring the documentation for its agentic test creation flow, the key idea is the same even beyond the product, keep the evidence close to the decision.

How to tell whether the agent or the policy is wrong

Not every bad recovery path means the agent was “wrong”. Sometimes the policy is too aggressive, too conservative, or optimized for the wrong goal.

Signs the agent misread the page

  • It selected a locator on the wrong element type
  • It treated a visible overlay as absent
  • It chose a recovery based on stale DOM state
  • It ignored a clear accessibility signal
  • It used a fallback even though the primary path was still valid

Signs the policy is the problem

  • The same bad choice appears across many runs
  • The agent prefers disruptive recovery over low-risk waiting
  • The policy rewards local continuation even when state integrity is compromised
  • The agent never escalates to human review on ambiguous states

Signs the app is the real problem

  • The UI exposes duplicate controls with identical labels
  • The page rerenders and detaches elements too often
  • The app does not provide stable test hooks or accessible roles
  • Backend latency creates unstable UI states

This distinction matters because the fix changes. A misread page may need better locators or better visual evidence. A bad policy may need a tighter action hierarchy. A flaky app may need product and engineering work, not a test patch.

A short example from a real debugging workflow

Imagine a checkout test where the agent clicks “Place order” and gets a click interception error. It then decides to retry the click on a fallback locator, but the page actually has a cookie banner blocking the button.

With weak logging, you only see:

  • click intercepted
  • retry failed
  • test failed

With good logging, you see:

  • screenshot shows cookie banner overlay
  • DOM indicates the button is present but obscured
  • accessibility tree exposes a close button for the banner
  • recovery candidates included dismiss overlay, scroll, retry click
  • policy selected retry click because it treated the target as unstable
  • second click hit the same overlay

Now the debugging decision is clear. The page needed a modal dismissal path, not a locator fallback. You can update the recovery policy, add a banner-specific rule, or improve the test’s guardrail to prefer overlay removal when a target is blocked.

Putting the checklist into CI and triage workflows

Logging is only useful if it lands where humans already work.

In CI

Make sure each failed run stores:

  • A run summary with the recovery chain
  • Direct links to artifacts
  • A diff between the final attempted path and the primary path
  • Tags for flaky environment, app regression, or policy issue

In triage

Give reviewers a simple question set:

  • What symptom triggered recovery?
  • What evidence supported the chosen path?
  • What alternatives were rejected?
  • Did the recovery change app state?
  • Was the failure likely app, test, environment, or policy related?

In retrospectives

Aggregate recurring patterns:

  • Same overlay causing repeated misclassification
  • Same locator family causing ambiguity
  • Same branch chosen under the same feature flag
  • Same environment constraint leading to retries

That is where agentic recovery observability becomes a real quality signal instead of a collection of pretty logs.

Final checklist

When an AI test agent selects the wrong recovery path, log the following at minimum:

  • The exact triggering failure
  • The full execution context before recovery
  • The last known good state
  • All candidate recovery options
  • The evidence behind each candidate
  • The reason the chosen path won
  • The rejected alternatives and why they were rejected
  • Screenshot, DOM, and accessibility snapshots
  • Retry counts, wait strategy, and timing thresholds
  • Network, browser, runner, and config state
  • The final branch taken and its side effects

If your team can reconstruct the decision, you can usually fix the root cause. If you cannot reconstruct the decision, you are debugging blind.

For teams building autonomous test creation and maintenance pipelines, this is also the difference between an agent that feels magical and an agent that is operationally trustworthy. The more your workflow resembles a reviewable history of decisions, the easier it becomes to scale agentic QA without losing control.