How to Instrument AI Test Agents With OpenTelemetry Spans, Structured Logs, and Replayable Artifacts

When an AI test agent fails, the obvious question is not just what broke, but why the agent chose the path it did. Traditional test logs often answer part of that question, yet they usually stop at browser actions, HTTP calls, or assertion failures. Agentic QA needs a richer trail: the model prompt that shaped the decision, the span that shows which branch was taken, the structured log event that explains the confidence or fallback, and the artifact that lets you replay the run later.

That is where the combination of OpenTelemetry spans, structured logs, and replayable artifacts becomes useful. If you instrument AI test agents with OpenTelemetry, you can trace autonomous test workflows end to end, correlate model calls with browser actions, and preserve enough evidence to debug failures without guessing.

This article focuses on a reproducible setup you can adapt in your own stack. The goal is not to create more telemetry for its own sake. The goal is to make agent behavior inspectable, searchable, and replayable.

What you should be able to answer after instrumentation

A good observability layer for agentic testing should let you answer these questions quickly:

• Which step in the test plan led to the failure?
• Did the agent retry because the locator was missing, the assertion was ambiguous, or the model output was malformed?
• Which prompt and tool inputs produced the action that changed state?
• Was the browser page in the expected state when the agent made its decision?
• Can we replay the same run against the same app build and see whether the failure is deterministic?

If you cannot reconstruct the agent’s decision path, you are debugging a black box with a screenshot attached.

The observability model for AI test agents

For agentic QA systems, I recommend thinking in three layers.

1) Spans for causal structure

Spans are the skeleton of the run. They should answer:

• What run is this?
• Which test scenario is being executed?
• Which plan step, model call, browser interaction, or assertion belongs to which parent decision?

OpenTelemetry’s trace model is a good fit because it already captures parent-child relationships and cross-process context propagation. The OpenTelemetry specification gives you the shape, and your backend can render the waterfall.

2) Structured logs for decision detail

Logs should carry the details that do not fit neatly into span names or attributes, such as:

• prompt version
• model name
• tool arguments
• retry reason
• parsed output shape
• validation errors
• warning conditions

Use JSON logs, not free-form strings. Free-form logs become difficult to query once you need to filter by run ID, scenario ID, tool name, or error type.

3) Replayable artifacts for reproducibility

Artifacts are the evidence bundle, usually including:

• the final prompt and all tool messages
• model responses, including rejected outputs
• browser console logs
• network HAR files where appropriate
• screenshots on each major decision point
• DOM snapshots or HTML captures
• the exact test plan or scenario definition
• environment metadata, such as app version, feature flags, and git SHA

A replayable artifact is not only for postmortems. It is also a review object for pull requests and a training signal for improving the agent later.

A practical span hierarchy for autonomous test workflows

A common mistake is to create one span per browser action and call it done. That gives you a thin activity log, but not a decision trace.

A better trace structure looks like this:

• test.run
  • agent.plan
    • agent.reason
    • llm.chat
    • agent.select_tool
  • browser.navigate
  • browser.wait_for_state
  • browser.click
  • assert.evaluate
  • artifact.capture

This structure gives you causal nesting. For example, if the agent decides to refresh the page before clicking, that decision should have a child span or sibling span that records the reason, not just the click itself.

Suggested span names and attributes

Use stable names. Do not encode variable data in the span name if you can avoid it.

A useful discipline is to avoid putting sensitive prompt text in span attributes unless you have already thought through privacy and retention. Use a hash or prompt version identifier if the full content is stored in artifacts.

Structured logs for AI agents, not just for browsers

Structured logs are where the agent explains itself. Here is a simple event shape that works well in practice:

{ “ts”: “2026-07-28T12:00:00.123Z”, “level”: “info”, “run_id”: “run_01HXYZ”, “scenario_id”: “checkout-happy-path”, “step_id”: “step_03”, “event”: “agent.decision”, “agent”: “checkout-agent”, “model”: “gpt-4.1-mini”, “tool”: “browser.click”, “reason”: “Proceeding after product page loaded and primary CTA became enabled”, “confidence”: “medium”, “locator”: “button[data-testid=’add-to-cart’]”, “attempt”: 1 }

This shape supports three important queries:

1. Show me every decision for a single run.
2. Show me all failures where the agent retried a locator more than twice.
3. Show me all runs where the model chose a fallback path after an assertion warning.

Fields worth standardizing early

If teams invent different names for the same concept, telemetry becomes painful to query. Standardize these fields early:

• run_id, a unique ID per execution
• scenario_id, a stable ID for the test intent
• step_id, a stable ID for the agent step or human-authored step
• trace_id, propagated by OpenTelemetry
• model, the model name or deployment alias
• tool, the exact tool or browser operation
• decision, the action type or branch taken
• reason, a short natural-language explanation
• confidence, a coarse bucket like low, medium, high
• artifact_ref, a pointer to stored evidence

A log line should be easy to grep, but more importantly, it should be easy to aggregate in your log backend.

A minimal instrumentation pattern in TypeScript

The following example shows a browser-driven agent that emits a span per decision and structured logs for each step. It is intentionally small so the shape is easy to copy.

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer(‘ai-test-agent’);

async function runScenario(scenarioId: string) { return tracer.startActiveSpan(‘test.run’, async (runSpan) => { runSpan.setAttribute(‘scenario.id’, scenarioId); runSpan.setAttribute(‘run.id’, crypto.randomUUID());

try {
  await tracer.startActiveSpan('agent.plan', async (planSpan) => {
    planSpan.setAttribute('plan.version', 'v3');
    logEvent('agent.plan.started', { scenarioId });
    planSpan.end();
  });

  await tracer.startActiveSpan('browser.navigate', async (navSpan) => {
    navSpan.setAttribute('url.path', '/checkout');
    // browser.goto('https://app.example.com/checkout')
    navSpan.end();
  });

  logEvent('agent.decision', {
    scenarioId,
    tool: 'browser.click',
    reason: 'Primary CTA enabled',
    confidence: 'high'
  });
} catch (err) {
  runSpan.recordException(err as Error);
  runSpan.setStatus({ code: 2, message: 'scenario failed' });
  throw err;
} finally {
  runSpan.end();
}   }); }

function logEvent(event: string, fields: Record<string, unknown>) { console.log(JSON.stringify({ ts: new Date().toISOString(), level: ‘info’, event, …fields })); }

The example omits exporter wiring, because the key idea is the structure. In a real system, you would attach a trace exporter and a JSON logger that emit to your collector and log backend.

Capture artifacts at decision boundaries, not only at failure

One of the most useful habits is to save artifacts at points where the agent changes state or makes a meaningful decision. If you only capture artifacts on failure, you often lose the context needed to understand why the failure happened.

Good capture points include:

• before and after navigation
• before a destructive action
• after a model-generated plan is accepted
• after each recovery attempt
• before assertion evaluation

What to store in a replay bundle

A replay bundle can be a directory or object storage prefix containing:

• trace.json, exported trace data or a pointer to it
• logs.jsonl, structured logs for the run
• prompt.txt, the exact prompt or prompt template expansion
• responses.jsonl, model outputs, including rejected outputs
• dom.html, page source or a DOM snapshot
• screenshot.png, a visual snapshot for each significant state
• network.har, when the flow involves backend behavior that matters
• metadata.json, env, git SHA, browser version, feature flags

A practical naming convention is to prefix everything with the run ID and scenario ID. That makes artifact collection and retention policies much easier to automate.

Replayability depends on environmental control

Replayable artifacts are only valuable if the environment is controlled enough to make a replay meaningful. That does not mean every run must be perfectly deterministic, because UI systems and LLMs are not deterministic by nature. It does mean you need a record of the variables that can change the outcome.

Capture at least:

• application build or deployment version
• commit SHA of the test agent
• browser version
• viewport size
• locale and time zone
• feature flags
• model version or endpoint alias
• seed values if your orchestration uses them

A replay without environment metadata is often just a second failure with better paperwork.

Failure modes you should design for

1) Span explosion

If you create a span for every keystroke or micro-action, tracing becomes noisy and expensive. Keep spans at the level of decisions, user-visible actions, and important waits.

2) Missing causal links

If logs and spans do not share a stable run_id or trace_id, you will waste time joining data manually. Propagate IDs explicitly into every log event.

3) Prompt drift

If prompts change without versioning, you lose the ability to compare runs. Treat prompt templates like code, version them, and emit the prompt version in both logs and spans.

4) Artifact bloat

Storing every screenshot, DOM snapshot, and HAR file forever is a storage problem. Use retention rules, compress where appropriate, and keep the highest-fidelity artifacts for failing or sampled runs.

5) Sensitive data leakage

Prompts and browser snapshots can include secrets, emails, or personal data. Redact at capture time when possible, and define retention and access controls before broad rollout.

A CI-friendly capture workflow

In continuous integration, the observability workflow should be predictable. A simple pattern is:

1. start a trace at the beginning of the job
2. export logs and artifacts after each test run
3. upload them as job artifacts
4. attach a trace link or run ID to the CI summary

A GitHub Actions example can help standardize the shape:

name: agent-tests
on: [push, pull_request]
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test
      - uses: actions/upload-artifact@v4
        with:
          name: agent-test-artifacts
          path: artifacts/

The important part is not the CI platform. The important part is that every run leaves behind the same evidence bundle, with the trace ID embedded in filenames or metadata.

How to evaluate whether the instrumentation is good enough

A useful evaluation guide is to pick one failing run and ask whether an engineer can reconstruct the sequence without reading source code.

You want to see whether the following are obvious:

• the initial scenario intent
• the plan produced by the agent
• the browser state before the failure
• the exact tool call that changed state
• the final assertion or guard that failed
• the artifacts needed to replay the issue

If the answer is no, the instrumentation is still too shallow.

Practical tradeoffs

There is always a balance between observability depth and operational overhead.

• More spans improve causal visibility, but too many spans make traces harder to read.
• More logs improve debugging, but too many verbose events make noise and storage costs grow.
• More artifacts improve reproducibility, but more captures increase runtime and retention burden.

The right balance is usually to instrument around decisions, retries, state changes, and assertions. That gives you enough context to debug most failures without drowning the system in telemetry.

A field-tested mental model for teams

If your team is building autonomous test workflows, treat observability as part of the test contract. A passing test that leaves no useful trace is only partially done. The test should produce:

• a trace that shows what happened
• logs that explain why the agent acted
• artifacts that let someone replay the run later

This is especially important when the agent is allowed to self-correct. Self-correction is useful, but it can also hide instability unless every branch is observable.

Final checklist for implementation

Before you roll this out broadly, verify these points:

• Every run gets a unique run_id
• run_id and trace_id appear in spans and logs
• Model calls, browser actions, and assertions are separated into meaningful spans
• Structured logs use stable keys and JSON output
• Prompt versioning is explicit
• Replay bundles include prompts, responses, DOM state, screenshots, and environment metadata
• Redaction and retention rules are defined
• CI artifacts are linked back to the trace

Closing thought

Instrumenting AI test agents is less about adding a telemetry stack and more about making agent behavior legible. OpenTelemetry gives you the trace backbone, structured logs for AI agents give you decision detail, and replayable artifacts give you the evidence to reproduce what happened. Put those three together, and autonomous test workflows become much easier to trust, debug, and evolve.

For background on the surrounding disciplines, see software testing, test automation, and continuous integration.