Skip to content

Agentic investigations

The QA Wolf agent takes the first pass at diagnosing failed tests.

Date
2026
Role
  • UX design
  • Interaction design
  • Prototyping

From investigating to reviewing

A failed test doesn't explain what went wrong. Is the product broken, or does the test need updating?

The agent takes the first pass. I helped design how people review its diagnosis, check the evidence, and correct it when it misses something.

UX research

I shadowed and interviewed QA engineers who regularly investigate test failures. I found that many still had to manually rerun failures to see if underperforming test environments were the cause.

I also audited the production run details page. These findings shaped my design proposal:

  • The diagnosis was buried behind performance stats. In my proposal, I moved investigation status and outcomes to the top so reviewers could see what the agent found first.
  • The workspace was built around code editing. I put the diagnosis and next action above the video, traces, and logs.
  • Tags competed with test names. I organized tests into folders and moved tags into filters.

Diagnosis lifecycle

When a test fails, the agent investigates whether the problem is in the product or the test itself. I designed a different review path for each.

A bug report notifies the customer’s team, so it waits for human confirmation. A test fix is reversible: the agent can apply it and verify a passing re-run, then a person reviews the fix before using it in future runs.

If the agent can’t produce a passing fix, it needs a person’s help.

Claims with receipts

The agent can reproduce a failure without knowing whether the behavior was intentional. I designed the review card to show both its evidence and the question it still needs a person to answer.

The agent has a diagnosis. It still needs your judgment.

Original run details UI with a proposed bug, three evidence receipts, and a Needs review banner asking whether the behavior change is intentional.

Dispute is a conversation

The reviewer may know something the agent doesn’t. I designed dispute around explaining that missing context, so correcting the agent gives it a starting point for investigating again.

You know the change is intentional. Tell the agent what it missed.

The original global chat beside the proposal, with an example message explaining that the behavior change is intentional.

A passing test still needs inspection

We let the agent apply and verify a reversible fix for the current run. But passing isn’t enough if the fix stops checking the behavior that matters. I put the code change beside the re-run evidence so reviewers can judge the fix before using it in future runs.

The agent applied a fix and verified it on the run’s branch.

Original Test fixed card with agent attribution, evidence claims, a passing re-run, and the Review fix action.

What guided my iterations

As I iterated, I kept asking: does the evidence make an automatic fix trustworthy? Does disputing feel like correcting a teammate? Is waiting for re-investigation tolerable?

I wanted people to understand the agent’s work well enough to decide what to trust and what to question.