M365Con 2027 – free online conference · 11–22 Jan 2027 · 120 sessions→ m365con.net

Blog · Mirko Peters

Can AI write test cases from user stories? What to check first

By · · Updated

Can AI generate test cases from user stories? Studies show fast drafts with gaps. See what to check first: coverage, ambiguity, edge cases and invented tests.

Can AI generate test cases from user stories? Yes, and it does so in seconds, but the draft needs a careful check before it enters your test suite. This guide is for QA leads who want to try it without lowering quality. You will see what two published studies found, which problems keep returning, and a checklist for checking a draft. A worked example with made-up numbers shows how a review changes a batch of generated test cases.

Key points

  • AI can draft test cases from user stories in seconds to minutes, and a Thoughtworks experiment found high coverage of stated acceptance criteria, but quality depended on how clear the story was.
  • The same experiment measured ambiguity in about 27 percent of generated cases and about 6 percent incorrect cases, so a review is not optional.
  • Studies describe four recurring problems: irrelevant or invented cases, redundant cases, missing edge cases and inputs that are not fully specified.
  • Review each draft against the acceptance criteria, remove duplicates, add edge cases and negative paths, and rewrite any step that is vague.
  • Treat the model as an assistant that drafts, and keep a person responsible for what enters the regression suite.

Can AI generate test cases from user stories? What studies found

A Thoughtworks article, AI generated test cases from user stories: an experimental research study, describes a small experiment. Three QA engineers with three to five years of experience wrote manual test cases for nine user stories. Then generative AI tools, mainly GitHub Copilot with ChatGPT as a comparison, produced test cases for the same stories from refined prompts.

On average, the generated cases covered 98.67 percent of the acceptance criteria and were consistent in structure. The article reports about 4 percent duplication, about 6 percent incorrect test cases and an ambiguity score of about 27 percent. It describes generation as taking seconds to minutes instead of hours.

The weaknesses were just as clear. Quality depended heavily on the quality and specificity of the input. The tools had trouble with complex business logic, found few edge cases without explicit guidance, and did little non-functional testing unless asked. The authors call their hypothesis partially validated, say human oversight remains essential, and note that the sample of nine user stories is small.

A second paper, an arXiv study titled Automatic High-Level Test Case Generation using Large Language Models, looked at use cases rather than user stories, so treat it as a related result. The authors built a dataset of 1,067 use cases and 3,606 test cases from 168 projects. In a human evaluation, GPT-4o scored 3.91 for completeness and 3.98 for relevance, against 4.04 and 4.30 for human-written cases. The authors found the output satisfactory, but they state that a complete human evaluation of the automated test cases is necessary.

What the numbers do and do not tell you

Be careful with the percentages. Both studies state their limits. The Thoughtworks experiment used nine user stories, specific tools and versions, and a partly subjective assessment, and it did not measure productivity over the long term. It also reports that ChatGPT did better than GitHub Copilot where detailed prompt interpretation was needed, so results depend on the tool and the prompt.

In the arXiv paper, the 26 human evaluators agreed only moderately with each other, with a Cohen's Kappa of 0.42, and the participants were a convenience sample. The authors also say an automatic similarity score cannot reliably detect missing edge or negative cases. That is why they call for a human evaluation.

The high coverage figure also needs a second look. It measures the criteria that were written down. It cannot show the rules that nobody wrote into the story, and those hidden rules are often where defects live.

Four problems that keep returning

The arXiv paper names four recurring problems, and the Thoughtworks results point the same way:

  1. Irrelevant cases. The authors link these to AI hallucination: the model produces plausible-sounding but incorrect information. One example was a test for a feature the application did not support.
  2. Redundant cases. Some cases repeat another with a minor change.
  3. Missing edge cases. Models missed cases that human testers included, such as rejecting an unsupported file type.
  4. Weak input specification. Some generated cases left out input fields that the test needs.

None of these problems makes a draft useless. They tell you what to look for.

What to check first: a review checklist

Thoughtworks recommends reviewing output before production use by checking it against the acceptance criteria, flagging unclear logic and verifying edge case coverage. Turn that into a checklist for every batch:

  • Coverage. Does each acceptance criterion have at least one test case? List the criteria and tick them off.
  • Invented tests. Does each case match something in the story? Remove cases for features that do not exist.
  • Duplicates. Merge cases that check the same behavior with the same data.
  • Specific steps. Does each step name the element and the data? Rewrite vague wording such as "enter valid data".
  • Expected results. Can each expected result be seen on the screen or in a record?
  • Edge cases and negative paths. Add empty input, wrong format, limits and permission failures.
  • Non-functional needs. Add performance, accessibility and security checks that the story implies, because the model may skip them.

For a large batch, review a sample first. Read the first ten cases in full. If many of them fail the checklist, fix the story or the prompt before you read on, because the same flaw will repeat in the rest. This is a working rule, not a result from the studies.

AI test case generation fits into software testing as a drafting step that comes before test automation. A QA team can use AI to create test cases for each test scenario in a story, check test coverage against the acceptance criteria, and then decide which cases to turn into automated tests. The AI tool does the typing, and the QA team keeps the test design. Used this way, you can generate tests faster without lowering the quality of the suite.

This check is human-in-the-loop work. The tester does not retype the draft. The tester judges it, and that judgment is the part the model cannot supply.

Make the input easier for the AI

Thoughtworks reports that the prompt evolved from a basic request to one that gave the model a role and asked it to use Given, When, Then where appropriate. It also reports that coverage for complex scenarios needed significant prompt work. Three habits help:

  1. Write explicit acceptance criteria. If the story is vague, the cases will be vague. Fix the story first.
  2. Add context. Name the user role, the product area and any rules the model cannot guess.
  3. Ask for a structure. Use AI with a clear format: request a title, preconditions, steps, data and expected result for each case, and ask the AI to list its assumptions.

Keep the prompt free of secrets and customer data. Use invented example values.

Example: reviewing one batch

The numbers are made up.

A team gives ten user stories to an AI assistant. It returns 60 test cases in a few minutes. The QA lead checks them with the product owner.

  • 6 cases are duplicates, so the team removes them.
  • 5 cases test behavior the product does not have, so the team removes them too. That leaves 49.
  • 7 of the remaining 49 cases have vague steps or results, so the team rewrites them.
  • 9 edge cases and negative paths are missing, so the team adds them by hand.

The final suite has 58 test cases. The review took about three hours, where writing all 58 from scratch might have taken a full day. The team keeps the review notes, records which kinds of cases were rejected, and uses them to improve the prompt for the next sprint. The gain is real, but it comes from the review and not from the model alone.

Common mistakes

  • Accepting the batch unread. Duplicates and invented cases enter the suite and erode trust in it.
  • Judging success by the number of cases. More cases do not mean more coverage.
  • Feeding vague stories to the model. Poor input gives poor output.
  • Forgetting edge cases and non-functional checks. The studies show that models tend to miss them.
  • Putting private data into prompts. Use made-up values and follow your data policy.
  • Skipping the link to the acceptance criteria. Without it, you cannot show what is covered.

How software can help

A spreadsheet or your test management tool can hold the criteria, the generated cases and the review status side by side. Mark each case as accepted, rewritten or rejected, and keep the counts, so that after a few sprints you know your own rejection rate. A code review tool or a shared document helps testers and product owners review together. If you use an AI assistant, choose one that lets you control the prompt and see its full output. For browser-level cases, a plain-language end-to-end testing tool can then run the approved steps. Pilot on one team before you roll it out.

Next step

Take one finished user story, ask an AI tool to generate test cases, and check the draft against the checklist above. Count what you remove, rewrite and add, and use those numbers to decide how far to let AI generate test cases from user stories in your team.

Entities in this article

Generative AI AI hallucination Human in the loop End-to-end testing

Related articles

RSS