M365Con 2027 – free online conference · 11–22 Jan 2027 · 120 sessions→ m365con.net

Blog · Mirko Peters

What is natural-language test automation? A plain explanation

Disclosure: Cognillo sponsors M365.FM. This page describes the company factually; paid partnerships and links are labelled as advertising.

By · · Updated

Natural language test automation lets testers write steps in plain English while AI runs them in a browser. See how it works, its limits and how to judge it.

Natural-language test automation means you write a test in everyday English, and an AI model carries it out in a real browser. Instead of a script full of selectors, you write "Click the Log In button" or "Check that the order total shows 49.00". This explanation is for QA leads and testers who hear the term and want to know what it is, how it works, and where it stops. You will see the idea step by step, the limits that tool makers state themselves, and a simple way to judge whether it fits your team.

Key points

  • Natural-language test automation means you describe a test step in plain English, and an AI model turns it into browser actions when the test runs.
  • It replaces hand-written selectors and waits with instructions such as click the Log In button, so testers who do not code can write and read tests.
  • Vendors document limits: broad or complex instructions fail more often, and single-step instructions are the most reliable.
  • AI tools built for generic web tasks can invent results without inspecting the real app, so every generated step needs review and a clear pass or fail rule.
  • Treat it as a way to write end-to-end tests faster, not as a replacement for test design, test data or sound environments.

What natural-language test automation means

A classic automated test of the user interface is code. A developer or automation engineer writes steps and tells the tool how to find each element on the page, for example by its ID or a CSS selector. The test runs the same steps every time. When the page changes, the selector can stop matching, and the test fails.

With this approach, the tester writes the intent instead. The Cypress team describes its cy.prompt command as a way to describe tests in everyday language, written as text strings inside the test file. An article on Cypress's blog says the generated steps can be inspected and debugged, and then ejected into static test code that you can save.

The underlying technology is generative AI: a model reads the text and the page, and decides what to do. In most tools it is a large language model, an AI model trained on large amounts of text. Some vendors call the software that does this an AI agent, because it takes actions rather than only answering questions. The goal is the same in each case. People who know the product can write and read tests without learning a programming language, so the approach is sometimes sold as codeless testing.

For software testing teams, this changes test creation, not the testing process. Test design, test data and test execution stay the same, and someone still decides which test case to cover.

In traditional test automation, engineers write and maintain a test script for every flow. AI tools use natural language processing to read each sentence instead. Some vendors go further and call their tools agentic. One vendor guide from Autosana, which is promotional, defines the difference: an agentic tool interprets a plain-English goal, explores the live page, acts and adjusts mid-run, and codeless is not the same thing as agentic. The same guide warns that many products claim AI-powered testing because they added autocomplete to a script editor. Treat these labels with care, and keep a pass or fail rule that you control.

How a written-instruction test runs

The details differ by tool, but the flow is similar. BrowserStack documents its beta "NL Test Automation Agent" for Selenium, and its page gives a clear picture:

  1. You write a short instruction. Each command is one English sentence, not a selector or code. The page advises starting with an action verb, such as "Click the Submit button".
  2. The AI interprets it at run time. Automation steps are generated while the test runs, so no selectors or waits are written by hand.
  3. The tool performs the action in the browser. The page lists command types such as click, type, search, hover and scroll.
  4. The tool reports the result. Successful runs show success logs, and failed runs show error logs with a reason.

Compare this with the approach Playwright recommends in its best practices: test what users see and do, and prefer role, text and test ID locators over XPath or CSS selectors that break when the page changes. This approach pushes the same idea further. The tester names what a user would see, such as "the Log In button", and the tool finds it.

Where natural-language testing helps

The approach fits some jobs better than others.

  • Smoke checks and common user paths. Log in, search, add to basket, check out. These are the end-to-end testing flows that matter most, and they are easy to describe in a sentence.
  • Teams where testers do not code. A tester who knows the business process can write and read the test.
  • Test generation. Some tools can generate a test from a short description. A person still reviews the test design and the coverage.

It helps less where the check depends on exact numbers, complex data setup, or timing that needs fine control, as in performance testing. A unit test of a calculation does not need a browser or an AI model, and exploratory testing still needs a curious person.

The limits that vendors admit

Honest tool makers state the limits. Use them as a checklist when you evaluate any product.

  • Complex instructions fail more often. BrowserStack's page says complex or overly broad objectives can fail with an UNSUPPORTED_OBJECTIVE error, that reliability drops as commands get more complex, and that single-step instructions are the most accurate. Split a long task into short steps.
  • Results can vary. An AI model interprets each instruction at run time, so a vague sentence may lead to different actions on different runs. Keep each step short and specific.
  • The feature may be in beta. The same page labels the feature beta and says supported environments may change.
  • Invented results are a risk. The Cypress article warns that many AI browser tools are built for generic web tasks, not testing. They can produce hallucinated output without inspecting the actual app, and they often lack support for debugging, assertions and the test lifecycle.

Keep tests short, review what the AI generated, and demand a clear pass or fail rule for every step.

Failures, maintenance and CI

Maintenance is where plain-language tests are meant to pay off, so ask how each tool handles change. The Cypress guide to AI test generation shows one design. It caches the generated code, and the cache is shared across machines and environments, including CI, so re-running a test does not call the AI again unless the prompt or the page invalidates the cache. When a selector or its text changes between runs, it regenerates code for that step and marks the step as self-healed in the logs. It warns that healing on every run adds AI calls and slows execution, and advises targeting elements by stable visible text. It also lets you view the generated code, with comments that map each command to its prompt step, and save it in place of the prompt.

The Autosana guide suggests a two-week proof of concept on real user flows, including ones that have broken before, with a review of screenshots and session replays to audit what the agent did. It gives no measured failure rates, so run a pilot of your own.

Can it replace QA engineers? Not on this evidence: people still plan what to test, prepare data and decide whether a result is right.

Questions to ask before you pick a tool

Use these questions in a vendor call or a pilot. They work for any product.

  1. What exactly does the model see? Does it inspect the real page, or does it guess from the text of your instruction?
  2. Can I read and export what it did? You want a log of each generated step, and ideally a fixed script you can keep in version control.
  3. How does it decide pass or fail? Look for explicit checks that you write, not only the model's own judgment.
  4. Where does the test run, and who can see the data? Test accounts and screenshots can contain real information.
  5. What does a failure report contain? Screenshots, logs and the failing step save hours of debugging.
  6. Does it fit our current test suite? Check your automation framework, your pipeline and your manual test process.

Example: one login test, three ways

The numbers are made up.

A team compares three ways to write a login test.

  1. Code with selectors. An engineer writes about 12 lines. It takes 30 minutes. When a developer renames the button's ID, the test fails and needs a fix.
  2. Natural language. A tester writes four lines: open the login page, type the user name, type the password, click "Log In" and check that the dashboard title appears. It takes 5 minutes. The tester reads the generated steps and confirms they match, and uses AI only to run them.
  3. Natural language, then export. The team keeps the plain-English version for exploration, and exports a fixed script for the nightly regression run, as the Cypress article describes for its own tool.

Over a month, two failures come from real bugs, and one comes from a vague step: "click the button" matched the wrong button. They rewrite it as "click the Log In button". Precision in the sentence matters as much as precision in a selector.

Common mistakes

  • Writing long, vague instructions. Use one intent per step, and name the element.
  • Trusting AI-generated steps without reading them. Read every step at least once, and keep the log.
  • Skipping the expected result. A test needs a clear check, such as a title or a total. "Make sure it works" is not a check.
  • Using it for everything. Unit and integration checks belong in code, close to the developers.
  • Adopting a tool without a pilot. Run ten real tests first and compare failure causes.

How software can help

Several kinds of AI testing tools exist. Some add plain-English commands to a code-based framework, such as the Cypress and BrowserStack features above. Others are separate platforms for the whole QA team. Open-source frameworks such as Playwright still work well for teams that prefer code.

Genta by Cognillo is one example of a separate platform. According to the vendor, testers describe in plain language what an application should do, and Genta runs the test in a browser on a cloud virtual machine and reports pass or fail, with screenshots and logs. The vendor says it is aimed at QA leads, SDETs and automation engineers. These are the vendor's statements. Compare it with other options in a pilot before you decide.

Disclosure: Cognillo sponsors M365.FM.

Next step

Pick one stable user path, such as login and search, and write it as five plain-English steps. Run it in a natural language test automation tool and in your current test framework, then compare effort, failures and the quality of the logs.

Entities in this article

End-to-end testing Generative AI AI agent Genta Automation

Related articles

RSS