Automated regression testing for web applications: what to automate and what to keep human

A form took two days to regression test. The hard part wasn’t the clicking. A multi-page quiz. More than ten questions. Different behavior depending on the account and user role. A calculated result that had to be saved correctly. A full regression pass took the QA team about two days. That’s how one of AlwaysQA’s founders remembers testing a feature at a former workplace. Each individual path looked manageable. The work grew when the team had to repeat it across accounts, roles, answer combinations, and expected results. Other development work waited while QA completed the checks. The obvious response is: “Just automate it.” But automate what, exactly? Clicking through the pages was only part of the job. Someone still had to define the account state, choose the right answers, know the expected calculation, and verify that the application saved the result. The hard part wasn’t teaching a browser to click Next. It was maintaining a trustworthy definition of what should happen. That’s where I’d start a regression automation project-not with the tool. First, separate regression testing from checking a fix Suppose you fix a bug that calculates the wrong result for one user role. Checking that the calculation now works is confirmation testing: did the fix resolve the defect? Checking that other roles still receive the correct results is regression testing: did the change break behavior that previously worked? The ISTQB Foundation Level syllabus makes this distinction. Regression effects can appear in the changed component, elsewhere in the application, or in connected systems. Automation changes how those checks run. It doesn’t change their purpose. A regression check can be a unit test, an API test, a browser test, or a manual check. You don’t need to drive a browser for every regression risk. Don’t turn the entire manual checklist into browser tests In the quiz example, a QA engineer had to sign in with a prepared account, complete the role-specific form, consult a separate source of truth, exercise valid and invalid paths, and verify the calculated and saved result. Then repeat the process for the next scenario. The team didn’t have automation for this workflow. A code-based browser suite appeared expensive to create and maintain because it would need to represent all those variations. That doesn’t mean Playwright or Cypress couldn’t handle it. It means that “write an end-to-end test” wasn’t a complete solution. Before choosing a tool, I’d split the problem into three questions: | Question | Where I’d start | |---|---| | Does this answer set produce the correct calculation? | Unit tests | | Are permissions enforced and results stored correctly? | API or integration tests | | Can a user with this role complete the quiz and view the expected saved result? | Browser end-to-end tests | These checks overlap intentionally, but they prove different things. A unit test can exercise calculation edge cases without signing in and answering ten questions every time. An API test can check authorization and persistence without rendering the form. A browser test can check whether the pieces work together from the user’s perspective. Cypress’s testing guidance describes these different scopes and the additional setup, infrastructure, and maintenance that end-to-end testing can involve. Use the browser to prove the journey. Don’t make it carry every internal rule. Your first automated workflow should be bounded, not impressive The most time-consuming feature isn’t automatically the best first automation project. For the quiz, I wouldn’t begin with every role and answer combination. I’d choose one important role, one prepared account, and one defined answer path. A useful first candidate has three properties: - It matters. A failure would block a meaningful user task or create release risk. - It repeats. QA already performs substantially the same check across releases. - It can be specified. The starting state and expected result are clear. That last property is easy to underestimate. “Check that the quiz works” is not a test specification. “Using a prepared account with role A, submit answer set 01 and verify that the expected result is displayed and available when the saved submission is reopened” is much closer. The first asks a tool to discover the requirements and judge the implementation simultaneously. The second gives it a defined question to answer. Write the test contract before the test script For a first browser check, I’d write something like this: Scenario Role A completes answer path 01. Starting state Staging environment. Dedicated test account with role A. Known quiz version. No existing submission that would change this path. Test data A defined answer set. An independently established expected result. Actions Sign in. Open the quiz. Complete the defined answer path. Submit. Reopen the saved submission. Pass conditions The displayed result matches the expected result. The reopened submission shows the expected saved result. Failure evidence Account role and scenario identifier. Answers submitted. Expected result versus observed result. The step where behavior diverged. Relevant screenshots, trace, or logs where available. This is a test-design example, not a tool-specific configuration format. Notice what’s missing: selectors, browser APIs, and prompts. Those come later. First, the team needs to agree on what the check must prove. Also, keep the expected result independent of the production logic being tested. If your test calculates its expectation by calling the same broken formula, both sides can agree while the behavior is wrong. For the quiz, the existing source of truth is therefore part of the automation design-not an annoying detail to resolve after the browser script is finished. Test data is part of the test A well-written script won’t rescue an uncontrolled starting state. Suppose the first run passes, but the second fails because the account has already submitted the quiz. Or another test changes its role. Or someone edits the answer data while the run is in progress. You now have a result to investigate, but it may say more about the test setup than the application. Playwright’s best practices recommend isolated tests with their own state to improve reproducibility and prevent failures from cascading between tests. For a state-changing workflow, answer this before scheduling it: What makes the next run start from the same meaningful conditions as the previous one? That could mean resetting a dedicated account, creating fresh test data, or using a supported cleanup process. The implementation depends on the application. The requirement doesn’t: the starting state must be deliberate. And don’t confuse an interface signal with every underlying guarantee. A success toast is evidence that a message appeared. Reopening a saved submission checks more of the user-visible behavior. Backend persistence rules still deserve their own lower-level checks. AI can change the execution. It doesn’t remove the assertions. Once you’ve justified a browser-level check, you can choose how to execute it. With code-driven automation, the team specifies actions and assertions explicitly. That can be a good fit when you want precise control over the path and already have the engineering capacity to maintain it. With AI-guided browser execution, you can describe the user’s goal and evaluate an alternative to maintaining fixed interaction steps. But the underlying contract remains: Start from this state, attempt this workflow, and verify this observable outcome. AI doesn’t remove the need for test data, expected results, safe boundaries, or useful evidence. “Use AI to test the quiz” is no more complete than “use Playwright to test the quiz.” Neither approach should get to invent the correct result because the page looks plausible. Human exploration belongs alongside both approaches, especially when the behavior is new, requirements are ambiguous, or someone needs to notice a problem that wasn’t anticipated in the test. An inconclusive run isn’t the same as a product failure Before automating a workflow, decide how you’ll interpret the result. There are at least three useful distinctions: Passed: the check reached the specified outcome and collected sufficient evidence. Failed: the observed behavior contradicted the expected behavior. Inconclusive: the run couldn’t establish a reliable verdict. For example, an incorrect quiz result is evidence of a failed check. An unavailable test fixture that prevents the scenario from starting is not, by itself, evidence that the quiz calculation is broken. Both need attention, but they need different investigations. A report should make that distinction visible. Otherwise, QA ends up replaying the entire workflow just to determine whether the application failed, the setup failed, or the runner got stuck. The same principle applies to passing results. A green status is only useful when you understand what was actually verified. One passing path is not a certificate that the application is regression-free. Where AlwaysQA fits Disclosure: I’m building AlwaysQA, an AI-guided browser QA product. Its test-case model follows the same basic structure: a starting URL, plain-language instructions, and an observable success condition. A run returns Passed, Failed, or Needs Attention, with evidence that can include a summary, observations, an action timeline, and a temporary replay. Needs Attention keeps an inconclusive run separate from an observed failure. Checks can run on demand or on daily or weekly schedules. The workflow is described in How AlwaysQA works. That makes it one option for the browser-execution part of this approach. It doesn’t replace the expected-result model, lower-level tests, or human judgment. And the two-day quiz example is a recollection of a manual testing problem-not a before-and-after result from using AlwaysQA. Run a one-workflow pilot before expanding You don’t need a la