Automated testing approaches and return on investment for AQA on medium and large-sized projects

On the testing processes of medium and large-sized projects, there is a lot of repetitive work for regression tests on each iteration of the development process. To reduce the expenses for QA processes, as a rule, companies use testing automation.

To demonstrate the automation QA process, let’s take a simple online shop checkout as an example. It consists of three steps: 

1. Open ‘Product’ page and click button ‘Add product to a cart’
2. Open ‘Cart’ page and click button ‘Buy’
3. Fill out billing & delivery info and click button ‘Submit’

First of all, we need to test these three steps manually, on the implementation stage, to verify that all functional and non-functional requirements are met and the system works as expected. After that, we start automating the functionality for all further regression tests. In that way, we are sure that our vital functionality is working properly after each update. 

To cover most of the possible issues checking, we use 3 different types of automation testing:

• Image comparing tool: let’s assume that the ‘Product’ page has a lot of web elements with different images (product photos, similar products photos, logos, etc). To verify that all of them are present on the expected place, we make a screenshot of the web page and compare it pixel by pixel after all further deployments like in the reference article;
• Functional tests are needed to ensure that all client-side logic is working properly: buttons of the critical path are clickable, all expected redirects between pages happen and all the necessary info is displayed on the page. This test opens a webpage in a real browser(s) and emulates the actions of the real user. It performs all actions from three steps above and checks if the info about the selected product is displayed on the pages.
• API testing: now we have to check integrations between client and server on the business layer. Here we use API calls testing to verify that sending the request on the third step has expected responses and the server got the info about the payment and returned expected data. 

One of the main differences between manual and automated tests is the time, when each kind of testing might be applied. The main purpose of the automated tests is to make sure that the exact requirement is satisfied. The machine can’t perform exploratory testing while checking a specific test case. For example, it won’t determine broken markup issues or unexpected color of the element while checking ‘add to cart’ functionality. Thus we can’t use automated tests for acceptance testing of the new features because they can’t cover the use cases widely. In other words, manual testing can’t be fully substituted by automation. All acceptance and exploratory testing should be performed manually, and only after that, positive use cases should be automated.

On the other hand, manual QA engineers can’t get the same speed of testing and the same low error rate as automated frameworks, so repetitive work on regression testing should be automated. In that way, we reduce expenses on testing and get much more stable test case execution.

We can’t deny one kind of test to the advantage of another. The expected ratio between manual and automated tests depends strictly on a ratio between existing and new functionality, which will add for each sprint.

Here we can see the scheme of auto testing usage in our company

In the examples below, we can see how to define if the exact implementation of automation QA processes is financially profitable. We can use ROI for these purposes.

Return on investment (ROI) is a financial ratio that illustrates business profit or loss margins.

For calculating automated tests usage ROI we should use the next formula:

AQA ROI  = (income - investments) / investments * 100%

As an income, we can consider the time we save on auto test runs. For this purpose we use next formula:

Saved T = (manual testing T - automation testing T) * N of the test runs per specific period

For example, for testing the use case from the example above, a manual tester will spend 12 minutes. At the same time, the automated test may deal with it in 7 minutes. During one sprint, we have to test this scenario 9 times: regression testing for 3 different features deployments, using 3 different browsers. Profit for this particular scenario equals 27 minutes. After that, we multiply the result by the amount of the expected test scenarios, for example, 30 scenarios, and get 810 minutes of the economy. But for short-sized projects, automation may not bring valuable time- and cost consumption reduction.

As investments, we can consider next time amounts:

We have two types of the investments: fixed and variable 

Fixed investments:

• Licence for products and tools costs
• Education for a specialists
• Time for the framework preparations
• Time for the initial infrastructure preparations

Variable investments:

• Project workflow investigation (depends on the size of the system)
• Tests creation time (same as a point above)
• Tests runs time (might be reduced by CI/CD test implementation)
• Tests supporting time (depends on test stability and a ratio of GUI / API tests)
• Framework supporting time
• Infrastructure supporting time

Here are the three main points, which affects on commercial wellness of the IT product:

1. Walkthrough (the sales) - can’t be affected by auto tests
2. Operating expenses - might be higher in automation case, due to higher work cost then manual QA
3. Time to market - in case of proper automation implementation might be reduced on 50% or even more (tests implementation to CI/CD processes, intime tests maintenance, immediate  test results delivery)

Considering the info above, we see that ‘time to market’ reduction is the main aim for automation tests, which can override higher operating expenses and make the AQA process profitable. So how can we reduce it?

First of all, we need:

1. Use those tests, which are easiest to develop and maintain, as well as which are covering functionality with maximum priority. In this way, we will have a robust positive test suite and robust vital functionality of the site at the same time 
2. Adding test execution and results reporting to CI/CD, immediate test results delivery will be organized with it  
3. Combining API, functional and screenshot based tests in a desirable ratio for each particular project, which may differ depending on the project specifications
4. Tests should be maintained in a parallel with the product development process: project requirements and GUI might be dynamic during agile SDLC 
5. Test execution optimization: test workflow should be planned carefully, to schedule maximum efficient usage of the system work time
6. Using data management tools and approaches to deliver a larger time difference between manual and automated tests for getting more saved testing time

In conclusion, considering ‘QA vs AQA timeline graphic’ we see that automated testing is more expensive in the initial stages of the project, but much more efficient during long term development and consulting. Also, we have to mention that there might be cases when manual testing could not cover part of the functionality due to a lack of the budget.