Increasing the Overall Quality of your Test Automation
For automation to be effective it is essential to have good requirements and an understanding of the testing objectives as reducing human intervention alone in the testing process is not enough. The solution requires tighter integration between the three core elements: “People, Process and Technology”.
Sogeti's webinar last month was held by our Automation Subject Matter Expert Alistair Gerrard, a non-functional test specialist with over 22 years in IT, who kindly shared his views on increasing the quality of test automation. Below I have summarized the salient points and I hope you find it useful.
What constitutes a higher quality delivery of automation?
The challenge for many organisations is delivering high quality testing with minimal cost/ time whilst maintaining a high level of accuracy. Therefore, having processes right from the outset and knowing those processes work is crucial to delivering high quality automated testing; if we automate a bad process, the process may be faster but it will not be better. Furthermore, if the input into the process is of low quality from the beginning then the output of that process can also be expected to be low.
To ensure requirements that fuel development and testing are of a high quality, the test process needs to be extended to cover requirements, assuring the quality of requirements prior to the commencement of development and testing activities.
Once the quality of requirements is assured, the delivery of automated processes will be:
- Accurate - Manual testing can be mundane, error-prone and therefore become exasperating. Test automation alleviates testers' frustrations and allows the test execution without user interaction while guaranteeing repeatability and accuracy.
- Quick – Automation can repeat tedious tasks over and over again as many times as requested and can run tests far quicker than a manual tester.
- Cost effective – Automation can work unrelentingly 24/7, so is more efficient and can complete test scripts without having to take breaks.
The dependencies for successful automation
For automation to be effective it is essential to have good requirements and an understanding of the testing objectives as reducing human intervention alone in the testing process is not enough. The solution requires tighter integration between the three core elements: “People, Process and Technology”. People and technology need to work together to support performance objectives. We need mechanisms in place that report detail from each feedback loop within the testing process, as well as using automation to “crank the handle” between each feedback loop where possible.
Historically another dependency of good automation has been one of requiring strong technical test resources to automate. Whilst this still holds true, there are a number of factors that are changing the dynamic of this situation to increase productivity:
- A shift towards automation experts building frameworks which enable less specialist testers to create automated scripts
- New technology being used to create a more integrated SDLC
Another significant dependency is the concept of “Done”, which means being able to know when a task is complete. This means that when defining the requirements, the expected outcome is also defined, thus enabling the testing process to validate if the actual results meet the expected results. Again, “Done” can be thought of as a quality gate out of development and testing. Within the testing process it’s reasonable to assume those dependencies of testing remain, such as an environment to test in and test data to test against.
Addressing technical debt in your regression suites
Technical debt is one of the major challenges faced by most organisations, typically with risk-based approaches to testing meaning new functionality takes priority, often accompanied by targeted regression testing which is time-constrained. This reflects a traditional project management approach driven with a focus on cost and time, whereby the project team is accountable in isolation for the changes they implement.
As projects are also held accountable for quality, their focus should change to finding efficiency through doing it right first time, as opposed to finding efficiency through activities designed to shorten time-lines and reduce cost – at the expense of Quality according to the “Time, Cost and Quality” project triangle:
This leaves no simple answer as to how to address technical debt, and in truth each organisation will adopt a varied approach to fit the overall operational model and culture. Usually such scenarios lead to two proposed approaches: Automation is incrementally built up within the existing core project structures; or a separate automation project is set up to address technical debt.
Both approaches have different merits:
- Addressing technical debt within projects will help embed automation throughout the organisation, but the rate at which technical debt is reduced may be slow
- A centralised project will be able to reduce technical debt faster, but will be at risk to the impact of changes from projects
Working with extended teams to make continuous testing more effective
For continuous testing to be more effective it is broader than just test automation. It incorporates a number of disciplines from different areas of the organisation working towards a single goal of alignment. Technology is enabling the dismantling of silo walls and increasing collaboration between teams, supporting them to implement multi-discipline processes which bring the theory of good testing to life:
- Good requirements
- Test early
- Build it right the first time
With extended teams, there is an unwritten requirement of integration, and only through the combination of People, Process, and Technology can full scale collaboration become successful. Good people will adapt existing processes to create a team that can collaborate effectively, whilst technology will facilitate that collaboration – BUT all three components are required:
What makes automation difficult?
One of the key challenges of automation stems from it being a mix of both testing and development activities in the first instance. This combination means resources, especially good resources, are challenging to find, which increases their cost whilst constraining the amount of activity they can complete in project timescales.
In combination with the cost of tools and the late engagement within the software development lifecycle, it has been hard to realize an effective return on investment for automation in the past. The continued refinement of how automation tools are employed (record and playback, data-driven, frameworks), and the emergence of rival test tools has improved the return on the investment in automation to make it more feasible, whilst the software under test is becoming increasingly written (or engineered) to support automation.
All of these changes are leading to broader adoption of test automation throughout the testing industry, also driven by a need to have high quality software releases faster to market but without seeing an increase in cost.
The challenges in terms of test automation remain the same although the significant difference in what we are witnessing today is the seismic shift to considering automation in a broader sense, where people meld processes and technology together to work collaboratively towards a single goal.
What is the solution?
Organisations who frequently focus their application development efforts on technology improvement strategies and business process need to also focus on the people aspect of the change initiative. Organisations need to fuse together the People, Processes and Technology and ensure that all three work collaboratively towards a single goal. Ultimately, the solution is to have good people following good processes, and to use technology to integrate separate disciplines into a single, collaborative production line; this is what delivers higher quality automation.
If you’d like to replay the webinar, please click here.
- Sogeti UKMake an enquiry
0330 588 8200
Sogeti UKMake an enquiry
0330 588 8200