Metrics – Putting Suppliers to the Test

As public services are increasingly digital by default, shorter production lifecycles and the demand for quality and continuous testing put increasing pressure on testing organisations to create a set of metrics that enable better decision-making. So how do you maximise test metrics to get the best result for clients and end-users?

~ Written By Gary Moore

Why are test metrics so important?

The right metrics enable risk assessment, earlier bug detection, quality output and an insight into the effectiveness of the current test strategy, while holding developers, testers and 3rd party vendors accountable in a transparent way. While the Cabinet Office strategy of working with smaller suppliers helps to bring about innovation, and cost savings, this agility also creates an IT environment where change is constant. Effective test management and having a strong set of test metrics is more critical than ever in these multi-supplier environments where more risk is introduced into the IT delivery process. Comprehensive test metrics give granularity at project and service level, they creates a safety net and provide core values to return to when innovation or problems cause you to veer away from the bigger picture.

What should we measure and who decides?

At Sogeti we use our Live Dashboard to gauge the performance of the Development team and other stakeholders who affect the test process. KPI-aligned metrics also determine our own performance as the testing services provider. This last measure directly impacts our revenue, encouraging quality output and allowing us to offer clients the fairest fees. At Sogeti we use a comprehensive set of around 20-25 metrics which are broken down into KPIs, Service Measures, and Quality Measures. Examples of KPI-aligned metrics include defect leakage into production and testing the percentage of defects raised in error. Service and Quality metrics include test design and execution productivity, defect leakage into integration or UAT, defect turnaround time and reoccurrence respectively. The set of metrics we use are very similar in the Private and Public sectors. Metrics should be determined by the CIO, Chief Transformation Officer and other C level executives, the Vendor management groups, the test services provider and whoever owns the QA function on the client’s behalf.

How do you get the Project team to buy into the metrics and feedback in a productive way?

To get the client on board it should simply be a case of explaining the cost, time, quality, transparency and accountability benefits we’ve discussed. To engage those team members whose work is showcased by the metrics, it’s a case of reminding them that the purpose is not to name and shame, but rather to continuously develop their expertise, innovate and provide a better service. As an employee of a top tier test provider, testers should expect their performance to be measured and see it as an opportunity for growth and to collect evidence to prove the need for changes that will actually make their jobs easier. With Sogeti’s Live Dashboard, 3rd party vendors have no reason to query the process as we are putting ourselves under the microscope as well.

This year’s World Quality Report shows that Government is focused on improving engagement and Customer Experience (CX); how does this impact test metrics?

Currently the emphasis is on functional defects, usability and accessibility from the internal client’s perspective. Now that CX is a major driver, the test metrics and analysis will be more focused on the true end user; the individuals and businesses that interact with our clients. As customer expectations rise, the responsibility for quality and reputation management grows. To meet this, testers must take a walk in the shoes of the end user and ensure test metrics are dynamic and adapting to reflect the customer journey, feedback and changing requirements.

This article has just been published on Brite Innovation here: http://www.nridigital.com/brite-magazine.html?wv=s%2FBrite%2520Innovation%2520Review%2Fff474f22-d1f2-5183-87bc-c251d0ea2751%2FBIR1603%2Fsogeti.html

todo todo
contact us
  • Sogeti UK
    Sogeti UK
    Make an enquiry
    0330 588 8200