Using coverage effectiveness to evaluate test suite prioritizations
empirical study
test coverage
test-suite prioritization
Proceedings of the International Workshop on Empirical Assessment of Software Engineering Languages and Technologies
Abstract
Regression test suite prioritization techniques reorder a test suite with the goal of ensuring that the reorganized test suite finds faults faster than the initial ordering. It is challenging to empirically evaluate the effectiveness of a new test case arrangement because existing metrics (i) require fault seeding or (ii) ignore test case costs. This paper presents a coverage effectiveness (CE) metric that (i) obviates the need to seed faults into the program under test and (ii) incorporates available data about test case execution times. A test suite is awarded a high CE value when it quickly covers the test requirements. It is possible to calculate coverage effectiveness regardless of the coverage criterion that is chosen to evaluate test case quality. The availability of an open source CE calculator enables future case studies and controlled experiments to use coverage effectiveness when evaluating different approaches to test suite prioritization.Details
Reference
@inproceedings{Kapfhammer2007,
author = {Gregory M. Kapfhammer and Mary Lou Soffa},
booktitle = {Proceedings of the International Workshop on Empirical Assessment
of Software Engineering Languages and Technologies},
title = {Using coverage effectiveness to evaluate test suite prioritizations},
year = {2007}
}