Using eXtreme mutation testing to discover hidden test suite weaknesses

Introduction

Ever wondered if your test suite is truly good at finding bugs? What if there are hidden weaknesses that traditional testing methods miss? In the paper (Maton, Kapfhammer, and McMinn 2024a) , my colleagues and I explore how eXtreme mutation testing (XMT) can uncover these hidden weaknesses in your test suite. By identifying pseudo-tested methods — or those methods in your program that appear to be tested even though they are not — XMT can help you improve your test suite’s effectiveness and ensure more robust software. Let’s dive into how these XMT works and why it may be a game-changer for software testing!

Understanding XMT

An XMT tool automatically detects deficiencies in a test suite by deleting method bodies in covered code and observing whether the test suite detects their absence. If a method deletion goes unnoticed, then the method is labeled as “pseudo-tested,” meaning that the test suite calls the method but its result isn’t checked by the assertions, indicating a potential blind spot in the test suite. As an extension to XMT, our paper explores the use of the statement deletion mutation operator (SDL) to uncover pseudo-tested statements within methods.

Research Insights

Using the PseudoSweep tool (Maton, Kapfhammer, and McMinn 2024b) to perform mutation testing with the XMT and SDL operators on four Apache Commons projects and 23 projects randomly selected from the Maven Central Repository, this paper’s experiments find 722 different cases of pseudo-tested statements. Interestingly, the results show that 48% of pseudo-tested statements exist outside of pseudo-tested methods! This suggests that XMT alone is insufficient and should be combined with pseudo-tested statement evaluation to find testing oversights that a developer should tackle before using standard mutation testing.

Future Directions

While our research focused on Java programs, the concepts of XMT and pseudo-tested methods are broadly applicable. These techniques can be used in various programming languages to improve test suite effectiveness and ensure more robust software. We’re excited about the potential of combining XMT with other testing techniques to further enhance test suite robustness. Future work will involve extending our approach to different programming languages and exploring additional ways to detect and repair pseudo-tested methods.

Return to Blog Post Listing