Introducing Major for the mutation analysis of the tests for Java programs

Introduction

If you have a JUnit test suite, it is important that you ask the question “how good are my test cases?” If you have a low quality test suite, then you may not be doing a good job at establishing a confidence in the correctness of the program under test — and there also could be defects lurking in the application! How can you find these bugs and establish a confidence in your tests?

One strategy for assessing the quality of your test suite is to run coverage analysis with a tool like JaCoCo. Since a test case cannot find a defect unless it first executes it, the idea is that you can judge a test suite to be good if it adequately covers the code in the program under test. However, coverage analysis does not help a tester to know with certainty that the tests are design in such a way that they are likely to find defects. As an alternative to, or in addition to coverage analysis, mutation testing inserts simple faults into the program under test to determine whether or not the tests can find them. Intuitively, a test case that can find many of these faults is a good one and faults that are overlooked suggest testing weaknesses.

Major

While mutation analysis is simple and useful, historically, it has been difficult to apply to large-scale programs because it was computationally expensive. This all changed with the design, implementation, and release of Major, a compiler-integrated mutation testing tool for the Java programming language. You can learn more about Major by reading (Just, Kapfhammer, and Schweiggert 2011b) and (Just, Kapfhammer, and Schweiggert 2011a) . More details about how Major makes mutation testing scalable through use of non-redundant mutation operators and test suite prioritization is available in (Just, Kapfhammer, and Schweiggert 2012) . Starting to use the tool is as easy as changing the Java compiler that you use at build time, as demonstrated by the following line from Major’s online documentation.

$ javac -XMutator:ALL MyFile.java
# Generated Mutants: 190 (96 ms)

The creator of Major, Rene Just, took great care to ensure that it is fast, allowing mutation testing to be applied to large real-world programs. Major version 2.0.0, for a recent version of the Java programming language, is now available for download—check Major’s site so that you can download the tool and get started with mutation analysis for your JUnit test suites! If you are interested in Major, then you may also want to investigate the use of PIT, another mutation testing tool for programs written in the Java language.

Return to Blog Post Listing