Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges

Authors: Sina Shamshiri, Rene Just, Jose Miguel Rojas, Gordon Fraser, Phil McMinn and Andrea Arcuri

Funded by: EPSRC and the National Research Fund, Luxembourg.

Rather than tediously writing unit tests manually, tools can be used to generate them automatically - sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include:

  • improving the obtained code coverage so that faulty statements are executed in the first instance
  • improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions
  • improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.

The findings reported in this paper have implications for software developers, who get to see the real state of the art in unit test generation - which is, surprisingly, worse than expected, The paper further has an impact on software engineering research, where it helps directing research efforts to address the main problems identified. In recognition of its importance, this paper was awarded an ACM SIGSOFT Distinguished Paper Award at the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015).