Prioritizing Manual Test Cases in Traditional and Rapid Release Environments

(Selected as one of the best papers for journal extension)

Written by: H. Hemmati, Z. Fang, M. Mantyla. IEEE International Conference on Software Testing, Verification and Validation (ICST), 2015. 

 

Test suite prioritization is one of the most practically
useful activities in testing, specially for large scale systems.
The goal is ranking the existing test cases in a way that they
detect faults as soon as possible, so that any partial execution of
the test suite detects maximum number of defects for the given
budget. Test prioritization becomes even more important when
the test execution is time consuming, e.g., manual system tests
vs. automated unit tests. Most existing test suite prioritization
techniques are based on code coverage, which requires access
to source code. However, system tests are mainly black-box.
Therefore, in this paper, we first examine the existing test suite
prioritization techniques and modify them to be applicable on
manual black-box system testing. We specifically study a coveragebased,
a diversity-based, and a risk driven approach for test suite
prioritization. Our empirical study on four older releases of
Mozilla Firefox shows that non of the techniques are strongly
dominating the others in all releases. However, when we study
nine more recent releases of Firefox, where the development has
been moved from a traditional to a more agile and rapid release
environment, we see a very signifiant difference (on average 65%
effectiveness improvement) between the risk-driven approach and
its alternatives. Our conclusion, based on one case study of 13
releases of an industrial system, is that test suites in rapid release
environments, potentially, can be very effectively prioritized for
execution, based on their historical riskiness; whereas the same
conclusions do not hold in the traditional software development
environments.
Read More…