November 24, 2016

Smarter Testing of Evolving Systems

Software systems involved in real-world activities must continually adapt in order to satisfy the increasing demands of their users. Evolving systems are typically validated through regression testing, which aims at ensuring that new changes in a system do not negatively affect the behavior of unchanged parts. Traditionally, regression testing is performed by comparing test results before and after changes are made, under the assumption that different outcomes imply that the changes had negative effects. As the software evolves, test suites of large systems tend to grow in size, and may become partly outdated or contain redundant test cases. Over time, it becomes prohibitively expensive to execute the entire test suite available for a system, rendering traditional regression testing highly inefficient. This project investigates an alternative approach, based on analysis of historical data on past changes in a software project, in order to predict the parts of the system affected by a given change. This is based on the notion that software artifacts that often changed together in past releases are likely to be interdependent, and hence are also likely to continue changing together. This knowledge allows regression testing to focus only on the software artifacts impacted by the changes, without having to execute the entire suite of tests.

Goals

The Certus project ‘Smarter testing of evolving systems’ aims at devising practical, scalable, cost-effective automated techniques that empower current industrial practices in regression testing. The goal of the project is threefold:

1) to develop advanced techniques to mine knowledge about past changes from software repositories, enabling the tester to

2) identify the parts of the system affected by a given change, and

3) prioritize the execution of test cases that exercise the changed parts, so that the most vulnerable parts of the system are tested as soon as possible.

Impact

Software systems are pervasive in our society, and are destined to fully integrate into all aspects of life. Increasing user demand for software functionality and reliability represents a significant challenge when delivering highquality software products. More time-efficient and costeffective regression testing allows our partner companies to significantly improve their software quality assurance processes. In particular, smarter testing of evolving systems allows for reallocation of regression testing resources to other key phases of software development, while keeping pace with fast-growing user demands. This ultimately allows our partners to increase their competitiveness in the market by delivering better products.

Partners

Our partners Kongsberg Maritime and Cisco Norway are world leaders in the development of high-quality evolving software systems in their respective sectors. However, due to the increasing software complexity, our partners face severe challenges when testing regressions in their own products. Specifically, reducing the time needed for investigating regressions in a set of changes, while thoroughly verifying that those changes do not yield any unintended effects, is among their top priorities in improving testing processes. For these reasons, both Kongsberg Maritime and Cisco Norway constitute ideal partners for devising advanced techniques for regression testing scaled to large and complex industrial systems.