On March 19th, 2015, Stefano Di Alesio successfully defended his PhD thesis “Supporting Stress Testing of Real-Time Systems with Constraint Programming”. The defense took place in the University of Luxembourg, Campus Kirchberg.
Safety-critical Real-Time Embedded Systems (RTES) are subject to strict performance requirements specifying constraints on real-time properties such as task deadlines, response time and CPU usage. Computing input combinations that are intended to violate performance requirements, i.e., stress testing, is one of the preferred ways for verifying RTES performance and assess the system operational safety. These input combinations are referred to as stress test cases, and, upon execution, are predicted to result in worst-case scenarios with respect to a performance requirement. Generating stress test cases is challenging because it is hard to predict how the environment state will affect the system performance, and because the set of possible task executions quickly grows as the system size increases. Therefore, search strategies based on Genetic Algorithms (GA) have been proposed to find stress test cases with high chances of violating performance requirements. In the context of stress testing, it is essential to investigate the trade-off between the time needed to generate test cases (efficiency), their capability to reveal scenarios that violate performance requirements (effectiveness), and to cover different scenarios where these violations arise (diversity). Even though GA are efficient and capable of finding diverse solutions, they explore only part of the search space, and their effectiveness depends on configuration parameters. This aspect justifies considering alternative strategies, such as Constraint Programming (CP), that explore the search space completely. Furthermore, to enable effective industrial application, stress testing has to be capable of seamless integration in the development cycle of companies. Therefore, it is both important to capture specific system and contextual properties in a conceptual model, and to map such conceptual model in a standard Model Driven Engineering (MDE) language such as UML/MARTE.
The thesis addresses the challenges above by presenting a practical approach, based on CP, to support performance stress testing in RTES. Specifically, the thesis makes the following contributions: (1) a conceptual model, mapped to UML/MARTE, which captures the abstractions required to generate stress test cases, (2) a constraint optimization model to generate such test cases, and (3) a combined GA+CP stress testing strategy that achieves a practical trade-off between efficiency, effectiveness and diversity, thus being more likely than GA and CP in isolation to scale to large and complex systems.
The thesis is written within the field of Software Engineering. The work has been conducted at Simula Research Laboratory in collaboration with the Interdisciplinary Centre for Reliability, Security and Trust (SnT) of the University of Luxembourg.
The adjudication committee
• Dr. Sébastien Gérard, Commissariat à l’énergie atomique et aux énergies alternatives (CEA)
• Prof. Jean-Charles Régin, Laboratoire d’Informatique, Signaux et Systèmes (i3s), Université de Nice-Sophia Antipolis
Chair of the disputation
• Prof. Nicolas Navet, University of Luxembourg
Supervisors
• Prof. Lionel Briand, Interdisciplinary Centre for Reliability, Security and Trust (SnT) of the University of Luxembourg.
• Dr. Shiva Nejati, Interdisciplinary Centre for Reliability, Security and Trust (SnT) of the University of Luxembourg.
• Dr. Arnaud Gotlieb, Certus Centre for Software Verification & Validation, Simula Research Laboratory
