On Tuesday 18 September, Muhammad Zohaib Iqbal successfully defended his PhD thesis Environment Model-based System Testing of Real-Time Embedded Systems. The defense took place at 13:15 in Storstua, Simula Research Laboratory.

Real-time embedded systems (RTES) are part of a vast majority of computing devices available today. They are widely used in critical domains where high system dependability is required. These systems typically work in environments comprising of large numbers of interacting components. There is usually a great number and variety of stimuli from the RTES environment with differing patterns of arrival times. Testing all possible sequences of stimuli is not feasible and only a fully automated testing approach can scale up to the testing requirements of industrial RTES.

This focus of thesis is on a black-box approach for system level testing of RTES based on environment models. The main motivation is to provide a practical approach to the model-based testing of RTES. To do so, we enable system testers, who are often not familiar with the system design but are aware of the application domain, to model the environment using well-supported modeling standards, to enable test automation.

The environment models are developed using Unified Modeling Language (UML), the UML profile for Modeling and Analysis of Real-Time Embedded Systems (MARTE), Object Constraint Language (OCL), and our proposed UML profile for environment modeling. We also propose a detailed modeling methodology that fits the practical needs for industrial adoption of a RTES testing approach. The environment models capture only the details in the environment that are relevant to the SUT, including the nominal behavior and failure behavior of various environment components. The environment behavioral models also capture what we call ‘error states’ that should never be reached if the SUT is implemented correctly. The ‘error states’ act as oracles for the test cases.

Once the environment models are developed they are transformed into simulator code. For this purpose, we extend the well-known state pattern for our specific purpose, i.e., model-based testing. The environment models are used for test case generation from environment models for black-box system testing of RTES. A test case in our context is a setting of environment simulator. We conducted a number of experiments to investigate the effectiveness of various testing algorithms, specifically, Genetic Algorithms (GA) and (1+1) Evolutionary Algorithm (EA), Adaptive Random Testing (ART), Random Testing (RT), and a Hybrid Strategy (HS) that combines (1+1) EA and ART, in our context. The goal of testing in our context is to reach an ‘error state’ of the environment with as few test case executions as possible. For search algorithms we provide and iteratively improve a fitness function for effective testing. We also provide heuristics to guide the search for solving OCL constraints on the environment models. The testing strategies are evaluated on an industrial case study and a number of artificial problems. HS shows consistently good results for different types of problems and is the most practical testing strategy. On the industrial case study we were able to automatically find new, critical faults.

The thesis is written within the field of software testing. The work has been conducted at Simula Research Laboratory

Prior to the defense, at 11:15, Muhammad Zohaib Iqbal presented his trial lecture Model-Based Testing for Security of Web Applications.

The adjudication committee

Professor Bruno Legeard, University of Franche-Comté & Smartesting, France.
Professor Ina Schieferdecker, Technischen Universität Berlin and Fraunhofer, Germany.
Professor Martin Steffen, Department of Informatics, University of Oslo.

Chair of the disputation

Associate Professor Arne Maus, Department of Informatics, University of Oslo

Supervisors

Professor Lionel Briand, Simula Research Laboratory
Dr. Andrea Arcuri, Simula Research Laboratory