Tuesday, May 17, 2016

Colloquium: Model Based Testing Approach for Objective Fidelity Evaluation

Next week I will be offering a colloquium in TU Clausthal Department of Informatics about one of the topics that I worked and published intensively during the last couple of years. This is a collaborative research effort that I conducted with Prof.Dr.Thorsten Pawletta and his research group Computational Engineering and Automation.
The increasing complexity of the systems entails an increasing complexity of simulation models. Likewise, heterogeneity in system components corresponds to heterogeneous simulation models. Due to the growth of simulation complexity and heterogeneity, large numbers of test cases are required to reach admissible coverage to assure adequate simulation fidelity which is the measure for the conformance of a simulation to the characteristics of the real system that it represents. Objective fidelity evaluation is an engineering approach that attacks the fidelity problem with comparison of simulation and the actual system behavior over some quantitative measures. In order to tackle emerging challenges, adaptability, flexibility and automation can be introduced as the key characteristics of a fidelity evaluation approach that determines its success. Model Based Testing (MBT) will be presented as the enabler to achieve these quality characteristics. Despite the fact that it is widely used in the software testing community, its application in modeling and simulation is quite limited. 
Model-based methodologies ask for metamodels to express models. Metamodeling on the other hand requires a complete and accurate specification of concepts. Regarding the testing of simulation models as experimentation, the concept of experimental frames (EF) that originates from the Discrete Event System Specification (DEVS) are proposed for formally specifying simulation test models. System Entity Structure (SES) is then promoted for metamodeling. While EF formally specifies a limited set of circumstances under which a model has to be observed, SES can be defined as an ontology with a limited set of elements and axioms that are used to describe various system structures and their configurations. For generating an executable EF, configurable Basic Models (BMs) for objective fidelity evaluation are provided by a Model Base (MB). BMs usually correspond to atomic or coupled models which are used to construct modular, hierarchical models. The SES is represented by a directed and labelled tree with links to BMs in the MB. The proposed MBT approach for the objective fidelity evaluation will first described in detail and then a prototype implementation in MATLAB/Simulink will be introduced. 
Objective fidelity assessment is a tedious and labor intensive effort also for flight simulators. Yet, in the global standards for qualification of flight training devices, like International Civil Aviation Organization (ICAO) 9625 Manual of Criteria for the Qualification of Flight Training Devices, the set of test cases for objective fidelity evaluation are specified. The root motivation of this effort was to attack objective fidelity evaluation of flight dynamics models that have been developed in German Aerospace Center (DLR). Accordingly, I will be presenting a case study from flight simulation domain and discussing anecdotal evidences about the pros and cons of the proposed approached based on this case study.

Edit:
The video footage of the colloquium is available in this link.