A generic testing framework for agent-based simulation models

Abstract

Agent-based modelling and simulation (ABMS) had an increasing attention during the last decade. However, the weak validation and verification of agent-based simulation models makes ABMS hard to trust. There is no comprehensive tool set for verification and validation of agent-based simulation models, which demonstrates that inaccuracies exist and/or reveals the existing errors in the model. Moreover, on the practical side, many ABMS frameworks are in use. In this sense, we designed and developed a generic testing framework for agent-based simulation models to conduct validation and verification of models. This paper presents our testing framework in detail and demonstrates its effectiveness by showing its applicability on a realistic agent-based simulation case study.

Keywords:

• agent-based modelling and simulation
• model testing
• verification and validation

Acknowledgements

The authors thank Kemal S. Türker and S. Utku Yavuz from Ege University Center for Brain Research for supplying scientific data about the activity of motoneurons. The work described here was partially supported by Ege University under the BAP 10-MUH-004 project. Önder Gürcan is supported by the Turkish Scientific and Technical Research Council (TUBITAK) through a domestic PhD scholarship program (BAYG-2211) and by the French Government through the co-tutelle scholarship program.

Notes

1 The term level and the term scale are often used interchangeably. Here, the term level is chosen since it situates the described system considering its size, its characteristic evolution time or an inclusion relation with other systems or subsystems. However, the term scale refers to a dimension of analysis in which the system of interest can be measured.

2 This algorithm arranges data points into clusters and it locates a centroid in each cluster This centroid is the point at which the distance from the rest of the points of the clusters is on average minimum.

3 Law (2007) defines a simulation as a numerical technique that takes input data and creates output data based upon a model of a system.

4 For example, Wolfram (1994) defines four classifications into which different Cellular Automata systems can be placed based on their long-term behaviours. The first one is evolving to a homogeneous state, which means that changes to the initial state have no impact on the final state.

5 Burnstein (2003) refers data generators as load generators since load generators are aimed at being used in system-level tests. However, we use the term data generator since it can be used in all levels.

6 The Unified Modeling Language (UML) is a standard that can be used ‘for analysis, design, and implementation of software-based systems as well as for modeling business and similar processes’. Thus, the UML is a visual language that can be used to create software architectures.

7 We do not address implementation issues on how to apply these concepts in practice, as this is highly dependent upon the simulation framework used and the objective of the simulation study.

8 Since many agent-based simulators use a global scheduler, such information can be retrieved from the scheduler of the simulation infrastructure.

9 http://repast.sourceforge.net/, latest accessed 13 July 2012.

10 http://cs.gmu.edu/eclab/projects/mason/, latest accessed 13 July 2012.

11 JUnit, http://www.junit.org/

12 A Dialog window (in Java) is an independent subwindow meant to carry temporary notice apart from the main Swing Application Window. Most Dialogs present an error message or warning to a user, but Dialogs can present images, directory trees, or just about anything compatible with the main Swing Application that manages them.

13 Motor units are composed of one or more motoneurons and all of the muscle fibres they innervate.

14 A reflex pathway is a type of neural pathway involved in the mediation of a reflex. Reflexes are involuntary reactions that occur in response to stimuli. They often bypass the brain altogether, allowing them to occur very quickly, although the brain receives information about the reflex as it happens.

15 The soleus is a powerful muscle in the back part of the lower leg (the calf). It runs from just below the knee to the heel, and is involved in standing and walking.

16 Ege University Center for Brain Research, http://www.eubam.ege.edu.tr/.

17 The time for the end of the scenario execution may change at each execution, according to the values given by the developer for ScenarioExecuter. See Algorithm 8.

18 In this article, testing methods and techniques with respect to the MAS properties they are able to address are extensively reviewed. But none of the reviewed studies focus on ABMS.

19 The meso-level testing example given in this paper (Section 4.2) is a good example of white-box testing. In this test, the behaviour of a sub-society in order to handle a problem in certain conditions is tested by controlling its internal organization. The same sub-society, in another condition, may behave differently to handle the same problem. For example, rather than creating a new synapse, the sub-society may create a new neuron agent. This kind of solution may also satisfy this sub-society and make it produce the same output. However, the internal organization will be totally different. Therefore, only testing inputs and outputs in order to control uncertainty is not enough.