Eliciting agents’ behaviour and model validation using role playing game in agent-based dairy supply chain model

Figures & data

Table 1. Summary of the literature survey.

Author	The use of the RPG	ABM development approach	How RPG was used to develop & validate ABM	Validation Levels
				Micro	Macro
Castella et al. (2005)	3	Evidence	Mode 1	Qual	Quant
Castella et al. (2005)	3	Evidence	Mode 1	Qual	Quant
Tykhonov et al. (2008)	3	Theory	Mode 2	Quant	Qual
Campo et al. (2009)	3	Evidence	Mode 1	Qual	N/A
Ligtenberg et al. (2010)	3	Theory	Mode 2	Qual	N/A
Worrapimphong et al. (2010)	2	Evidence	Not relevant*
d'Aquino and Bah (2014)	3	Evidence	Mode 1	Qual	N/A
Chaturvedi et al. (2014)	1	Theory	Not relevant*
Joffre et al. (2015)	3	Evidence	Mode 1	Qual	N/A
Salvini et al. (2016)	3	Evidence	Mode 1	Qual	N/A
Papazian et al. (2017)	3	Evidence	Mode 1	Qual	N/A
Amadou et al. (2018)	3	Theory	Mode 2	Quant	N/A
This paper	3	Theory	Mode 3 (new)	Quant & Qual	Quant

The use of the RPG (Guyot & Honiden, 2006): (1) to train the stakeholders; (2): to facilitate collective learning and negotiation; (3) to validate and improve the design of the ABM. ABM development approach (Moss, 2008): Theory-driven model development; Evidence-driven model development. How RPG was used to develop & validate ABM (Ligtenberg et al., 2010): Mode 1 or Mode 2. The micro/macro validation: Qual: Qualitative approaches (e.g., observation, discussion, interview, visual inspection, subjective validation, or theoretical validation); Quant: Quantitative approaches (e.g., descriptive statistics or operational validation); N/A: Not discussed.

* Not relevant because the aim of the RPG is solely for training the decision makers or for facilitating collective learning.

Table 2. Levels of ABM validity adapted from Ligtenberg et al. (2010) that is mapped to the previous validity classification.

Levels of ABM validity	Description
System representation (Macro-validation)	This validation evaluates how well the model represents the aggregate or global pattern. If data are available, then the ABM can be validated by matching the model’s outputs with empirical data.
Process representation (Micro-validation)	This validation evaluates the representativeness of the processes within an ABM to the tasks that real actors must carry out to fulfil their goals.
Knowledge representation (Micro-validation)	This validation aims to verify whether the information and knowledge used by the agents sufficiently represent those of the real-world actors.

Figure 1. Research method flowchart.

Table 3. conceptual model developed for the case study.

Organisational Aim (from the view point of the government and farmers cooperative)
To propose policies that may support the sustainability of dairy supply chain in Pangalengan
Modelling Objectives
To model the farmers behaviour in producing milk and selling milk To understand the relationship between the farmers behaviour and parameters such as cow population, cattle population, and milk production
Outputs (to determine achievement of objectives)
Time series of the number of farmer households Time series of the cow population Time series of the cattle population Time series of the milk production
Components	Included / excluded	Justification
Agents:
Farmer household	Included	The agent rear the cattle and produce milk.
Farmers’ cooperative	Included	The agent who determine the milk price
Environment	Included	Provide forage as an input for milk production
Cattle (bulls and cows)	Included within farmer household agent	Determine the number of milk that can be produce by the farmer agent and influence the income of the farmer household agents.
Activities / decision rule:
Forage growth	Included	Determine the number of forage that can be obtained by the farmer agents
Farmers collect forage	Included	Influence the cattles’ health and the amount of milk that can be produced
Farmers produce and sell milk	Included	Influence the farmers’ income and eventually the number of farmer households
Cooperative determine milk price	Included	Influence the farmers’ income and eventually the number of farmer households
Farmers buy or sell cattle and cow	Included	Influence the cattle population, milk production and eventually the farmers’ income
Cattle reproduction	Included	Influence the cattle population, milk production and eventually the farmers’ income

Figure 2. Flowchart of the base ABM of dairy supply chain in West Java Indonesia, accompanied with the main literature used to develop each module. (Please note that procedures using the term “cattle” apply to both male and female animals, while those that use the term “cow” only apply to a female animal.).

Figure 3. (a) The RPG game board, (b) the back side of cow card, (c) the front side of cow card, (d) the vehicle card, (e) the event card, (f) the mortality card for the dead cow, and (g) the mortality card for the surviving cow.

Table 4. Summary of RPG’s parameters.

Parameter	Estimation	Explanation
Size of a cell	2.5 km × 2.5 km	The reason for selecting this resolution has been discussed in the Section 4.3.2.
Maximum number of moves by a player	12 cells per round	By considering the uncertainty of working hours utilisation explained in Section 4.3.4 farmers can travel up to 30 km per day; therefore, 12-sided dice is used.
Amount of forage in one button	10.95 tonnes	On average, a cow needs 10.95 tonnes of forage per year to stay healthy.
Number of buttons	27 to 49 buttons	The total forage production on the case study’s site is estimated between 303.23 and 545.82 tonnes.
Capacity in a vehicle card	2 to 4 buttons	A farmer household is able to transport between 22.95 and 43.98 tonnes per year
Distribution of cow’s productivities in cow cards (Prod)	Prod (5) = 3 cards, Prod (10) = 7 cards, Prod (15) = 51 cards, Prod (20) = 79 cards, Prod (25) = 51 cards, Prod (30) = 7 cards and Prod (35) = 3 cards	The empirical distribution of cow productivity is $\mathcal{N}\left(20.44;{4.33}^2\right).$ We divide this distribution into 7 intervals and use the midpoint of each interval. The number of cow cards with a certain level of is calculated accordingly (e.g., $P\left(\mathit{Productivity}=10\right)=3.5\mathrm{\%}$ of all cow cards)
Distribution of cow’s productive lifetimes in cow cards (PL)	PL (5) = 67 cards, PL (7) = 67 cards, PL (10) = 67 cards	It is assumed to be uniformly distributed $U\left(4,10\right).\mathrm{}$ We divide this distribution into 3 intervals and use the midpoint of each interval. The number of cow cards with a certain productivity lifetime is calculated accordingly (e.g., $P\left(\mathit{Productivity}\mathrm{}\mathit{lifetime}=10\right)=33.33\mathrm{\%}$ of all cow cards)
Number of events related to the birth of a new cow in event cards (birth)	Birth (0) = 149 cards, Birth (1) = 42 cards, Birth (2) = 6 cards, out of 197 event cards	The survey data shows that the birth of new cows has a Poisson distribution with mean of 0.29 per year.
Distribution of events related to cash flows (CF) in event cards	CF (−16,000) = 1 cards, CF (−13,000) = 5 cards, CF (−10, 000) = 24 cards, CF (−7,000) = 56 cards, CF (−4,000) = 64 cards, CF (−1.000) = 37 cards, CF (2,000) = 10 cards, out of 197 cards.	The typical additional income comes from selling bulls. The distribution of money received from selling bulls per year is $\mathcal{N}\left(977.21;{1,604.03}^2\right).$ The distribution of expenses is $\mathcal{N}\left(7,370.02;{3,112.47}^2\right)$ annually. By combining the two distributions altogether, the distribution of cash flow is$\mathcal{N}\left(-6,392.81;{3,501.48}^2\right).$ We divide this distribution into seven intervals with mid points of −16,000; −13,000; −10,000; −7,000; −4,000; −1,000 and 2,000. The units are in Rp.10,000.
Milk selling price	Rp. 120 per litre in 10,000	The average milk selling price from the survey is Rp. 4,480 per litre. A cow can produce milk for nine months in a year. Thus, in a year the selling price of one litre of milk is Rp. 1,209,600.
Market price of a vehicle	Rp. 2,000 in 10,000	This price is paid when a farmer buy a vehicle from the market. If a farmer buys it from another farmer, they can negotiate the price. Based on survey data, the average vehicle price in the area is Rp. 22,600,000.
Market price of a cow	Rp. 1,500 in 10,000	This price is paid when a farmer buy a cow from the market. If a farmer buys it from another farmer, they can negotiate the price. Based on survey data, the average cow price in the area is Rp. 15,670,000 per head.

Table 5. The final RPG experiment sets.

		Experiments			Additional Experiment after pilot
Main treatments		Probability of cow’s mortality
		Low (30%)	Medium (50%)	High (70%)	Very High (90%)
Forage Availability	Low (24 Forage Buttons)		Experiment 4
	Medium (36 Forage Buttons)	Experiment 1	Experiment 2	Experiment 3	Experiment 6
	High (48 Forage Buttons)		Experiment 5

Figure 4. Clustering analysis result.

Table 6. Player composition in the RPG experiments.

RPG Experiment	Cluster 1	Cluster 2	Cluster 3
1	1 player	1 player	2 players
2	0 player	1 player	3 players
3	1 player	1 player	2 players
4	1 player	1 player	2 players
5	0 player	0 player	4 players
6	0 player	1 player	3 players
#players out of #farmers in the sample	3 out of 3 farmers	5 out of 20 farmers	16 out of 131 farmers

Table 7. Cattle population, cow population and average daily milk production in Pangalengan West Java 2010–2012 (KPBS, 2016).

Year	Cattle population (head)	Cow population (head)	Average daily Production (litre)
2010	21,322	21,083	159,333
2011	21,438	20,960	136,694
2012	22,366	22,073	138,904

Table 8. The macro-validation of the ABM.

	Cattle Population			Cow Population			Daily Milk Production
Model Name	$\overline{ME}$	$S_{ME}$	Sig.	$\overline{ME}$	$S_{ME}$	Sig.	$\overline{ME}$	$S_{ME}$	Sig.
M0 Base model	1672.7	5023.9	0.08	898.0	4576.2	0.29	−8475.4	17239.6	0.01
M1 Forage collection	6974.5	10408.8	0.00	5533.1	9303.0	0.00	12726.4	37094.4	0.07
M2 Selling decision	1715.4	5082.5	0.07	1101.7	4514.8	0.19	6011.0	22554.4	0.16
M3 Buying decision	1716.5	5048.4	0.07	933.8	4593.6	0.27	−8356.4	17398.2	0.01

Note: Shaded cells indicate that the null hypothesis cannot be rejected at 5% significance level.

Table

Session	Round	Player	Forage Level	Cow Mortality	Forage Collection Strategy	Forage Transaction	Forage in / out	Forage deficit	Mortality Card Taken	Cow transaction	Cow bought	Cow sold	Vehicle bought	Vehicle Sold	Comment
1	1	NE	48	0.5		1	−2
1	1	NW	48	0.5		−1	2
1	2	NE	48	0.5				−2	2	1	2
1	2	SE	48	0.5	1
1	2	SW	48	0.5				−1	1
1	3	NE	48	0.5	1
1	3	SE	48	0.5	1	−2	3
1	3	SW	48	0.5		−2	3
1	4	NE	48	0.5						2	1				never buy cows from a farmer
1	4	SE	48	0.5	1			−4	3	2		1			never buy cows from a farmer
1	5	NE	48	0.5						1	1
1	5	NW	48	0.5						1	1
1	5	SE	48	0.5	1					1	3		3
1	5	SW	48	0.5						1	1
2	1	NE	48	0.5						1	1
2	1	NW	48	0.5						1	1
2	1	SE	48	0.5	1
2	1	SW	48	0.5						1	2
2	2	NW	48	0.5	1	−1				1	1	1			upgrading
2	2	SE	48	0.5	1	1	−4
2	3	NE	48	0.5		1	−3
2	3	SE	48	0.5	1	2	−3
2	4	NW	48	0.5									1
2	4	SE	48	0.5	1								1
2	5	NW	48	0.5		1	−1
2	6	NW	48	0.5		1	−1
1	1	NW	36	0.3	4
1	1	SW	36	0.3	4
1	2	NE	36	0.3	2					1		1
1	2	NW	36	0.3				−3	2	1	1
1	2	SE	36	0.3				−1	1	1	1
1	2	SW	36	0.3	5			−1	1
1	3	NE	36	0.3	3					1	1
1	3	NW	36	0.3						1	1
1	3	SE	36	0.3				−4	3	1		1
1	4	NW	36	0.3									1
1	4	SE	36	0.3	2
1	4	SW	36	0.3	1					1	1
1	5	NE	36	0.3						1	2
1	5	NW	36	0.3						1	1
1	6	SE	36	0.3						2	1				never buy cows from a farmer
1	6	SW	36	0.3	1					2		1			never buy cows from a farmer
2	1	NE	36	0.3						1	1
2	1	NW	36	0.3						1	1
2	1	SE	36	0.3						1	1
2	1	SW	36	0.3	1					1	1
2	3	NW	36	0.3											shared farming
2	3	SE	36	0.3		1	−1
2	3	SW	36	0.3	1					1	1	1			Trade in
2	4	NW	36	0.3		2	−2
2	4	SE	36	0.3		1	−1
2	4	SW	36	0.3	1
2	5	SW	36	0.3	1
2	7	SE	36	0.3	4
2	8	NW	36	0.3									1
1	1	NE	36	0.5	1			−1	1
1	1	NW	36	0.5				−2	2
1	1	SE	36	0.5	4
1	1	SW	36	0.5	4
1	2	NE	36	0.5	1
1	2	SE	36	0.5	4
1	2	SW	36	0.5	1
1	3	NE	36	0.5	1	2		−1	0	1	1
1	3	NW	36	0.5	2
1	3	SE	36	0.5		−2	1	−1	0	1		1
1	3	SW	36	0.5						1	1
1	4	NE	36	0.5	1
1	4	SE	36	0.5				−1	1
1	5	SW	36	0.5	2
1	6	NW	36	0.5	2
1	6	SW	36	0.5	2
2	1	SE	36	0.5						1		2		1
2	1	SW	36	0.5						1		1		1
2	2	NE	36	0.5	1	1
1	1	NW	24	0.5				−2	0	1		2
1	1	SE	24	0.5				−2	1	1		1
1	1	SW	24	0.5				−2	1	1		1
1	2	NE	24	0.5	1
1	2	NW	24	0.5	1
1	2	SE	24	0.5		2	−1
1	2	SW	24	0.5		2	−1
1	3	NE	24	0.5		2
1	3	NW	24	0.5		−1
1	4	SE	24	0.5		2	−1
1	5	SW	24	0.5		−2	1
1	6	NW	24	0.5		1
2	1	SE	24	0.5						1	1
2	2	NE	24	0.5				−2	2
2	2	NW	24	0.5						1		1
2	2	SW	24	0.5				−1	1
2	3	NE	24	0.5		3
2	3	NW	24	0.5		3
2	4	SE	24	0.5		2	−1
1	1	SE	36	0.7				−2	2
1	1	SW	36	0.7				−4	4
1	2	NE	36	0.7						1	2
1	2	NW	36	0.7	4
1	2	SE	36	0.7						1	1
1	3	NE	36	0.7		3	−2
1	3	NW	36	0.7				−1	1
1	3	SE	36	0.7						1	1
1	3	SW	36	0.7		3	−1
1	5	SE	36	0.7	4
1	5	SW	36	0.7	1
1	6	NE	36	0.7	1
1	6	SW	36	0.7						1	1	1			upgrading
1	7	NW	36	0.7	1	3	−1
2	1	SE	36	0.7		3	−1
2	3	NE	36	0.7						1	1
2	3	SE	36	0.7						1	1
2	3	SW	36	0.7						1	1
2	4	NW	36	0.7		3	−1
2	4	SW	36	0.7						1	1	1			upgrading
2	6	NE	36	0.7		3	−1
2	6	NW	36	0.7		3	−1
2	7	NE	36	0.7	1
2	7	SE	36	0.7	1
2	7	SW	36	0.7						2	1				never buy cows from a farmer
1	1	NE	36	0.9				−2	1	1		2
1	1	NW	36	0.9	1			−2	0	1		2
1	1	SE	36	0.9		−1	1
1	2	NE	36	0.9						1	2
1	2	NW	36	0.9	1					1	1
1	2	SW	36	0.9	1					1	1	1			upgrading
1	3	NE	36	0.9						1	1
1	3	NW	36	0.9	1					1	1	1			upgrading
1	3	SW	36	0.9	1					1	1	1			upgrading
1	4	NE	36	0.9						1	1	1			upgrading
1	4	NW	36	0.9	1
1	4	SE	36	0.9						1	1
1	4	SW	36	0.9	1
1	5	NE	36	0.9						1			1
1	5	NW	36	0.9	1					1	2		1
1	5	SW	36	0.9	1					1	1		1
1	6	NE	36	0.9		2	−2			1	1	1			upgrading
1	6	SE	36	0.9						1	1
1	6	SW	36	0.9	1					1	1
2	1	NE	36	0.9				−1	0	1	1	1
2	1	NW	36	0.9	1										shared farming
2	1	SE	36	0.9				−2	0	1		2
2	2	NE	36	0.9						1	1
2	2	NW	36	0.9	1					1	1	1			upgrading
2	2	SE	36	0.9		1	−4
2	2	SW	36	0.9	1					1	1	1			upgrading
2	3	NE	36	0.9				−2	0	1		2
2	3	NW	36	0.9	1			−1	0	1	1	1
2	4	NE	36	0.9		3	−1
2	4	NW	36	0.9	1	2	−2
2	4	SE	36	0.9		2	−2
2	5	NE	36	0.9				−1	0	1		1
2	5	NW	36	0.9	1	2	−4

Table

Case Processing Summary
		N	Marginal Percentage
Willing_to_sell	No	18	66.7%
	Yes	9	33.3%
Valid		27	100.0%
Missing		124
Total		151
Subpopulation		15^a

^a The dependent variable has only one value observed in 12 (80.0%) subpopulations.

Table

Model Fitting Information
Model	Model Fitting Criteria	Likelihood Ratio Tests
	−2 Log Likelihood	Chi-Square	Df	Sig.
Intercept Only	25.243
Final	12.134	13.109	3	.004

Table

Pseudo R-Square
Cox and Snell	.385
Nagelkerke	.537
McFadden	.381

Table

Likelihood Ratio Tests
Effect	Model Fitting Criteria	Likelihood Ratio Tests
	−2 Log Likelihood of Reduced Model	Chi-Square	df	Sig.
Intercept	12.211	.077	1	.781
For_Level	12.794	.660	1	.567
Mortality_Rate	22.802	10.668	1	.001
Forage_Deficit_Post	14.847	2.713	1	.149

The chi-square statistic is the difference in -2 log-likelihoods between the final model and a reduced model. The reduced model is formed by omitting an effect from the final model. The null hypothesis is that all parameters of that effect are 0.

Table

Parameter Estimates
Willing_to_sell^a		B	Std. Error	Wald	df	Sig.	Exp(B)	95% Confidence Interval for Exp(B)
								Lower Bound	Upper Bound
Yes	Intercept	−1.109	3.839	.078	1	.779
	For_Level	−.081	.105	.593	1	.591	.922	.751	1.133
	Mortality_Rate	7.953	3.407	6.403	1	.011	2.844E3	6.984	4415995.063
	Forage_Deficit_Post	−1.444	1.138	1.610	1	.253	.236	.025	2.195

^a The reference category is: No.

Castella, J.-C., Boissau, S., Trung, T. N., & Quang, D. D. (2005). Agrarian transition and lowland–upland interactions in mountain areas in northern Vietnam: Application of a multi-agent simulation model. Agricultural Systems, 86(3), 312–332. https://doi.org/10.1016/j.agsy.2004.11.001

 Web of Science ®Google Scholar

Tykhonov, D., Jonker, C. M., Meijer, S., & Verwaart, D. (2008). Agent-based simulation of the trust and tracing game for supply chains and networks. Journal of Artificial Societies and Social Simulation, 11(3), 1–30.

 Web of Science ®Google Scholar

Campo, P. C., Mendoza, G. A., Guizol, P., Villanueva, T. R., & Bousquet, F. (2009). Exploring management strategies for community-based forests using multi-agent systems: A case study in Palawan. Journal of Environmental Management, 90(11), 3607–3615.

 PubMed Web of Science ®Google Scholar

Ligtenberg, A., van Lammeren, R. J. A., Bregt, A. K., & Beulens, A. J. M. (2010). Validation of an agent-based model for spatial planning: A role-playing approach. Computers, Environment and Urban Systems, 34(5), 424–434. https://doi.org/10.1016/j.compenvurbsys.2010.04.005

 Web of Science ®Google Scholar

Worrapimphong, K., Gajaseni, N., Le Page, C., & Bousquet, F. (2010). A companion modeling approach applied to fishery management. Environmental Modelling & Software, 25(11), 1334–1344. https://doi.org/10.1016/j.envsoft.2010.03.012

 Web of Science ®Google Scholar

d'Aquino, P., & Bah, A. (2014). Multi-level participatory design of land use policies in African drylands: A method to embed adaptability skills of drylands societies in a policy framework. Journal of Environmental Management, 132, 207–219. https://doi.org/10.1016/j.jenvman.2013.11.011

 PubMed Web of Science ®Google Scholar

Chaturvedi, A., Armstrong, B., & Chaturvedi, R. (2014). Securing the food supply chain: Understanding complex interdependence through agent-based simulation. Health and Technology, 4(2), 159–169.

 Google Scholar

Joffre, O. M., Bosma, R. H., Ligtenberg, A., Tri, V. P. D., Ha, T. T. P., & Bregt, A. K. (2015). Combining participatory approaches and an agent-based model for better planning shrimp aquaculture. Agricultural Systems, 141, 149–159. https://doi.org/10.1016/j.agsy.2015.10.006

 Web of Science ®Google Scholar

Salvini, G., Ligtenberg, A., Van Paassen, A., Bregt, A., Avitabile, V., & Herold, M. (2016). REDD + and climate smart agriculture in landscapes: A case study in Vietnam using companion modelling. Journal of Environmental Management, 172, 58–70.

 PubMed Web of Science ®Google Scholar

Papazian, H., Bousquet, F., Antona, M., & d'Aquino, P. (2017). A stakeholder-oriented framework to consider the plurality of land policy integration in Sahel. Ecological Economics, 132(13), 155–168. https://doi.org/10.1016/j.ecolecon.2016.10.020

 Web of Science ®Google Scholar

Amadou, M. L., Villamor, G. B., & Kyei-Baffour, N. (2018). Simulating agricultural land-use adaptation decisions to climate change: An empirical agent-based modelling in northern Ghana. Agricultural Systems, 166, 196–209. https://doi.org/10.1016/j.agsy.2017.10.015

 Web of Science ®Google Scholar

Guyot, P., & Honiden, S. (2006). Agent-based participatory simulations: Merging multi-agent systems and role-playing games. Journal of Artificial Societies and Social Simulation, 9(4).

 Web of Science ®Google Scholar

Moss, S. (2008). Alternative approaches to the empirical validation of agent-based models. Journal of Artificial Societies and Social Simulation, 11(1), 5.

 Web of Science ®Google Scholar

KPBS. (2016). Data Populasi dan Penghasilan Anggota KPBS. [Data set].

 Google Scholar