Designing empirical experiments to compare interactive multiobjective optimization methods

Figures & data

Table 1. Composite indicator values for sustainability dimensions in Finland.

	Social	Economic	Environmental
Ideal values	2.3209	2.5932	4.0000
Current values	2.3142	1.8395	2.0546
Nadir values	2.3107	1.6207	1.7377

Table 2. Selected desirable properties from (Afsar et al., 2021b) with the corresponding research questions.

Selected desirable properties	Research questions
“The method sets as low cognitive burden on the DM as possible.”	RQ1
“The method allows the DM to fine-tune solutions in a reasonable number of iterations and/or reasonable waiting time.”	RQ2
“The method captures the preferences of the DM.”	RQ3
“The DM feels being in control while interacting with the method.”	RQ3
“The method allows the DM to learn about the conflict degree and trade-offs among the objectives in each part of the Pareto optimal set explored.”	RQ4
“The method allows the DM to be fully convinced that (s)he has reached the best possible solution at the end of the solution process.”	RQ4

Table 3. Items to be answered after applying each method.

Purpose of questions	Questionnaire items	Answer type
RQ1	1) A lot of mental activity was required (e.g. thinking, deciding, and remembering).	Likert scale
	2) Finding the preferred solution was simple.	Likert scale
	3) I had to work hard to find my preferred solution.	Likert scale
	4) I felt frustrated in the solution process (e.g. insecure, discouraged, irritated, stressed).	Likert scale
RQ2	5) It took too many iterations to arrive to the acceptable solution.	Likert scale
	6) I felt tired.	Likert scale
RQ3	7) The preference information was easy to provide.	Likert scale
	8) I was able to express my preferences as I wanted.	Likert scale
	9) It was easy to learn to use this method.	Likert scale
RQ4	10) Why did you stop iterating?	Open-ended
	11) I am satisfied with the final solution.	Likert scale
	12) I think that the solution I found is the best one.	Likert scale
	13) What degree of conflict do you think exists among each pair of objectives? a) Among f1 and f2 b) Among f1 and f3 c) Among f2 and f3	Semantic differential: Very low (1)– Very high (5)

Table 4. Concluding items after all methods are applied.

Purpose of questions	Questionnaire items	Answer type
Comparative questions	1) Which method was the easiest one to use? Please describe why	Multiple-choice, Open-ended
	2) Which method allowed you to learn most about the problem? Please describe why	Multiple-choice, Open-ended
	3) Which method would you like to use again? Please describe why	Multiple-choice, Open-ended
	4) Which solution do you like most? Please describe why	Multiple-choice, Open-ended
Involvement of the participants as DMs	5) The problem was easy to understand. Please describe why	Likert scale, Open-ended
	6) The problem was important for me to solve. Please describe why	Likert scale, Open-ended

Figure 1. The UI for RPM.

Figure 2. The UI for NIMBUS.

Figure 3. Procedure of the experiment.

Table 5. Responses as average scores for measuring cognitive load.

Questionnaire items	Average scores (SD)
	RPM	NIMBUS
1) A lot of mental activity was required (e.g. thinking, deciding, and remembering).	3.81 (1.97)	4.88 (1.45)
2) Finding the preferred solution was simple.	4.44 (1.63)	4.00 (1.59)
3) I had to work hard to find my preferred solution.	4.31 (1.49)	4.50 (1.59)
4) I felt frustrated in the solution process (e.g. insecure, discouraged, irritated, stressed).	3.31 (1.78)	3.56 (1.86)
5) It took too many iterations to arrive at an acceptable solution.	3.94 (1.57)	4.19 (1.68)
6) I felt tired.	2.63 (1.41)	2.75 (1.53)

Table 6. Responses as average scores for capturing preferences.

Questionnaire items	Average scores (SD)
	RPM	NIMBUS
The preference information was easy to provide.	6.00 (1.03)	6.00 (1.03)
I was able to express my preferences as I wanted.	5.31 (1.14)	6.00 (1.03)
It was easy to learn to use this method.	6.19 (1.22)	5.88 (1.15)

Table 7. Responses as average scores for measuring satisfaction.

Questionnaire items	Average scores (SD)
	RPM	NIMBUS
I am satisfied with the final solution.	5.13 (1.09)	5.25 (1.00)
I think that the solution I found is the best one.	4.56 (1.15)	4.69 (1.35)
What degree of conflict do you think exists among each pair of objectives?:
Among f1 and f2	2.63 (1.09)	2.81 (1.22)
Among f1 and f3	2.44 (1.21)	2.13 (1.15)
Among f2 and f3	4.50 (0.73)	4.63 (0.81)

Table 8. Why did participants stop iterating?.

Why did you stop iterating?	NIMBUS (N = 16)	Example statements	RPM (N = 16)	Example statements
Compromise: no further improvements	n = 11	“I realised I could not achieve my overly optimistic results and stopped when found a decent compromise”.	n = 6	“I found the solution quite satisfactory for me and I am afraid the next solution is worse and I can’t go back”
Compromise: maintaining preferred objective	n = 4	“Was unable to further improve environmental objective without sacrificing the economic objective”.	n = 8	“I tried to improve the economic and environmental objectives and impair the social one but was not able to do it satisfactory so settled on this solution”.
Frustration	n = 0	–	n = 2	“The iteration felt like random generation that does not obey my reference point and I got frustrated”
Matching solution to previous method	n = 1	“I found a solution close to the solution I found with a previous method”	n = 0	–

Table 9. Results of the comparative items asked after completing both methods.

Questionnaire items	Number of participants (N = 16)
	RPM	NIMBUS
Which method was the easiest one to use?	9	7
Which method allowed you to learn most about the problem?	3	13
Which method would you like to use again?	3	13
Which solution do you like most?	6	10

Table 10. Responses as average scores on involvement levels of participants as DMs.

Questionnaire items	Average scores (SD)
The problem was easy to understand.	5.44 (1.41)
The problem was important for me to solve.	5.63 (0.96)

Afsar, B., Miettinen, K., & Ruiz, F. (2021b). Assessing the performance of interactive multiobjective optimization methods: A survey. ACM Computing Surveys, 54(4):85, 1–27. https://doi.org/10.1145/3448301

 Web of Science ®Google Scholar