Verifying the X for design framework capabilities in improving user experience evaluation activities

Figures & data

Figure 1. Adoption of the XfD

Figure 2. The irMMs-based method

Figure 3. Drawing of the socket to model (units are millimeters)

Figure 4. Excerpt of the user guide containing the instructions to generate the goals

Table 1. Excerpt of an irMM (first two columns) and its evaluation against the real interaction (last two columns)

User actions	Product reactions/feedback	Check	Reason and judgment
Open the menu to perform Boolean operations		Passed
	The menu to select the Boolean operations is shown	Passed
Select the command to subtract volumes		Failed	The menu command extrude is unclear. The evaluators show me that it also allows performing subtractions but its label seems to mean the opposite. Negative gap.
	The label of the selected operation is colored	Passed
Select the main prism		Passed
	The first prism is highlighted	Passed
Select the prism to subtract		Passed
	The second prism is highlighted	Passed
	The preview of the resulting volume is shown	Failed	I was expecting to see only the preview of the resulting volume. Nevertheless, in addition to that, the package shows also the starting prisms as shaded. This allows me verifying if the performed Boolean operation (cut) succeeded. Positive gap.
Confirm the operation		Passed
	The hole in the model is shown	Passed

Table 2. Examples of UX aspects coming from the adoption of the original and of the improved releases of the irMMs-based method

	Original release		Improved release
Interaction topic	Positive	Negative	Positive	Negative
Feature Extrusion	-	(1) This feature performs both volume addition and subtraction but they are opposite to each other; this confuses the users (2) When this feature is used to make subtractions, only the arrow indicating the addition appears (3) …	-	(1) The feature performs both volume addition and subtraction but they are opposite to each other; this confuses the users (2) When this feature is used to make subtractions, only the arrow indicating the addition appears (3) The feedback does not support at best this feature because the color of the volumes does not change according to the operator selected (addiction vs. subtraction) (4) …
Main toolbar	-	(1) The order of the buttons in the toolbar is wrong. For example, the sketch command is after create and modify, but sketch is usually used first (2) Drop-down menus of the toolbar show commands for 2D sketching and for generating 3D primitives as mixed (3) The icons in the drop-down menus of the toolbar are too small; it is hard to understand their meanings	(1) The icon symbols in the drop-down menus of the toolbar help the users to understand what the commands are for	(1) The order of the buttons in the toolbar is wrong. For example, the sketch command is after create and modify, but sketch is usually used first (2) Launching the software package, the drop-down menu of the toolbar to change the workspace is not intuitive (3) Drop-down menus of the toolbar show commands for 2D sketching and for generating 3D primitives as mixed (4) The toolbar is not context-sensitive; during the generation of a 2D sketch, the buttons about the generation or modification of 3D objects remain active (5) …
3D shapes generation	(1) Predefined 3D shapes of the create command are very useful to speed up the modeling activities	-	(1) Predefined 3D shapes of the create command, are very useful to speed up the modeling activities	-

Table 3. Quantitative results of the adoptions of the original and of the improved releases of the irMMs-based method, as well as of the TA method

	Original release
	AB users		RB users		RE users		SUM
	Positive	Negative	Positive	Negative	Positive	Negative	Positive	Negative
UX aspects (#; no doubles) (F,B,S)	4 (3, 1, 0)	35 (18, 14, 3)	5 (2, 2, 1)	25 (13, 9, 3)	9 (2, 3, 3)	22 (8, 9, 5)	16 (7, 6, 3)	68 (30, 28, 10)
Gaps (#; no doubles) (F,B+,S)	2 (2, 0, 0)	21 (12, 8, 1)	3 (2, 1, 0)	14 (5, 7, 2)	5 (2, 2, 1)	9 (2, 2, 5)	9 (5, 3, 1)	39 (19, 14, 6)
Desired meanings (#; no doubles)	5		10		8		14
Desired emotions (#; no doubles)	8		5		5		8
Changes of meanings (#)	0	3	0	1	2	4	2	8
Changes of emotions (#)	8	22	16	7	11	3	35	32
Time (hours)	16		12		8		36
	Improved release
	AB users		RB users		RE users		SUM
	Positive	Negative	Positive	Negative	Positive	Negative	Positive	Negative
UX aspects (#; no doubles) (F,B,S)	6 (3, 2, 1)	49 (20, 17, 12)	8 (4, 3, 1)	42 (19,14, 9)	14 (4, 5, 5)	31 (9, 10, 12)	24 (9, 9, 6)	88 (35, 28, 25)
Gaps (#; no doubles) (F,B+,S)	5 (2, 1, 2)	30 (14, 9, 7)	6 (2, 2, 2)	24 (9, 9, 6)	9 (3, 2, 4)	14 (4, 4, 6)	16 (6, 5, 5)	52 (21, 17, 14)
Desired meanings (#; no doubles)	4		8		7		13
Desired emotions (#; no doubles)	12		9		10		13
Changes of meanings (#)	0	1	1	3	1	2	2	6
Changes of emotions (#)	9	21	17	10	15	4	41	35
Time (hours)	17		14		9		40
	TA
	I group		II group		III group		SUM
	Positive	Negative	Positive	Negative	Positive	Negative	Positive	Negative
UX aspects (#; no doubles) (F,B,S)	5 (0, 2, 3)	22 (6, 6, 10)	5 (1, 2, 2)	24 (8, 5, 11)	8 (1, 2, 5)	18 (4, 5, 9)	13 (2, 3, 8)	60 (17, 14, 29)
Time (hours)	12		10		6		28

Appendix A. Symbols used in the XfD

Symbol	Meaning
F	Function. This variable describes the aim of the product, i.e., what the product is for, what it allows performing
B	Behavior. This variable describes the attributes derived or expected from the structure of the product, i.e., what the product does
S	Structure. This variable describes product components in terms of shape, dimensions and material, and their relationships, i.e., what the product is
N	Need. This variable expresses perceived desirable situations to be attained or undesirable situations to be avoided, explicitly stated by the designers or perceived by observing the real user
A	Aesthetics. This variable describes the appearance of the product, how it is perceived by the user through his/her senses
M	Meaning. This variable represents the symbolic significance of a product derived from the previous experiences of the user
E	Emotion. This variable describes the affective evaluation of the product
hB	Human behavior. This variable represents what the user does with the product
B+	This variable represents human and product behaviors together
variable^e	The variable belongs to the external world; it is external to the user mind and it exists independently from him/her
variable^i	The variable belongs to the interpreted world; the user generated a representation of it in his/her mind
variablee^i	The variable belongs to the expected world; the user elaborated the interpreted variable and generated an expected representation of it respect to his/her goals
⊆	Push-pull class. It represents the interaction of an agent with its external environment (the external world), push, as well as that with its internal environment (the interpreted world), pull. The pull class also represents the influence of a variable on a different one
⇔	Focussing class. It points out some aspects of the interpreted world and these aspects are used as goals in the expected world
→	Action class. It represents an effect, which brings about changes in the external world according to the goals in the expected world. This action class also represents the transformation inside the same world of a variable into another one
↔	Comparison class. It represents the comparison between expected and interpreted variables

Appendix B. XfD model of the irMMs-based method

Phase (agent)	Activities	Translation	Reason	Refinement	Meaning
Phase 1. Input setting (evaluators)	Selection of the product features to evaluate	-	-	-	-
	Selection of the users who undergo the tests	-	-	-	-
	Selection of the evaluators	-	-	-	-
Phase 2. Material and environment setup (evaluators)	Customization of the user guides	-	-	-	-
	Customization of the documents for the evaluators	-	-	-	-
	Selection and setup of the environment	-	-	-	-
Phase 3. Text execution (users)	Free interaction with the product	S^e⊆S^i	The user interprets the product focusing on its F, B and S characteristics to highlight the components of the irMM (M, E, B and hB)
		S^i⊆
		B^e⊆B^i
		F^e⊆F^i
				S^i→B^i	Generation of new behaviors from structures and of new functions from behaviors ('a')
		B^i⊆
				B^i→F^i
		F^i⊆
		F^i→M^i
		F^i→E^i
		B^i→M^i
		B^i→E^i
				S^i→A^i	Generation of new meanings and emotions due to the influence of structures and aesthetics ('b')
		S^i→M^i
		S^i→E^i
				A^i⊆
				A^i⊆M^i
		M^i⊆
				A^i+M^i⊆E^i
		E^i⊆
		M^i→hB^i
		E^i→hB^i
	Need perception and interpretation	N^e⊆N^i	In order to consider the assigned need as a real need, the user must elaborate it in his/her mind
		N^i⊆
	Recovering and selection of existing irMMs	N^i→M^i	Thanks to the need interpretation, the user can recover all the elements of past irMMs (M, E, B and hB) that satisfied similar needs
		N^i→E^i
		N^i→hB^i
		N^i→B^i
	Setting the goals	N^i→F^i	Since the goals are intermediate results coming from the execution of specific product functions, the need are transformed into interpreted functions and these interpreted functions are focussed onto expected functions afterwards
		F^i⊆
		F^i⇔Fe^i
	Assignment of desired meanings and emotions	Fe^i→Me^i	In order to achieve the desired meanings and emotions and to write them in the user guides, the meanings and emotions deriving from the interpretation of the goals are compared to those coming from past irMMs
		Fe^i→Ee^i
				M^i⊆E^i	Generation of new emotions due to the influence of meanings ('c')
		M^i↔Me^i
		E^i↔Ee^i
		Me^i→M^e
		Ee^i→E^e
	Definition of the best user and product behaviors	M^e⊆M^i	In order to define the best user and product behaviors to achieve the desired meanings and emotions, these meanings and emotions are elaborated and externalized in the user guides
		M^i⊆
		E^e⊆E^i
				M^i⊆E^i	Generation of new emotions due to the influence of meanings ('c')
		E^i⊆
		M^i→hB^i
		M^i→B^i
		E^i→hB^i
		E^i→B^i
				hB^i⊆B^i	Generation of new product behaviors due to the influence of user behaviors ('d')
		hB^i⇔hBe^i
		B^i⇔Be^i
		hBe^i→hB^e
		Be^i→B^e
	Interaction with the real product	hB^e⊆hB^i	In order to identify possible gaps, the user must evaluate the real product behaviors and his/her own real behaviors allowed by the product respect to the expected ones described in the irMM. Then, the user must report possible gaps in the user guides
		B^e⊆B^i
				S^e⊆S^i	Evaluation of the product behaviors by considering its structures ('e')
				S^i⊆
				S^i→B^i
		hB^i⊆
		B^i⊆
		hB^i↔hBe^i
		B^i↔Be^i
		hBe^i→hB^e
		Be^i→B^e
			In order to give a final evaluation of meanings and emotions, the real ones are compared with the desired one and the user guides report the results of this comparison	S^e⊆S^i	Generation of new meanings due to the influence of structures and aesthetics ('f')
				S^i⊆
				S^i→A^i
				A^i⊆
				S^i→M^i
				A^i⊆M^i
		hB^i→M^i
		B^i→M^i
		M^i⊆
				S^i→E^i	Generation of new emotions due to the influence of structures and aesthetics ('g')
		B^i→E^i
		hB^i→E^i
				A^i+M^i⊆E^i
		E^i⊆
		M^i↔Me^i
		E^i↔Ee^i
		Me^i→M^e
		Ee^i→E^e
Phase 4.	Data interpretation	M^e⊆M^i	To allow the elaboration of the data contained in the user guides, the evaluators must interpret all of them (M, E, B and hB)
		E^e⊆E^i
		M^i⊆
				M^i⊆E^i	Generation of new emotions due to the influence of meanings ('c')
		E^i⊆
		hB^e⊆hB^i
		B^e⊆B^i
		hB^i⊆
		B^i⊆
Data analysis (evaluators)	Data elaboration	M^i→F^i	To allow the elaboration, the interpreted data must be transformed into those functions, behaviors and structures that represent the positive and negative UX aspects referred to that product
		M^i→B^i
				M^i→S^i	Generation of new structures from meanings, emotions and user behaviors ('h')
		E^i→F^i
		E^i→B^i
				E^i→S^i
		hB^i→F^i
		hB^i→B^i
				hB^i→S^i
		B^i→F^i
		B^i→S^i
		F^i⇔Fe^i
		B^i⇔Be^i
		S^i⇔Se^i
	Documentation	Fe^i→F^e	In order to make the previous positive and negative UX aspects available and readable by everybody, the UX aspects must be written in the final report containing the results of the evaluation
		Be^i→B^e
		Se^i→S^e

Appendix C. Implementation of the activities suggested by the refinement

Suggested activity	Sub-processes	Implementation
Generation of new behaviors from structures and of new functions from behaviors (“a”)	S^i→B^i B^i→F^i	Addition of the new instruction in the part of the RB user guide regarding the free interaction “during your free interaction, please pay attention also to the product behaviors that its components (shape, sounds, etc.) suggest to you, as well as to the functions suggested by these behaviors”
Generation of new meanings and emotions due to the influence of structures and aesthetics (“b”)	S^i→A^i A^i⊆ A^i⊆M^i A^i+M^i⊆E^i	Addition of the new instruction in the part of the RB user guide regarding the free interaction “during the interaction with the product, please pay attention to the meanings that this experience suggests to you as well as to the emotions that it arouses. Please consider the functions you perform, the behaviors of the product, the icons, labels, buttons and all the other elements of the product interface”
Generation of new emotions due to the influence of meanings (“c”)	M^i⊆E^i	Addition of the new instruction to the user guides during the assignment of the desired meanings and emotions “before to write down your desired emotions, think about the meanings that the achievement of the specific goal arouses and consider possible new emotions”
		Addition of the new instruction to the user guides during the definition of the best user and product behaviors “when you think about your behaviors and about the expected product behaviors, please pay also attention to the emotions aroused by the desired meanings”
		Addition of the new instruction to the documents for the evaluators during the data interpretation “during the analysis of users’ emotions, please consider also users’ meanings, together with users’ judgments on both of them. Thanks to these judgments, new emotions could be generated”
Generation of new product behaviors due to the influence of user behaviors (“d”)	hB^i⊆B^i	Addition of the new instruction in the user guides to help thinking about product behaviors correctly during the definition of the best user and product behaviors “when you write down the expected product behaviors, please think about the best behaviors you expect from it in order to perform your behaviors as easily and naturally as possible”
Evaluation of the product behaviors by considering its structures (“e”)	S^e⊆S^i S^i⊆ S^i→B^i	Addition of the new instruction in the user guides before starting the interaction with the product “you are going to start the interaction with the product. When you will analyze the differences between the expected product behaviors and the real ones, please think also about the components, shapes, colors, materials, etc., that could help you in judging the positivity or negativity of these differences”
Generation of new meanings due to the influence of structures and aesthetics (“f”)	S^e⊆S^i S^i⊆ S^i→A^i A^i⊆ S^i→M^i A^i⊆M^i	Addition of the new instruction in the user guides after the evaluation of user and product behaviors and before that of the desired meanings and emotions “your interaction with the real product has just finished. In analyzing the differences between the desired meanings and emotions and the real ones, please think also about the components, shapes, colors, materials, together with aesthetics, style, etc., that can help you in judging these differences”
Generation of new emotions due to the influence of structures and aesthetics (“g”)	S^i→E^i A^i+M^i⊆E^i
Generation of new structures from meanings, emotions and user behaviors (“h”)	M^i→S^i E^i→S^i hB^i→S^i	Addition of the new instruction in the documents for the evaluators to help them in analyzing meanings, emotions and user behaviors “during your data analysis, please consider the influence of meanings, emotions and user behaviors on the aesthetics and on the structures of the product; they can suggest new UX aspects that the consideration of the sole product behaviors could miss”