Evaluating usability, functionality, and usefulness of the warehousing error prevention tool

Abstract

This paper presents an evaluation of the usability, functionality, and usefulness of the Warehouse Error Prevention (WEP) tool that consists of seven modules. The WEP tool is framed in a simple yes/no form, which can be used to identify human factors related to sources of pick errors in a warehouse. Thirty-three participants in 27 organisations from three different countries participated in a trial application and evaluation of the tool. The evaluation included a survey study and semi-structured interviews. Survey results show that participants agreed on the usability and functionality of the WEP tool. In the interviews, participants generally reported the WEP tool as being both accurate and functional with the potential to support engineers, ergonomists, and warehouse managers to improve order picking quality. Further quantitative field testing of the WEP tool’s potential to identify costly warehouse errors is needed.

KEYWORDS:

• Order picking-human factors
• errors
• quality
• user evaluation
• statistical analysis

1. Introduction

1.1. Order picking

Order Picking (OP) is the process of retrieving items in a warehouse to satisfy customer orders (Grosse et al. 2015). There are many design aspects that can significantly impact the efficiency and effectiveness of this process (Vanheusden et al. 2023). This article presents an end-user study evaluating the usability, functionality, and usefulness of the Warehouse Error Prevention (WEP) tool (freely available from Setayesh et al. 2022a).

Key OP system design elements include: (a) palletizing/batching decisions, which determine how requested stock-keeping units (SKUs) are grouped to facilitate retrieving them (Calzavara et al. 2017); (b) pick information provision about item numbers, item locations, and requested quantities (e.g. on a pick-list or using information technologies, Grosse, Glock, and Neumann 2017); (c) layout, which determines the shelf configuration of the warehouse (Roodbergen, Vis, and Taylor 2015); (d) routing, which determines the sequence in which item locations are visited (Masae, Glock, and Vichitkunakorn 2020); (e) storage assignment, which allocates items to storage locations (Battini et al. 2015); and (f) organisational behaviour management, which focuses on the design of work procedures and work organisation (Lattal 2014). Although automation and robotisation have become an essential part of many warehouse operations (Fragapane et al. 2021), about 80% of the OP processes are still performed manually due to the superior flexibility of humans to accommodate unexpected changes in the system (Vijayakumar et al. 2022; Zhao et al. 2019; De Koster, Le-Duc, and Roodbergen 2007). The use of assistive devices has been reported to increase though (Winkelhaus, Grosse, and Morana 2021; Grosse 2023). As a result, manual OP systems tend to be labour-intensive and time-consuming (Zhao et al. 2019), which also creates the potential for pick errors (Setayesh et al. 2022b).

1.2. Human and system design interaction in manual order picking

Pick errors may occur as a result of poor human and OP system design interactions, and they can materialise as a selection of incorrect items, a selection of a wrong item quantity, or a selection of damaged items (Setayesh et al. 2022b). Pick errors may cause increases in operational costs, customer dissatisfaction, and unexpected penalty costs while degrading system performance. The high amount of manual labour makes the OP process the most expensive operation in a warehouse, which can account for more than 50% of the warehouse’s expenses (Tompkins et al. 2010; Frazelle 2015). A high number of pick errors can further increase these costs.

Pick errors may result from a problematic system design leading to mismatches between human capability and demands in OP operations (Setayesh et al. 2022b; Grosse, Glock, and Neumann 2017). Rasmussen (1981) and Rasmussen and Rouse (2013) underscored the relationship between problematic system design and human-system error. Lack of attention to human and OP system interactions in the design process can contribute to pick errors (Setayesh et al. 2022b; Grosse et al. 2015). This suggests that the application of Human Factors (HF) considerations in the design of OP systems can improve operator-system interactions and reduce pick errors. HF is a multidisciplinary scientific approach considering both psychological and physiological principles and aims to optimise system performance and human well-being by understanding system design elements and human–system interactions (IEA Council 2014). Since the roots of errors often lie in the system design, errors in system operations should be considered human-system errors rather than the more common ‘human error’ (Salmon, Walker, and Stanton 2016; Di Pasquale, Miranda, and Neumann 2020).

Although humans are key to OP operations, HF aspects in OP research appear to be an understudied topic (Grosse, Glock, and Neumann 2017; De Lombaert et al. 2022). HF consists of four aspects that determine the human–system design interaction that affects humans’ well-being and overall performance (Vijayakumar et al. 2022; International Ergonomics Association 2014). These four aspects are: (a) Physical demands, including physical activities such as repetitiveness of tasks, carrying of loads, hand and arm postures, and strained hand and arm movements (Coenen et al. 2014); (b) cognitive demands, comprising of mental workload and mental fatigue; (c) perceptual demands, involving sensory systems such as vision, hearing, smell, and touch; and (d) psychosocial demands, referring to an individual’s psychological perceptions of the social dimensions of the workplace such as motivation, job satisfaction, and supervisory support (Grosse et al. 2015; Salvendy 2012). If the design of the system places demands on human operators that exceed their capabilities, then negative outcomes, such as errors or injuries, can be expected. These human-system mismatches incur unexpected costs for the system – a phenomenon referred to as the ‘innovation pitfall’ (Neumann et al. 2021).

1.3. Warehouse Error Prevention (WEP) tool design strategy

The Warehouse Error Prevention (WEP) tool is designed to help users identify sources of errors in OP systems to help improve quality performance. It has been developed based on recent research that identified a wide range of different HF considerations in warehouse design that contribute to pick errors including visual demands, mental workload, physical workload, and psychosocial aspects (Setayesh et al. 2022b). The WEP tool contains seven modules and 118 questions and sub-questions based on specific OP system design and operational aspects which can help warehouse engineers and managers to identify specific opportunities to reduce human-system error risks. The first five modules focus on specific OP system design elements: (1) pick information/technology (Grosse 2023), (2) layout, (3) routing, (4) palletizing/batching, and (5) storage assignment. Modules six and seven cover organisational behaviour management aspects, which are organisational level elements that have been shown to have an impact on order pickers’ performance and pick error occurrence (Setayesh et al. 2022b; Grosse et al. 2015). The WEP tool is available online (Setayesh et al. 2022a).

For each module, specific design elements were identified from a previous analysis of error sources in OP systems (Setayesh et al. 2022b). A specific verifiable design requirement was then identified for each element by reviewing scientific literature, existing guidelines, and HF handbooks. Table 1 presents key references used to define the HF demand threshold values and OP design elements in the WEP tool.

Table 1. Key references and types of HF demand threshold values and OP design elements addressed that were used to define the design requirements in creation of the WEP tool.

Authors	OP design element	Module # in the WEP tool	HF demands
Weisner and Deuse (2014)	Layout	Module 1	Cognitive
Manzini (2012)			Physical
Jacobs (2004)	Pick information/technology	Module 2	Cognitive
De Vries, De Koster, and Stam (2016)			Physical
Guo et al. (2014)			Perceptual
Bindi et al. (2009)	Storage assignment	Module 3	Cognitive
Chuang, Lee, and Lai (2012)			Physical
Zhang (2016)
Lind, Forsman, and Rose (2019)	Palletizing/batching	Module 4	Cognitive
Lind, Forsman, and Rose (2020)			Physical
Abe, Muraki, and Yasukouchi (2008)
McAtamney and Corlett (1993)
Bódis, Botzheim, and Földesi (2018)	Routing	Module 5	Physical
Elbert et al. (2017)			Mental
Chen et al. (2015)	Organisational behaviour management	Module 6	Psychosocial
Kampkötter, Harbring, and Sliwka (2018)
Häusser et al. (2010)

Based on the identified design requirements in the OP context, each element was broken down into specific binary yes/no criteria that formed the basis of each item rather than using multiple level scale or measurement approaches. The binary approach makes the decision-making process easier for users as it provides a fixed and consistent list of responses based on a single specific criterion. While a ‘multiple options’ or scoring approach was considered, the developers were concerned with the vagueness and complexity that this might cause, thereby compromising tool usability and functionality.

In the WEP tool, any ‘No’ response indicates a specific opportunity to improve that OP system design element according to the criteria provided. In this way, each item in the WEP tool is a kind of decision support tool to identify where further investigation and design changes might be needed. Figure 1 shows an example of the WEP tool from the palletizing/batching module. The WEP tool has never been evaluated with regards to potential users’ views of usability, functionality, and usefulness – key features to evaluate for any production engineering tool.

Figure 1. A section of the WEP tool showing a question associated with the physical weight of items in a palletizing/batching process.

Shows a section of the developed WEP tool which, in this study, has been evaluated by potential end-users.

1.4. Research aim and questions

The WEP tool represents a boundary spanning object (Ojansivu, Kettunen, and Alajoutsijärvi 2021; Broberg, Andersen, and Seim 2011) – in this case spanning the gap between researchers and practitioners – that mobilises research knowledge on the design-related sources of errors in OP systems (Setayesh et al. 2022b) in an actionable and practical way around the source of errors in manual OP operations. Having designed and built the tool based on scientific evidence (Setayesh et al. 2022a), it is important to test it with the target user groups ensuring their needs have been adequately met.

The purpose of this study is to assess the usability, functionality, and usefulness of the developed WEP tool from the perspective of potential end-users in qualitative and quantitative terms. Usability determines how easily and well a user can use a system to obtain the defined goal (Gulliksen, Boivie, and Göransson 2006). Functionality determines a system’s capability and accuracy to support making effective and practical decisions (Laumer et al. 2019). Usefulness determines a system’s capability to provide useful indicators to help users achieve higher system performance (Wright, Paroutis, and Blettner 2013). A tool with good usability, functionality, and usefulness can support users without requiring specific expertise in a particular area. These key dimensions are evaluated in the context of the WEP tool as part of the development of sociotechnical systems. The specific research questions addressed in this article are as follows:

RQ1 – What are the views of potential users, when surveyed quantitatively, on the following features of the WEP tool?

  RQ1-a. What are potential users’ views on the usability of the WEP tool?

  RQ1-b. What are potential users’ views on the functionality of the WEP tool?

  RQ1-c. What are potential users’ views on the usefulness of the WEP tool?

RQ2 – How do participants perceive the usability, functionality, and usefulness of the WEP tool in qualitative terms?

The remainder of this paper is organised as follows: Section 2 presents the survey study and interview results. Section 3 discusses the results of both approaches and highlights possible implications for both research and practice. Section 4 concludes the paper.

2. Fundamentals of the research techniques

2.1. Methodology for user evaluation study

This study used mixed (quantitative and qualitative) methods in evaluating the WEP tool to complement the limitations of qualitative data collection with the strengths of quantitative techniques (Greene, Caracelli, and Graham 1989). The quantitative data measures the participants’ views of the tool’s usability, functionality, and usefulness. The qualitative data adds contextual information to explain complex issues, and it complements the quantitative data with insights on the participants’ thinking that influences their quantitative ratings (Grosse et al. 2016). This study was approved by the institutional ethics review board prior to the data collection.

Thirty-three participants from different industry sectors (manufacturing, production, fresh products, retail) were recruited for data collection and engaged in both the survey and the semi-structured interviews. Participant recruitment was initiated with an e-mail invitation that was sent to warehouse designers, facility designers, and warehouse supervisors that are familiar with the design of warehousing and manual OP operations in the sectors mentioned above. The recruiting process was conducted in Canada, the United Kingdom, and Iran providing different industrial contexts for the study. The WEP tool and user manual were provided to the recruited participants in advance. In each data collection event, the researcher worked with participants as they used the WEP tool to assess their operations and provided help as required. Both survey and semi-structured interview data collection were stopped when data saturation occurred and no additional insights were provided by the participants (Fusch and Ness 2015), suggesting an appropriate sample size had been achieved.

2.2. Survey study data collection

After using the WEP tool, participants were asked to complete the 10-question System Usability Scale (SUS) survey (Brooke 1986; see Table 2) to assess usability and functionality aspects. The SUS technique is a standardised, reliable instrument for measuring various system and product usability and functionality aspects from the user’s perspective (Brooke 2013; Bangor, Philip, and James 2009). Each question in the SUS questionnaire uses a 5-point Likert scale format from 1 (strongly disagree) to 5 (strongly agree). Table 2 shows the 10 SUS statements. Usability can be assessed by questions # 1, 2, 3, 7, 8, and 10 of the SUS survey. The functionality of the WEP tool can be assessed through questions # 4, 5, 6, and 9. The overall SUS score can be calculated by first subtracting five from the total score of odd-numbered questions and subtracting twenty-five from the total score of even-numbered questions. The outcome of these should be added up and multiplied by 2.5 (Brooke 1986). The usefulness of the WEP tool is examined by complementary survey questions (Table 3).

Table 2. SUS statements (Brooke 2013) were measured using a 5-point Likert scale in 10 questions.

Q#	SUS statements	Usability/functionality
1	I would like to use the tool	Usability
2	I found the tool unnecessarily complex	Usability
3	I found the tool was easy to use	Usability
4	I would need the help of a technical support	Functionality
5	I found various functions in the tool well integrated	Functionality
6	I thought there was too much inconsistency in the tool	Functionality
7	I would imagine most people would learn to use the tool very quickly	Usability
8	I found the tool very cumbersome to use	Usability
9	I felt very confident using the tool	Functionality
10	I needed to learn lots of things before I was able to use the tool	Usability

Table 3. Supplementary questions used to assess participants’ opinions using 5-point Likert scale (3 questions).

Q#	Section three of survey statements
11	I found the WEP tool helpful to consider human factors in warehouse design
12	I found the WEP tool helpful for OP system design improvements
13	I found the WEP tool applicable and useful for engineers and designers to satisfy OP operator’s needs with respect to HF

In the last section of the survey study, three supplementary questions, shown in Table 3, were asked to capture the participants’ opinions on the applicability and usefulness of the WEP tool. These questions also determine whether participants would like to use the WEP tool for their human-system design assessment.

2.3. Quantitative data analysis

Table 4 summarises the used methods in this study. The Cronbach’s alpha was calculated for each variable to check the internal consistency of the survey instrument (Cronbach 1951; Tavakol and Dennick 2011; Vaske, Beaman, and Sponarski 2017; Bujang, Omar, and Baharum 2018).

Table 4. Techniques used for conducting the user evaluation study.

Qualitative data collection technique	Evaluation goal	Data form	Analysis technique
Survey study	Usability	5-point Likert scale SUS* Questionnaire	One sample Chi-square
	Functionality
	Usefulness	5-point Likert Questionnaire	One sample Chi-square
Semi-structured interview	Usability	Transcribed interview texts	Inductive thematic analysis
	Functionality
	Usefulness

*System Usability Scale (SUS) survey (Brooke 1986).

Since the responses are collected based on a distinct (ordinal) outcome variable, a non-parametric one-sample Chi-Square test was used to compare the distribution of participant responses to a chance distribution. A 5% level of significance was used to evaluate the statistical significance (Hamooya et al. 2016; Tanveer et al. 2021) with the null hypothesis being that the responses are distributed according to chance. The outcome of the survey analysis is used to answer RQ2.

2.4. Qualitative data collection – semi-structured interview

A qualitative, semi-structured interview was performed to supplement the questionnaire for each participant and provide a better understanding of the participants’ opinions on the usefulness, functionality, and usability of the WEP tool (Grosse et al. 2016). The interview helped to understand the WEP tool from the user’s standpoint as it allowed the interviewees to describe the logic and opinions behind their responses in their own words (Fontana and Fery 2000). The interview questions asked the interviewees to determine: (a) the parts of the WEP tool that they liked most or least, (b) the participant’s opinions on any design elements or error sources that were not included in the WEP tool, and (c) any recommendations or comments that may help improve the WEP tool. Probe questions were used when needed to clarify the participants’ responses. The interview sessions lasted about 45 min and were audio-recorded and transcribed for later analyses to identify key themes (Armat et al. 2018; Elo and Kyngäs 2008).

2.5. Qualitative data analysis

The interview data were coded in NVivo and analysed by using the thematic approach according to the methodology advocated by Braun and Clarke (2006) to identify common themes in participants’ responses. In this study, data was reviewed thematically with an inductive approach to see what common issues/themes emerged (Elo and Kyngäs 2008). The defined RQs supported the themes which were defined deductively based on user evaluation criteria (per Table 3 and Table 4). To check the reliability of the coding scheme, two independent raters counted the occurrence of the defined themes on 20% of the data. The Interclass Correlation Coefficient (ICC) was generated between the two independent raters to check the agreement between the two raters (Nowell et al. 2017). Based on the two independent raters, an ICC was calculated. The ICC > 0.64 for all the defined themes indicated a strong agreement between the two raters (McHugh 2012, 2013). The remaining 80% of the data were coded by a single rater.

3. Results

The results from both the SUS survey and the interviews from the 33 recruited participants from different industry sectors and three different countries are presented below.

3.1. Results from the SUS survey technique

The Cronbach’s Alpha for the SUS survey outcome is equal to 0.812 indicating high reliability of the collected data (cf. Cronbach 1951). The summary of responses to the SUS survey is presented in Table 5.

Table 5. Descriptive statistics for SUS survey results (5-point Likert scale) for all participants.

No.	SUS statements	Median	SD	Chi Sq.	Mode value (Corresponding responses)	p-value
1	I would like to use the tool	4	1.10	18.97	Agree	0.001
2	I found the tool unnecessarily complex	1	0.61	12.18	Disagree	0.002
3	I found the tool was easy to use	4	1.15	24.72	Agree	0.0001
4	I would need technical support	2	0.67	7.09	Disagree	0.029
5	I found various functions in the tool well integrated	4	1.22	13.51	Agree	0.009
6	I thought there was too much inconsistency in the tool	1	0.61	14.72	Disagree	0.001
7	I would imagine most people would learn to use the tool very quickly	4	1.11	19.87	Agree	0.001
8	I found the tool very cumbersome to use	2	1.75	3.45	Disagree	0.178
9	I felt very confident using the tool	4	1.09	21.69	Agree	<0.001
10	I needed to learn lots of things before I was able to use the tool	1	0.71	16.90	Disagree	<0.001

The results from the Chi-square test for the SUS responses rejected the null hypothesis (that the responses were a chance distribution) for 9 out of 10 questions (Table 5). This outcome indicates that participants generally agreed on their responses for these 9 items. Participants were inconsistent on question # 8, which evaluates how ‘cumbersome’ the participant found the WEP tool. SUS scores greater than 71.4 indicate ‘good’ WEP tool usability (Bangor, Philip, and James 2009). In this study, the SUS score from participants (mean SUS score = 81.06, SD = 16.75) indicates that participants found the WEP tool useable.

The participants’ opinions on the usability of the WEP tool can be supported through questions #2 (median = 1, p-value = 0.002) and #8 (median = 2, p-value = 0.178) in Table 5. The mode value responses indicate that the WEP tool was seen to be neither complex (question #2) nor cumbersome (question #8) to use. Although question #8 showed inconsistency between participants’ responses (p-value = 0.178), this inconsistency is generated as four participants’ responses to question #8 were neutral, while the same four participants selected ‘disagree’ to question #2 (tool complexity), indicating the WEP tool is unnecessarily complex, and suggesting mixed views on this point. For the functionality assessment, as seen in questions #5 (tool integration) and #6 (tool inconsistency) of Table 5, participants found the WEP tool generally functional. Participants expressed their opinions in their ratings of the functions in the WEP tool as well integrated (median = 4, p-value  = 0.009) and that the WEP tool was seen to be consistent (median = 1, p-value = 0.001) in each different module.

The data analysis on the complementary questions (Table 6) showed that participants generally ‘strongly agree’ on their responses to these three questions. The statistical data analysis indicates a significant agreement amongst the participants’ responses on the WEP tool usefulness with p < 0.05, as shown in Table 6. As seen in question #11, participants ‘strongly agreed’ on the helpfulness of the WEP tool for novice engineers with less field, ergonomics, and design experience (median = 4, p-value = 0.002). Question #12 revealed that participants also ‘strongly agree’ that the WEP tool is helpful for OP system improvement (median = 4, p-value = 0.004). Participants also ‘strongly agree’ that the WEP tool helps engineers and designers meet the operators’ HF needs in warehouse and OP system design (median = 4, p-value = 0.001).

Table 6. Descriptive statistics for significant Chi-square results for the usefulness of the WEP tool based on a participant’s opinion.

No.	Section three of survey statements	Usefulness	Median	SD	Chi Sq.	Mode value (corresponding responses)	p-Value
11	Helpfulness of the tool for novice designers	Yes	4	1.22	17.15	Strongly Agree	0.002
12	Helpfulness of the tool for OP system improvement	Yes	4	1.29	15.33	Strongly Agree	0.004
13	Helpfulness of the tool for engineers and designers to meet OP operators’ needs with respect to human factors	Yes	4	1.06	19.58	Strongly Agree	0.001

3.2. Themes emerging from the analysis of the interviews

Table 7 provides a list of themes that emerged from the interviews and their corresponding definitions along with example modifiers from participant statements. The ‘+’ and ‘-’ modifiers in the ‘examples and modifiers’ column of Table 7 indicate positive and negative comments within the theme, with the exception being in the ‘recommendation/feedback for future improvements’ theme, where the information addresses the participants’ suggestions/recommendations. The themes from the inductive analysis in Table 7 are matched with survey criteria to cluster the findings with at least one survey criterion. For example, tool interaction, tool issues, and match/mismatch between tool and user manual themes address the WEP tool usability through the survey conducted in this study. The frequency distribution of participants’ comments for the defined themes associated with positive and negative modifiers is displayed in Figure 2.

Figure 2. Frequency distribution of positive and negative comments from participant interviews for the defined themes.

Figure 2 shows the frequency of participants’ positive and negative responses to the interview questions based on the defined themes. Note that participants’ comments on tool issues had implications for both usability and functionality themes.

Table 7. List of themes and their corresponding definitions used in the thematic analysis.

Themes (inductive)	Survey criteria (deductive)	Description	Examples and modifiers
Tool Interaction	Usability	The usability of the tool and the individual qualitative experience with the general use of the tool and user manual is suitable to support engineers/designers during the design process.	+: ‘It’s easy to understand and follow the tool’. -: ‘The tool is unnecessarily long, complex, and time-consuming’.
Match/Mismatch Between Tool and User Manual	Usability	The state of the tool and user manual is cohesive and aligned.	+: ‘The questions and sub-questions in the tool, note section, and user manual are aligned’. -: ‘The tool and user manual explanations are not consistent’.
Tool issues	Usability Functionality	The applicability of the tool is based on the individual capability and job responsibilities that drive the quality outcome. The practicality of the tool can be assessed based on the comments on applying the tool in practice.	+: ‘The WEP tool is a multi-disciplinary tool’. -: ‘Module 4 of the tool mainly focused on HF which is not part of my job description’.
Tool Validity	Functionality	The functionality of the tool and user manual is suitable to support engineers/designers during the design process to ensure creating flawless system operations.	+: ‘Very accurate tool. It covers all the required design elements’. -: ‘The tool has not considered various design aspects’.
Recommendation/Feedback for Future Improvements	Usefulness	A suggestion or proposal as to the best course of action, especially one put forward by an authoritative body.	‘I would prefer to use the application/web-based software to be shareable with other departments’.

3.2.1. Usability

The usability of the WEP tool is identified by participants’ responses, which were classified under ‘user interaction’, ‘tool issues’, and ‘match/mismatch between tool and user manual’ themes per Table 7 and Figure 2. In the interview sessions, 29 out of 33 participants stated that the WEP tool was easy to use and requires minimum technical knowledge or support as different aspects of designs were addressed through straightforward questions and sub-questions. One participant commented, for example: ‘The tool is classified wisely … makes it very simple and straightforward’. Although the majority of the participants found the WEP tool usable, four participants made negative statements about the usability of the WEP tool. One participant described it as ‘long and time-consuming’.

3.2.2. Functionality

The functionality of the WEP tool is identified by the participants’ responses (Table 5) that were classified based on ‘tool validity’ and ‘tool issues’ themes per Table 7 and Figure 2. Twenty-two out of 33 participants expressed their satisfaction with the functionality of the WEP tool. The majority of the participants found the WEP modules to be very accurate and detail-oriented as different aspects of the OP system design were addressed through questions and sub-questions. One participant commented, for example: ‘The tool and user manual are very accurate and elaborate all the design elements precisely  …  The palletizing/batching model even indicates the range of appropriate movements to prevent physical fatigue’. Despite the majority of positive responses, eleven participants raised a few concerns about the functionality of the WEP tool. Three out of eleven of these participants mentioned the important role of assistive information technologies in their warehouse by addressing the limitation of the WEP tool in covering those technologies. One participant commented, for example: ‘We are using assistive pick technologies such as gloves and smart glasses … the tool has not addressed any requirements in selecting pick technologies to become efficient’. Four out of eleven concerns raised were related to the lack of functionality of the WEP tool in considering optimisation and simulation modelling, such as optimised storage assignment and routing strategies. A few participants evaluated the paper checklist style as a limitation of the WEP tool. They expressed a desire to have the WEP tool embedded in their main IT system. One participant explained:

The WEP tool works pretty well on Microsoft Office Word, but we are using assistive warehousing software management tools such as SAP … having a tool in program or application-based to make it more practical … It could be useful for a wider range of companies, especially those having international warehouse locations.

A few participants criticised the lack of functionality of the WEP tool in quantifying error risk levels. They expressed a desire that the WEP tool should measure the error risk levels to provide the opportunity to take corrective actions on those having the highest risk level with an extensive impact on worker and system performance. One participant explained:

The WEP tool provides a summary of the human-system influencing factors that have not been considered in the design. However, it does not quantify the associated risk level which makes the decision-making on prioritizing the design and/or process improvements challenging.

3.2.3. Usefulness

The usefulness of the WEP tool is identified by the participants’ responses that were classified under the ‘recommendation/feedback for future improvements’ theme per Table 7 and Figure 2. Participants described the WEP tool as very useful to not only design an effective warehouse facility, but also for process improvements and training purposes. Participants found the WEP tool very useful for addressing the need of novice engineers to address HF aspects in design. They also indicated that the WEP tool is useful for both warehouse facility design as well as human-system design improvements. One participant commented, for example:

It can assist engineers during the lean Six Sigma processes to identify the design flaws … module 4 also can help ergonomics improvement as determining the low-risk movements.

There were also a few negative instances with respect to the usefulness of the WEP tool. A few designers and engineers criticised the focus of the WEP tool on HF rather than the design process. One example comment was: ‘ … HF and ergonomics is not part of my department, and it is not my expertise … My department is only responsible for the optimized design process’. Two more senior participants also found the WEP tool not useful, and one engineer explained: ‘The tool can be very useful for recent graduate engineers, for me with more than fifteen years of experience, it is not useful as I am already familiar with all the requirements’.

4. Discussion

Evaluation of a novel tool with potential users is unusual in production research. The interview results supported the quantitative survey study outcomes, but also helped to identify a number of issues that could be used to improve the WEP tool. The participants’ positive comments indicated that they saw the WEP tool as a suitable and practical method for supporting designers, engineers and managers to prevent the source of errors in their OP systems. This section first discusses the acceptance of the WEP tool by end-users (sub-section 4.1), and then it points out implications for managers and engineers (sub-section 4.2). Sub-sections 4.3 and 4.4 then discuss the possibility of considering other elements (e.g. assistive pick technologies, risk scores) in the design of the WEP tool. The final sub-section (4.5) discusses some methodological aspects of this study.

4.1. WEP tool acceptance by users

Participants found the WEP tool generally useable and useful for considering the role of humans in OP system design to identify possible error sources. These results can be interpreted in light of the broadly discussed technology acceptance model (TAM) of Davis (1986), which suggests that a given technology or innovation – in this case, the WEP tool – is likely to be acceptable to users if it has both high perceived usefulness and good ease of use; two key precursors to the adoption of new technologies (Venkatesh and Davis 1996). The current study suggests that the WEP tool is capable to fulfil both cost and utility needs since it suggests design changes at an early stage of the design where the cost of changes are lower when aiming for a better human-system design interaction (Miles and Swif 1998). Despite the promising outcome on the WEP tool evaluation, research in behavioural operations management suggests that some workers may be averse to relying on unfamiliar algorithms/tools to perform a task (Healy and Moore 2007). In this study, some participants reported being ‘familiar with all the requirements to effectively design an OP system’; this view may represent an example of overconfidence bias (Healy and Moore 2007), which can be overcome by using appropriate decision and improvement efforts. As an example, providing comprehensive guidelines to end-users similar to the WEP tool user manual could be a practical strategy to be considered.

4.2. Implications for managers and engineers

The WEP tool illustrates the relationship between HF in system design and system performance, in this case, quality. Quality has been referred to by Sgarbossa et al. (2020) as an important element in logistics system design. HF aspects have also been broadly linked to errors and quality deficits in manufacturing settings (Kolus, Wells, and Neumann 2018) with fatigue as a noteworthy common causal element (Yung et al. 2020). In OP systems, fatigue, both mental and physical, emerged as an important contributor to human-system errors (Setayesh et al. 2022b); long-term fatigue can cause a greater risk of musculoskeletal disorders, human error, and quality deficits (Bláfoss et al. 2019; Sarter and Amalberti 2000). The chain of effects in manual OP operations are reported to start from system design elements that define the tasks in OP operations, and therefore the HF demands on order pickers (Setayesh et al. 2022b). If an individual’s capacity is exceeded while performing a task, individual and system performance may be affected negatively, causing human-system errors to occur. Previous research has noted that the absence of tools is a barrier for engineers in applying HF in design (Village et al. 2015; Broberg 2010) – a gap addressed by the WEP tool. While the WEP tool evaluation conducted here aimed at identifying error risks in existing OP systems, it could also be applied in system design stages where the costs of eliminating error risks are much cheaper than retrofitting scenarios (Neumann and Village 2012; Neumann et al. 2006). Items identified in the WEP tool can help reduce pick errors as result of wrong item selection, wrong quantity of item selection, and damaged items due to poor material handling. Each item in the WEP tool presents specific design criteria for potential system improvements. This research needs to be extended to consider cost and benefit profiles of creating high reliability in OP systems.

Much too often, organisations do not consider HF as a discipline that can contribute to organisational goals such as higher quality and lower error likelihood (Dul and Neumann 2009). Similarly, HF aspects are often ignored in production and logistics research (Neumann et al. 2021; Grosse, Glock, and Neumann 2017). This neglect of HF in systems design may be due to its lack of attention from both engineers and designers (Sgarbossa et al. 2020; Neumann et al. 2021). Engineers often see HF as merely a safety issue (Theberge and Neumann 2013; Broberg, Andersen, and Seim 2011; Broberg 2010) and fail to recognise the extent to which HF in system design can contribute to quality and process improvements (Kolus, Wells, and Neumann 2018; Yung et al. 2020). This approach is suggesting that the organisation integrates HF into engineering requirements for performance improvement objectives. The critical question is who should attend to humans in a system and to what extent? Particularly, as it was quoted by a participant ‘my department is only responsible for the optimized design process’, for example. The users identified the potential for the WEP tool to impact cross-departmental relationships by connecting HF to an organisational strategy that not only increases the potential of social goals (workers’ well-being), but also improves economic goals (higher quality and system performance). In this case, the WEP tool may be able to act as a ‘boundary-spanning object’ (Ipsen et al. 2020; Broberg, Andersen, and Seim 2011) allowing and educating even non-specialists (engineers, managers) to identify and communicate the possible impacts of poor human-system design interaction on the risks of errors in their logistic systems. We note that this systematizing of the quality risk assessment may be useful even in the face of overconfidence (Bendoly et al. 2010; Healy and Moore 2007), as systematic checklists like the WEP tool have been successfully applied by experts in aviation and medical systems (Clay-Williams and Colligan 2015) to avoid missing or mis-sequencing necessary hazard checks.

4.3. On the possibility of risk scores

Some participants expressed interest in obtaining quantitative data based on risk scores. The WEP tool provides a report on the factors causing poor human-system interaction, but does not measure the error risk levels. The WEP tool instead provides simple pass/fail rules aimed at helping users identify improvement opportunities. There are different human reliability assessment (HRA) models that have been developed to quantify human error probability. Applying these models for assessment is challenging as current HRA models do not aim at low-risk domains such as OP systems and do not connect error sources to system design elements (Setayesh, Di Pasquale, and Neumann 2022c). Further research studies are needed to understand if such quantitative output, and the associated complexity for users, are needed by potential users who want to balance the costs of change against the possible benefits of improved quality.

4.4. Integrating new technologies and operational paradigms

A few participants raised concerns about the lack of inclusion of particular assistive information technologies as a WEP tool limitation. Since the focus of the WEP tool is on HF in system design that are associated with quality deficits in OP, the current version of the WEP tool only considers very common assistive technologies (e.g. barcodes, and scanners). Considering the fast pace of technology development, OP 4.0 (Winkelhaus and Grosse 2020; Winkelhaus, Grosse, and Morana 2021), and assistive technologies (Glock et al. 2021; Grosse 2023), embedding new assistive technologies in the WEP tool will be an ongoing issue – particularly since HF considerations are routinely ignored in Industry 4.0 research (Neumann et al. 2021). With growing interest in Industry 4.0 and 5.0 paradigms (Breque, De Nul, and Petridis 2021), assistive technologies can contribute to performing tasks faster and more efficiently with fewer pick errors (Teo et al. 2021; Mandar, Dachry, and Bensassi 2020). Other studies have identified downsides to these technologies such as users’ mental fatigue and stress (Setayesh et al. 2022b; Neumann et al. 2021). This is consistent with sociotechnical systems theory – an extension of general systems theory – that identifies the mismatch between the technical system design and the social (human) sub-system as a source of dysfunction that leads to both system performance and human well-being problems (Skyttner 2001; IEA 2014). When human-system interactions are poorly designed, then errors and injuries can be expected that will compromise the anticipated profitability of the investment – a phenomenon referred to as ‘phantom profits’ (Neumann et al. 2021). Since the general human condition is not changeable, responsibility for errors lies substantially with the system designers who establish the conditions for error. While these design-induced error risks might be reduced with extensive operator training, appropriate attention to human-system interactions in design can address error risks already in early design stages where costs are lowest and benefits of the enhanced reliability are maximal. As such, the WEP tool can help optimise the fit between the technology and human elements of the OP system by helping practitioners improve their system designs in cost-effective ways that reduce the impact of known error inducing design flaws.

4.5. Future work and methodological considerations

A few participants commented on their interest to embed optimisation modelling into the WEP tool and/or implement the WEP tool directly in IT systems (e.g. Enterprise Resource Planning or Warehouse Management System). Embedding optimisation modules such as storage location assignment with respect to item turnover and minimising workers’ energy expenditures (e.g. Diefenbach and Glock 2019) can generally be considered. However, optimisation is a different research area which is not compatible with the goal of the WEP tool as optimisation mostly reflects on efficiency (Lee, Chung, and Yoon 2020; Roodbergen, Vis, and Taylor 2015). These remarks signal the need to integrate the WEP tool human-system design determination with other parts of the planning system. Embedding the WEP tool into the IT system remains for future research to allow for holistic planning such as layout, storage assignment, and routing optimizations.

This study has some limitations. While the sample is limited to only 33 participants, we observed that the qualitative data were saturated and the survey study showed statistical consistency on most constructs. Although the results were largely consistent, outlier views and useful improvement opportunities did emerge from the supporting interviews – a benefit of mixed methods research. It is possible that other participants working in different OP system contexts might raise new issues on identifying pick error risks in their system designs. The WEP tool can also be tried by order pickers which may bring a different perspective as they have to put the design into action; which remains for future validation studies. The focus of the WEP tool was on improving the quality of an OP system by integrating HF into the system design. This study used a questionnaire survey; however, using other usability assessment methods such as focus groups (Ryan et al. 2014; Onwuebuzie et al. 2009), contextual interviews (Holtzblatt and Jones 1995), and heuristic evaluations (Nielsen 1994) may provide further insights into the tool’s usability and utility. The data analysis was conducted based on users’ experience by using the WEP tool in a single interactive session. Further empirical studies for WEP tool validation in different OP application scenarios as well as continuous improvement efforts is warranted. Finally, we note that field validation – of the impacts of each WEP tool item on actual OP quality performance in OP systems – remains a task for future research.

5. Conclusion

The Warehouse Error Prevention (WEP) tool represents a mechanism for the transfer of scientific knowledge of HF sources of error in OP system design to warehouse stakeholders. Evaluating the tool is an important part of the creation of new approaches for practitioners to ensure the tool design matches user needs; an important objective and a precursor to future impact validation studies. The WEP tool was evaluated in trial applications using a survey (the SUS questionnaire with supplemented questions) and semi-structured interviews with 33 participants from 27 companies in three countries. The study outcome showed that 21 out of 33 participants found the WEP tool usable, functional, and useful. Most of the participants were satisfied with the usability and functionality of the WEP tool which helps to integrate HF into the warehouse and OP design and organisational management criteria. The majority of the participants found that the WEP tool could help them identify, and therefore prevent, the design level source of pick errors in warehousing and OP systems. Participants suggested the WEP tool could support both OP system process improvement and OP system design processes. As a tool that addresses the HF issues in system design, the WEP tool has boundary-spanning potential helping engineers and managers understand the design-related HF issues in their systems and providing an opportunity for ergonomists to understand the quality (and not just injury) implications of HF in the system. Future extensions to the WEP tool could address optimisation modelling, risk scoring, and the rise of new technologies while maintaining the goal of optimising human-system interaction by supporting good design. While further development is warranted, the current version of the WEP tool appears to be a useful and easy-to-use tool to help improve quality in OP system design and management.

Acknowledgements

The first author wishes to thank the very kind support provided to her, as a visiting researcher, by the Technical University of Darmstadt, Darmstadt Germany. This work was also supported by a fellowship within the Doctoral Programme of the German Academic Exchange Service (DAAD).

Data availability statement

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

Funding for this research was provided by the Natural Sciences and Engineering Research Council of Canada (NSERC) RGPIN-2018-05956.

Notes on contributors

Azin Setayesh

Azin Setayesh received her PhD at the Department of Mechanical and Industrial Engineering of Ryerson University, Canada. Her research interests include operations management and human factors engineering.

Eric H. Grosse

Eric H. Grosse is a Junior Professor and the Head of the Chair of Business Management and Digital Transformation in Operations Management at Saarland University, Germany. He is also Senior Research Fellow at the Institute of Production and Supply Chain Management at Technical University of Darmstadt, Germany. His research interests include warehouse optimisation as well as sustainability and digitalisation in production and logistics.

Michael A. Greig

Michael A. Greig is a postdoctoral fellow and research associate in the Human Factors Engineering Lab at Ryerson University. His research work develops approaches to understand and improve human factors and ergonomics inclusion and reporting in organisation processes.

Christoph H. Glock

Christoph H. Glock is a Full Professor and the Head of the Institute of Production and Supply Chain Management at Technical University of Darmstadt, Germany. His research interests include inventory management, supply chain management, warehousing, sustainable production, and human factors in logistics and inventory systems.

W. Patrick Neumann

W. Patrick Neumann is a Full Professor in the Department of Mechanical and Industrial Engineering of Ryerson University, Canada. Current research, conducted at the Human Factors Engineering Lab, focuses on the design of work systems that are both effective and sustainable from human and technical perspectives. He is a Design Scientist and certified human factors professional (Eur. Erg.).