https://orcid.org/0000-0003-4834-4761
Abstract
Industrial exoskeletons are globally developed, explored, and increasingly implemented in industrial workplaces. Multiple technical, physical, and psychological aspects should be assessed prior to their daily application in various occupational environments. The methodology for evaluating these aspects is not standardised and differs in terms of focussed research objectives, used types of analyses, applied testing procedures, and use cases. The aim of this paper is to provide a matrix comparing the prevalence of different types of analyses combined with their respective research objective(s). A systematic review in the database ‘Web of Science’ identified 74 studies, mainly in laboratory settings, with a focus on short-term effects as well as with male-dominated samples being low representative for industrial workforces. The conducted evaluation methodologies are further discussed and compared in terms of testing procedure, sample, and research objectives. Finally, relevant aspects for prospectively evaluating industrial exoskeletons in a more harmonised and comprehensive way are suggested.
Practitioner summary: Industrial exoskeletons are still inconsistently and insufficiently evaluated in scientific studies, which might hamper the comparability of systems, threaten the human health, and block an iterative system optimisation. Thus, a comprehensive evaluation methodology is needed with harmonised and multicriteria types of analyses.
1. Introduction
The industrial interest in implementing exoskeletons in several working processes increasingly rises, since their use can facilitate the workload, reduce the risk of work-related musculoskeletal disorders, or gain working productivity and precision of the workforce (Bogue Citation2018; Baltrusch et al. Citation2019). Industrial exoskeletons are typically designed for assisting tasks with high repetition, in non-neutral postures (Hensel and Keil Citation2019), and with heavy workloads. Thus, exoskeletons can be a promising personal measure if other technological (e.g. cranes, power lifts) or organisational techniques (e.g. modified workstations, redesigned work processes, ergonomic trainings) are not applicable (Madinei et al. Citation2020a; Pacifico et al. Citation2020). This might be the case for, e.g. highly flexible workplaces (i.e. in terms of location or content), difficultly accessible work locations, individual work-related disorders, or high capital expenditures for conventional approaches.
For meeting specific workplace requirements and characteristics, exoskeletons technically pursue different system philosophies and feature various morphologies, which can lead to diverse physical, psychological, and cognitive effects on the wearer (Fox et al. Citation2019; Schroeter et al. Citation2020; Weidner and Karafillidis Citation2018). To assess these effects, investigators can choose from many objective and subjective evaluation methods.
The (support) performance of exoskeletons is already summarised and contrasted in different systematic reviews. For instance, de Looze et al. (Citation2016) scanned 40 papers for actuation type and potential effects on physical workload (period: 01/1995−08/2014). An update with 33 publications (period: 01/2015−06/2020) by Kermavnar et al. (Citation2021) focussed on exoskeletons for back support and summarised study designs, (in)dependent variables, and statistical results. Pinto-Fernandez et al. (Citation2020) identified in 187 publications performance indicators for practical benchmarking lower limb exoskeletons and included motor skills in an overview (period: 01/1989−04/2018). Del Ferraro et al. (Citation2020) dealt with upper limb exoskeletons and summarised investigations on metabolic costs (period: until 09/2020). Weidner et al. [Citation2020) identified 67 systems with a maturity level to be sufficient for (at least) industrial pilot studies (state: 02/2019) and present a distribution of supported body regions, technical properties, and relational patterns between human and technology (Weidner and Karafillidis Citation2018).
Thus, this paper focuses on conducted evaluation methodologies in laboratory and field studies with industrial applicable exoskeletons – independent of supported body parts or research objective(s). The prevalence of each type of analyses combined with its respective research objective(s) is quoted and contrasted in a matrix. This overview should help to emphasise the common variety in evaluations, the focus on specific research objectives, and the potential lack in standardisation in respect to methods, procedures, and considered tasks. In addition, summarised information about prevalence rates or patterns as well as promising individual handlings can be the basis for prospectively deriving best practise approaches and first steps towards harmonised and comprehensive evaluation methodologies. Besides, evaluators can get specific information about common methodologies and inspiration for applicable analyses and research objectives.
2. Methods
The systematic review was conducted via online research using the database of ‘Web of Science Core Collection’. The search considered all types of documents written in the English language, published between January 2000 and December 2020, and cited in the ‘Science Citation Index Expanded’ or ‘Emerging Sources Citation Index’. The study selection process is presented in . NH and GP independently screened the identified records and articles. RW respectively resolved any disagreement between the other two authors. All three authors agreed on the final study selection.
Figure 1. PRISMA diagram of literature review and study selection.
Afterwards, the selected studies were summarised, and respective characteristics of the evaluated exoskeleton(s) and test setup were extracted. Here, the following distinctions were made for exoskeletons:
maturity level: prototype vs. commercial system,
actuation: active vs. passive as well as
supported body region: lower limb vs. trunk vs. upper limb.
Accordingly, for test setups:
use case: laboratory vs. field,
sample: size, gender, average age, height, weight, and health status,
applied type of analyses:
objective: applied forces, executed errors, fine motor skills, metabolic costs, modelling, movement patterns, muscular activity, system data, testing machine data, walking speed, working speed,
subjective: survey and
objective or subjective: observation.
Concerning the sample, the data were only statistically considered, if all above mentioned information were given. Previously, any gender specifications were averaged over the total sample. In case of varying samples in a paper, only the biggest sample group was considered. Applied evaluation methods were not further considered if the purpose was principally for designing or constructing the exoskeleton at an early development stage. Finally, each considered method was matched with its respective research objective(s). Here, the following constructs and items (italicized) were distinguished, whereas some constructs cannot be strictly separated from each other (duplications in matrix possible):
Work performance principally addresses the productivity with exoskeletons. Common aspects are endurance time, number of errors, number of repetitions in a specific period, and task completion time (Maurice et al. Citation2020). All these criteria were respectively subsumed under working speed and quality of work.
Kinematic aspects comprise principally the movability with the exoskeleton, since the user’s range of motion should not be impaired (Baser, Kizilhan, and Kilic Citation2019) or changed in motion sequences in terms of movement patterns (Baltrusch et al. Citation2019) or load transfers (i.e. postural control, risk of falling (Maurice et al. Citation2020)). Ideally, the system features a synchronicity with the user’s dynamics (i.e. acceleration and velocity (Yong et al. Citation2019)) and motion trajectories (i.e. position, direction, and orientation (Li, Yuan et al. Citation2018)). The latter can also cause discomfort, since, e.g. misalignments between exoskeleton and human can lead to skin irritations or sores (Pacifico et al. Citation2020).
The comfort is crucial for user’s health and system acceptance especially in case of longer wearing periods. It can be observed in either general or specific inquiries (i.e. whole body, certain body regions, (specific) working tasks).
Since exoskeletons should reduce the user’s workload, the relief can be evaluated in either physical (i.e. reduced muscle activity or increased endurance time) or mechanical dimensions (i.e. supporting torques or forces). The support of exoskeletons can also influence the individually perceived task difficulty.
Acceptance and usability are two constructs influencing each other and being influenced by many aspects (Hensel and Keil Citation2019; Alabdulkarim and Nussbaum Citation2019). In particular, the perceived discomfort might be a major factor for the system acceptance (Luger et al. Citation2019). In this paper, system acceptance mainly includes concrete questions to the user’s voluntary intention to use the system. Usability comprises factors like user-friendliness and the effectiveness, efficiency, and satisfaction of the system. This includes aspects like ease-of-use or effort for donning and doffing. However, reductions in perceived task difficulty might also influence usability ratings.
3. Results
The systematic review identified 74 studies. The extraction results are summarised in . Most of the scanned studies were generally performed with humans in laboratories (n = 65). Occasionally, work environments were simulated with real working tools, objects, or motion scenarios. Four studies evaluated exoskeletal prototypes in modelled environments (Aoustin and Formalskii Citation2018; Nelson et al. Citation2020; Pan, Gao, and Miao Citation2014; Yan, Zhang, and Qi Citation2018) without any sample. Five studies were performed in field analysing subjective opinions (Smets Citation2019; Hensel and Keil Citation2019) combined with muscular activity (Gillette and Stephenson Citation2019), range of motion (Iranzo et al. Citation2020), as well as movement patterns and simulated compression forces (Graham, Agnew, and Stevenson Citation2009). Only commercial exoskeletons – except of Graham, Agnew, and Stevenson (Citation2009) – were used in field, whereas prototypes are predominant in laboratories (ca. 70%). Overall, 45 prototypes and 40 commercial systems were investigated. In terms of the system’s characteristics, 59 passive and 26 active systems were evaluated, but only passive systems in field studies. The total amount of systems is higher than the number of scanned papers, since some paper evaluated more than one system (e.g. Alabdulkarim and Nussbaum Citation2019). 28 systems supported the upper limb, 37 the trunk, 20 the lower limb.
Table 1. Summary of reviewed studies evaluating industrial exoskeletons.
The sample was fully described in 48 papers (mean size: n = 13.06 ± 7.27 (males: n = 10.19 ± 6.98; females: n = 2.88 ± 4.12); mean age: 27.05 ± 5.06; mean weight: 73.99 ± 5.20; mean height: 176.51 ± 4.06). In general, the sample was small (6% with less than five subjects, 19% with less than ten subjects), young (79% younger than 30 years), and male-dominated (58% without female subjects, 20% with a gender-balanced sample). The highest sample sizes were found in Luger et al. (Citation2019) with 45 subjects. In more than 82% of all studies with humans (n = 68), it was explicitly mentioned that the tested subjects were healthy or without any musculoskeletal problems. Comparing all field and laboratory studies, the total sample size was 13.60 ± 8.43 and 11.28 ± 7.05, respectively.
Inspired by the illustration of Pinto-Fernandez et al. (Citation2020), summarises the prevalence of different types of analyses – partly including several evaluation methods – with its respective research objective. The last column exhibits the total number of accessible research objectives with a certain type of analyses, whereas the last line reveals existing focus areas. Since physical support systems are an integration of the human, machine, and activity (Weidner, Kong, and Wulfsberg Citation2013), each type of analyses is also qualitatively assigned to these superordinate dimensions on the left side.
Figure 2. Matrix for illustrating the respective prevalence of different types of analyses for selected research objects (in the style of Pinto-Fernandez et al. [Citation2020]).
![Figure 2. Matrix for illustrating the respective prevalence of different types of analyses for selected research objects (in the style of Pinto-Fernandez et al. [Citation2020]).](/cms/asset/94917009-283a-452b-b2db-529e91c7eef8/terg_a_1970823_f0002_b.jpg)
In total, 193 evaluation methods (161 objective and 32 subjective) were used. More than three quarters of all studies applied more than one type of analyses; about 50% of them combined a subjective and an objective method. The physical relief by the exoskeleton was evaluated most of the time, followed by the mechanical support. Further key research objectives were movability, motion sequence, usability, and comfort. The most frequently applied type of analyses were surveys with questionnaires or interviews (e.g. Maurice et al. Citation2020; Smets Citation2019) that often assess the perceived physical support, comfort as well as the system’s acceptance and usability. Since they were easily adaptable to different research questions, they were used in almost every category except for kinematic aspects. The usability was also usually evaluated with questionnaires, but Baltrusch et al. (Citation2018) used objective observations of functional performance tests. The analysis of muscular activity with electromyography (e.g. Wang et al. Citation2020; Pillai, van Engelhoven, and Kazerooni Citation2020) is predominantly applied for evaluating the physical relief for selected (in particular supported) muscle groups. The analysis of movement patterns (e.g. recorded with force sensors (Baser, Kizilhan, and Kilic Citation2019), inertial measurement units (Maurice et al. Citation2020), video camera (Wang et al. Citation2018), optical marker system (d’Elia et al. Citation2017)) focussed mainly on kinematic aspects or working speed comparisons. Modelling (e.g. musculoskeletal simulations (Blanco et al. Citation2019; Nelson et al. Citation2020; Weston et al. Citation2018), mathematical/numerical calculations (Aoustin and Formalskii Citation2018; Han et al. Citation2020; Pan, Gao, and Miao Citation2014; Wang et al. Citation2018) addressed mainly the system’s support as well as the movability and motions during its use. Occasionally, input data from sample’s anthropometry or force sensors are also considered. Other analyses were more seldomly applied. This includes time impacts on workplaces (Dahmen and Constantinescu Citation2018), the number of executed errors (e.g. with a detection ring (Alabdulkarim and Nussbaum Citation2019)), the fine motor skills with a pegboard (Madinei et al. Citation2020a), applied forces (e.g. with a contact pressure mat (Huysamen, Looze, et al. Citation2018), load cell (Liu et al. Citation2020), dynamometer (Li, Yuan, et al. Citation2018), force plate (Maurice et al. Citation2020)), metabolic costs (e.g. with ergospirometry (Baltrusch et al. Citation2019), oxygen consumption (Junius et al. Citation2018), heart rate (Maurice et al. Citation2020), blood lactate concentration (Galle et al. Citation2014), blood oxygenation (Gams et al. Citation2013), minute ventilation (Gams et al. Citation2013)), individual working speeds (e.g. walking on a treadmill (Baltrusch et al. Citation2019)), task execution with maximum acceptable working frequency (Alabdulkarim and Nussbaum Citation2019)), influences on the work load with sample independent data (e.g. from a humanoid testing machine (Nabeshima et al. Citation2018; Ito et al. Citation2018)), or analysis of system’s motion synchronicity and cycle stability with a joint simulator (Shamaei et al. Citation2014). Godwin et al. (Citation2009) and Lotz et al. (Citation2009) applied a testing machine assessing the human’s maximum remaining strength after exhausting tasks with or without exoskeletal support. Objective observations with measureable criteria were made one-dimensionally (Wang et al. Citation2018) or multidimensionally (Baltrusch et al. Citation2018; Kozinc et al. Citation2020).
4. Discussion
In evaluation studies, the research objective generally depends on the system’s level of development as well as the evaluator’s perspective (Hoffmann et al. Citation2019). In studies with exoskeletal prototypes, the evaluation purpose is often a ‘proof of concept’. Therefore, technical functionalities (i.e. mechanical or physical support to the user, range of motion, usability) are in the centre of interest. Alternatively, field studies that are mainly conducted with matured systems focus more on downstream effects like usability and acceptance. According to the evaluation horizon, exoskeletons should be ideally evaluated in prolonged sessions to reflect a daily use in industry as well as to detect crucial long-term effects like changes in the subjective attitude towards the system as well as possible learning or adaption processes. For instance, users can perceive discomfort in certain body areas due to a poor microclimate on interfaces, increase the total range of motion due to system fit adjustments (Smets Citation2019), improve the handling and consequently the task execution (Sadler, Graham, and Stevenson Citation2011), or optimise the self-selected support level (Hensel and Keil Citation2019). However, the evaluation of short-term exoskeletal effects is still predominant, particularly in laboratory studies. Possible reasons might be limited resources and increased practicability (Alabdulkarim and Nussbaum Citation2019; Rashedi et al. Citation2014). Thus, short-term results are sometimes carefully interpreted for a longer timespan (e.g. Pacifico et al. Citation2020).
Due to the selected search pattern in terms of, e.g. database, time horizon, and search terms, the presented matrix is neither comprehensive nor universally applicable. For instance, Spada et al. (Citation2017) assess the task performance, endurance time, and perceived discomfort in a field setting with a large representative sample (n = 31, age: 51.5 ± 4.7 years). Schroeter et al. (Citation2020) focus on cognitive effects (i.e. ability to concentrate, proneness to errors) in field caused by physical support of an exoskeleton. In our filtered data set, Maurice et al. (Citation2020) and Li, Cheng, et al. (Citation2018) are the only ones investigating effects on mental support.
In the following, several other aspects and differences for evaluating industrial exoskeletons are objectively discussed concerning testing procedure, sample as well as used methods and criteria. By exemplifying certain publications, the authors of this article never intend to personally address or challenge any author(s) or scientific study. Finally, suggestions for evaluating industrial exoskeletons in a harmonised and comprehensive way are presented.
4.1. Testing procedure
It becomes apparent that testing procedures are generally inconsistently documented and processed. For instance, it is assumed that all authors have conducted an individual fitting of the exoskeleton and a familiarisation of the subject with the test setup (i.e. system, tasks, evaluation method) before starting the study, but this is not always explicitly written in the paper. Otherwise, questionnaires with complicated scale classifications submitted to inexperienced young participants might not provide accurate results (Pacifico et al. Citation2020).
For deriving usage effects, test subjects usually perform different tasks with the exoskeleton (and often for benchmarking without the system). The tasks are often reduced from a possible large scope of activity to selected core activities of an end-user. Static tasks are held in predetermined positions for a longer period (e.g. 30 s), whereas dynamic tasks often show a repeated motion range and a neutral position. Both types are frequently varied in (end-)position or weight to increase industrial representativeness, but without any standardisation along the above-mentioned studies (also when using the same exoskeleton). However, more cycles, longer static postures, or other motions could also appear in real industrial applications, particularly by considering side tasks like walking or resting. In practise, the task selection for studies is often (pre-)determined by the study environment (e.g. Hensel and Keil Citation2019), the chosen research objective(s) and evaluation method(s), and the anticipated support region of the exoskeleton (i.e. lifting tasks for exoskeletons of the trunk (e.g. Huysamen, Bosch, et al. Citation2018) or overhead tasks for exoskeletons of the shoulder-neck region (e.g. Otten, Weidner, and Argubi-Wollesen Citation2018).
Since results naturally depend on the test setup, it must be mentioned that authors differently specify task executions and exoskeletal support levels. For instance, task velocity formulations can vary from as fast as possible (Maurice et al. Citation2020), self-selected paces (Pacifico et al. Citation2020) to given executions within a certain period (Hefferle et al. Citation2020). Other evaluators only entitle the task (e.g. Huysamen, Bosch, et al. Citation2018; Pillai, van Engelhoven, and Kazerooni Citation2020), which simulates real working conditions in a better way but hampers the comparability between different test subjects (Pillai, van Engelhoven, and Kazerooni Citation2020). Furthermore, most authors do not declare the respective support level that is adjustable for many active (and some passive) systems, although the adjusted support level can influence evaluation results (e.g. reduced electromyography can increase work endurance time (Gillette and Stephenson Citation2019; Bosch et al. Citation2016), but more support does not necessarily imply a higher system acceptance (van Engelhoven et al. Citation2019)). For instance, Alabdulkarim and Nussbaum (Citation2019) or Madinei et al. (Citation2020a) let test subjects to individually select their support level, which can naturally vary between subjects (Pillai, van Engelhoven, and Kazerooni Citation2020). Alternatively, Pacifico et al. (Citation2020) set the support to the minimum, whereas Maurice et al. (Citation2020) individually adapt the support referring to the user’s arm weight. Practically, any declaration of support can be difficult for evaluators since support levels are usually quoted in different ways. Alternatively, Sadler, Graham, and Stevenson (Citation2011) and Lotz et al. (Citation2009) adjust the respective task load in accordance with the previously measured user’s strength characteristics. Graham, Agnew, and Stevenson (Citation2009) digitise photos of subjects and use a biomechanical computer program to calculate individually required forces of the actuator.
Furthermore, some authors only evaluate a subsystem to reduce the development effort. For instance, Dong et al. (Citation2019) evaluate an exoskeleton for gait assistance only on the right shank and place an additional weight as a dummy for counteracting imbalances on the other shank. Wang et al. (Citation2018) focus on effects according to the range of motion if exoskeletal degrees of freedom are reduced, but solely construct a kinematic version of the system without any actuation.
4.2. Sample
Samples are generally inconsistently described, since some authors only mention the total sample size, whereas others typically add several more distributional details for age, weight, height, and gender. In some descriptions, working experiences or arm dominances are additionally considered, since posture, fine motor skills, and manageable tool mass can heavily vary in these cases (e.g. van Engelhoven et al. Citation2019; Maurice et al. Citation2020). Alternatively, Blanco et al. (Citation2019) and Pacifico et al. (Citation2020) consider the arm dominance in the task execution. However, the test subjects of Maurice et al. (Citation2020) always use the right arm. Finally, authors should weigh which sample description is really needed, since the total number of participants can be sufficient for specific questionnaires (see, e.g. Otten, Weidner, and Argubi-Wollesen Citation2018). For modellings, samples are normally not needed (e.g. Yan, Zhang, and Qi Citation2018). Here, sample information is only given if these data are used as input data for equations or model building (e.g. Weston et al. Citation2018). Besides, sample sizes can vary between conducted methods (e.g. Otten, Weidner, and Argubi-Wollesen Citation2018; Galle et al. Citation2014) or different phases of intervention (Smets Citation2019; Kang, Hsu, and Young Citation2019).
Various authors (e.g. Madinei et al. Citation2020a; Maurice et al. Citation2020; Pacifico et al. Citation2020) concede limitations for the representativeness of their tested sample, since test subjects are often sparse, young, male-dominated, without pre-existing diseases as well as inexperienced with the executed manual tasks and the handling of the exoskeleton. This can have various practical reasons and is differently argued. For instance, evaluators can disclaim any risk by considering only healthy individuals. Initially planned test subjects can be finally extracted due to, e.g. massive system aversion or discomfort, medical backgrounds, unsuitability of the system or the method to the workplace (Hensel and Keil Citation2019) or the user (e.g. unusual anthropometry, skin wounds close to the interface (Pacifico et al. Citation2020), no adhesion of electromyography electrodes on user’s skin due to sweating (Pillai, van Engelhoven, and Kazerooni Citation2020)). Furthermore, the male domination in samples can be appeased by the facts that industrial and physical demanding workplaces are primarily occupied by men, the design and size of certain exoskeletons might not be generally suitable for female anatomy (Luger et al. Citation2019; Hensel and Keil Citation2019), difficulties appear in obtaining approvals by the Investigational Review Board for female participants (van Engelhoven et al. Citation2019), or females are unable to complete all experimental conditions (Rashedi et al. Citation2014). Elsewhere, solely female (Galle et al. Citation2014; Godwin et al. Citation2009) and gender-balanced samples appear (e.g. Madinei et al. Citation2020a; Sadler, Graham, and Stevenson Citation2011). Here, gender effects on, e.g. task completion time (Madinei et al. Citation2020a) or handled tool mass (Alabdulkarim and Nussbaum Citation2019) can be carefully derived.
4.3. Methods and criteria
Authors used different methods in their studies for various purposes like system design, assessment of usage effects, or observation of task executions. For instance, systems can be designed with the help of motion capture and fitness equipment (Li, Yuan, et al. Citation2018), kinematic models (Yan, Zhang, and Qi Citation2018) combined with finite elements analysis (Yong et al. Citation2019), or human simulations based on motion capture (Blanco et al. Citation2019). Because of possible multidimensional exoskeletal usage effects, authors often consider more than one evaluation criterion by combining different evaluation methods. Here, questionnaires are particularly easy to additionally implement in studies. Some authors also try to investigate statistical dependencies between evaluation criteria like muscular activity in supported body regions, error proneness, and working speed (Alabdulkarim and Nussbaum Citation2019), reductions in physical demand, discomfort, usability, and intention-to-use (Hensel and Keil Citation2019), reductions in muscular activity or reduced muscle fatigue and usage decision or system acceptance (Gillette and Stephenson Citation2019). Other authors multidimensionally use or interpret the results of electromyography for, e.g. calculating supporting moments (Koopman, Näf, et al. Citation2020), metabolic costs (Grazi et al. Citation2020), mean user’s power frequency (Rashedi et al. Citation2014), or the system’s efficiency (Yong et al. Citation2019) as well as detecting changes in working speed (Baltrusch et al. Citation2018) or in motions (Madinei et al. Citation2020a). The analysis of more muscle groups than only the directly supported ones can help to detect a physical burden as well as to derive further comfort and usability issues (Maurice et al. Citation2020; van Engelhoven et al. Citation2019; Weston et al. Citation2018). However, certain muscle groups might not be accessible due to overlying exoskeletal components (Alabdulkarim, Kim, and Nussbaum Citation2019; Madinei et al. Citation2020b). Other authors observed the correct task execution with, e.g. motion capture or force sensors to delete any outliers in the sample data for further analyses (Alabdulkarim and Nussbaum Citation2019; Weston et al. Citation2018). Alternatively, a video camera (Hefferle et al. Citation2020) or auditory feedbacks (Yin et al. Citation2020) can be used.
Finally, it must be mentioned that several evaluation artefacts may occur, since sensors may deliver wrong data, questionnaires may be prone to personal preferences as well as both objective and subjective methods may depend on the interpretation of evaluators (Hoffmann et al. Citation2019). It also needs to be emphasised that most scanned evaluation studies do not consider any safety aspects to eliminate hazards to users (e.g. climbing stairs, working with certain machines). In the future, evaluations might intensify clarifications for this case as well as according to long-term effects (Theurel and Desbrosses Citation2019) with resulting recommendations to the wearing time, the overall dimensions of the user when wearing the exoskeleton, the connectivity with personal protection equipment and work tools (e.g. forklift trucks), the usability in specific working or climate environments or during side tasks, the maintenance and cleaning effort, possible damping effects for work with vibrating tools as well as the influence of physical support on mental relief and of different (industrial) implementation strategies on later system acceptance. Even in laboratory settings, more practical insights might be derivable if work-experienced subjects are considered (see (Koopman, Näf, et al. Citation2020)).
4.4. Suggested improvements
Based on the findings of this review and the previous discussion with referring to different study examples, it becomes obvious that industrial exoskeletons need to be generally evaluated in a more harmonised and comprehensive way. Prospectively, established guidelines or standardizations in processing, parametrising, and documenting might generate harmonised evaluation results and enable knowledge- or artificial-intelligence-based system improvements by identifying central exoskeletal core properties, user- or application-related system requirements, and best practise system developments. Due to detected interdependencies between some evaluation criteria, comprehensive exoskeletal evaluations might be prospectively reducible to selected research objectives and evaluation methods.
Since results naturally depend on the initial situation, it is generally recommended to familiarise test subjects with the exoskeleton, the tasks, and the applied evaluation methods before starting the evaluation procedure. Exoskeletons should also be individually adjusted in terms of fitting and support characteristic. Task execution should consider the individual handedness. The test subjects should represent end-users and use cases (industrial main task with side tasks) as best as possible. Concerning the evaluation itself, summarises six recommended evaluation aspects for deriving the exoskeletal performance and workplace suitability, taking up the personal experiences of the authors from several field and laboratory studies as well as investigations concerning different research projects like ‘smartASSIST (2014-2020)’ and ‘exo@work’ (since 2018). It is based on the following four principles:
Figure 3. Considerable aspects for a harmonised evaluating methodology of industrial exoskeletons.
Consideration of the unit human, support, and activities: In terms of exoskeletons, the human-machine-interaction should be evaluated by considering the activity as well (see (Weidner, Kong, and Wulfsberg Citation2013), since all three dimensions influence each other in support situations. In an occupational context, the respective dimensions are characterised by the user (e.g. with physiological precondition, personal technical affinity, pre-existing diseases), the support/exoskeleton (e.g. support characteristic with, e.g. maximum support level, system handling, path of force, overall dimensions), and the industrial workplace (e.g. with profile of main and side tasks as well as spatial, climatic, and social working environment).
Harmonised methodology across studies and investigations: The applied types of analyses (i.e. surveys, (biomechanical) analyses, and modelling) should be expanded as well as synchronised or harmonised with their respective research objectives to a profound comprehension of the exoskeletal technical functionality and performance. A multidimensional perspective with both objective and subjective evaluation methods is beneficial. For instance, Maurice et al. (Citation2020) suggest evaluating objectively the posture, movement, and effort as well as subjectively the user’s perception and system acceptance. Hefferle et al. (Citation2020) propose to evaluate a physical relief objectively with different local and global methods combined with a general subjective evaluation. Certain research objectives should be repeatedly evaluated with a time shift of, e.g. four to six weeks to reveal learning or adaption process of the user to the system itself or the workplace.
Harmonised test cases: Test parcours should be applied with test rows consisting of multiple work tasks with varying tools and motions (see as an example for load handling (Bostelman et al. Citation2019)). Any standardizations can generally increase the transparency and comparability of the systems’ performance. This implies also comprehensive task description in terms of, e.g. frequency, velocity, rang of motion, or weight. It is also recommended to observe and record (correct) task executions to exclude outliers and ease the data interpretation. Intermediate pauses between test cycles and randomisation of test orders (i.e. tasks, use of exoskeleton) can prevent test effects. Since movements with the biggest support of the exoskeleton are usually only a small percentage in a total work shift, evaluations should also include side tasks. Furthermore, several exclusion criteria for a principal applicability of the system in a targeted use case can be usually quickly checked with common sense. This should be conducted first and before any elaborate evaluation.
Objectivity and general validity due to standardised testing machines: Sample-independent evaluation methods (e.g. testing machines with internal measurement sensors, modelling) should be used due to their objectivity and general validity. Inspired by a testing machine for lifting tasks (Nabeshima et al. Citation2018), a similar machine seems to be feasible to tasks at head level or above, since exoskeletons for upper limbs were predominantly evaluated in the scanned papers with overhead drilling tasks and a 90° angle in shoulder and elbow, respectively. Derived exoskeletal data sheets can also help to interpret the recorded data in biomechanical analyses.
5. Conclusion
The systematic review identified 74 papers and emphasises that previously applied methodologies for evaluating exoskeletons with industrial application are generally neither comprehensive (Maurice et al. Citation2020) nor consistent (de Looze et al. Citation2016; Kim et al. Citation2018). This might have different practical reasons but also biases evaluation results (Hefferle et al. Citation2020) and hampers any iterative system optimisation when considering, e.g. the transdisciplinary development approach for wearable systems by Weidner et al. (Citation2017).
Support situations are always individually formed by the complex interaction of the user, the exoskeleton, and the industrial workplace with their specific characteristics. However, the user can be considered as the crucial link between the two other dimensions and should be in the centre of interest (Weidner and Hoffmann Citation2020). Thus, system developers or manufacturers should continually integrate the targeted end-user in system development processes or should at least conduct comprehensive and harmonised evaluations by considering as many aspects suggested in as possible. However, even comprehensive evaluations cannot consider all real application circumstances in advance (Hoffmann et al. Citation2019). Thus, industrial exoskeletons probably remain a case-by-case treatment, which can lead to guided and individual self-evaluations of end-users. In this context, Ralfs, Hoffmann, and Weidner (Citation2021) present an approach of a decision support matrix by merging task properties, work profiles, and exoskeletal characteristics in every single use case. This should sensitise end-users for a multidimensional decision perspective and finally enable an objective comparability and selection of systems for individual application cases.
Disclosure statement
The authors declare that they have no competing interests and no conflicts of interest with respect to this authorship or the publication of this article. The authors are solely responsible for the manuscript content.
Additional information
Funding
References
- Abdoli-Eramaki, M., J. M. Stevenson, S. A. Reid, and T. J. Bryant. 2007. “Mathematical and Empirical Proof of Principle for an on-Body Personal Lift Augmentation Device (PLAD).” Journal of Biomechanics 40 (8): 1694–1700. doi:https://doi.org/10.1016/j.jbiomech.2006.09.006.
- Alabdulkarim, S., and M. A. Nussbaum. 2019. “Influences of Different Exoskeleton Designs and Tool Mass on Physical Demands and Performance in a Simulated Overhead Drilling Task.” Applied Ergonomics 74: 55–66. doi:https://doi.org/10.1016/j.apergo.2018.08.004.
- Alabdulkarim, Saad, Sunwook Kim, and Maury A. Nussbaum. 2019. “Effects of Exoskeleton Design and Precision Requirements on Physical Demands and Quality in a Simulated Overhead Drilling Task.” Applied Ergonomics 80: 136–145. doi:https://doi.org/10.1016/j.apergo.2019.05.014.
- Alemi, M. M., S. Madinei, S. Kim, D. Srinivasan, and M. A. Nussbaum. 2020. “Effects of Two Passive Back-Support Exoskeletons on Muscle Activity, Energy Expenditure, and Subjective Assessments during Repetitive Lifting.” Human Factors 62 (3): 458–474. doi:https://doi.org/10.1177/0018720819897669.
- Aoustin, Y., and A. M. Formalskii. 2018. “Walking of Biped with Passive Exoskeleton: Evaluation of Energy Consumption.” Multibody System Dynamics 43 (1): 71–96. doi:https://doi.org/10.1007/s11044-017-9602-7.
- Baltrusch, S. J., J. H. van Dieën, S. M. Bruijn, A. S. Koopman, C. A. M. van Bennekom, and H. Houdijk. 2019. “The Effect of a Passive Trunk Exoskeleton on Metabolic Costs During Lifting and Walking.” Ergonomics 62 (7): 903–916. doi:https://doi.org/10.1080/00140139.2019.1602288.
- Baltrusch, S. J., J.H. van Dieen, C. A. M. van Bennekom, and H. Houdijk. 2018. “The Effect of a Passive Trunk Exoskeleton on Functional Performance in Healthy Individuals.” Applied Ergonomics 72: 94–106. doi:https://doi.org/10.1016/j.apergo.2018.04.007.
- Baser, O., H. Kizilhan, and E. Kilic. 2019. “Biomimetic Compliant Lower Limb Exoskeleton (BioComEx) and Its Experimental Evaluation.” Journal of the Brazilian Society of Mechanical Sciences and Engineering 41 (5): 575. doi:https://doi.org/10.1007/s40430-019-1729-4.
- Bastide, S., N. Vignais, F. Geffard, and B. Berret. 2017. “Analysis of Human-Exoskeleton Interactions: An Elbow Flexion/Extension Case Study.” Computer Methods in Biomechanics and Biomedical Engineering 20 (sup1): 9–10. doi:https://doi.org/10.1080/10255842.2017.1382835.
- Blanco, A., J. M. Catalán, J. A. Díez, J. V. García, E. Lobato, and N. García-Aracil. 2019. “Electromyography Assessment of the Assistance Provided by an Upper-Limb Exoskeleton in Maintenance Tasks.” Sensors 19 (15): 3391. doi:https://doi.org/10.3390/s19153391.
- Bosch, T., J. van Eck, K. Knitel, and M. de Looze. 2016. “The Effects of a Passive Exoskeleton on Muscle Activity, Discomfort and Endurance Time in Forward Bending Work.” Applied Ergonomics 54: 212–217. doi:https://doi.org/10.1016/j.apergo.2015.12.003.
- Bostelman, R., Y.-S. Li-Baboud, A. Virts, S. Yoon, and M. Shah. 2019. “Towards Standard Exoskeleton Test Methods for Load Handling.” In Wearable Robotics Association Conference (WeaRAcon), Scotsdale, MO, 21–27. doi:https://doi.org/10.1109/WEARRACON.2019.8719403.
- Bogue, R. 2018. “Exoskeletons – A Review of Industrial Applications.” Industrial Robot: An International Journal 45(5): 585–590. doi:https://doi.org/10.1108/IR-05-2018-0109.
- Cha, D., S. N. Oh, H. H. Lee, K. S. Kim, K. I. Kim, and S. Kim. 2015. “Design and Evaluation of the Unmanned Technology Research Center Exoskeleton Implementing the Precedence Walking Assistance Mechanism.” Journal of Electrical Engineering and Technology 10 (6): 2376–2383. doi:https://doi.org/10.5370/JEET.2015.10.6.2376.
- Chandrapal, M., X. Q. Chen, W. H. Wang, B. Stanke, and N. Le Pape. 2013. “Preliminary Evaluation of Intelligent Intention Estimation Algorithms for an Actuated Lower-Limb Exoskeleton Regular Paper.” International Journal of Advanced Robotic Systems 10 (2): 147. doi:https://doi.org/10.5772/56063.
- Dahmen, C., and C. Constantinescu. 2018. “Methodology for Evaluating of the Time-Management-Impact of Exoskeleton-Centred Workplaces.” Acta Technica Napocensis, Applied Mathematics, Mechanics, and Engineering 61 (4): 677–684.
- del Ferraro, S., T. Falcone, A. Ranavolo, and V. Molinaro. 2020. “The Effects of Upper-Body Exoskeletons on Human Metabolic Cost and Thermal Response during Work Tasks - A Systematic Review.” International Journal of Environmental Research and Public Health 17 (20): 7374. doi:https://doi.org/10.3390/ijerph17207374.
- d’Elia, Nicolò, Federica Vanetti, Marco Cempini, Guido Pasquini, Andrea Parri, Marco Rabuffetti, Maurizio Ferrarin, Raffaele Molino Lova, and Nicola Vitiello. 2017. “Physical Human-Robot Interaction of an Active Pelvis Orthosis: Toward Ergonomic Assessment of Wearable Robots.” Journal of NeuroEngineering and Rehabilitation 14 (1): 29. doi:https://doi.org/10.1186/s12984-017-0237-y.
- Dong, W., C. Liu, Q. Zhang, and C. Xiong. 2019. “Design and Evaluation of an Active Ankle Exoskeleton in Gait Assistance.” IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Hong Kong, China, 318–322. doi:https://doi.org/10.1109/AIM.2019.8868740.
- Fox, S., O. Aranko, J. Heilala, and P. Vahala. 2019. “Exoskeletons: Comprehensive, Comparative and Critical Manufacturing Performance.” Journal of Manufacturing Technology Management 31(6). doi:https://doi.org/10.1108/JMTM-01-2019-0023.
- Gabardi, M., M. Solazzi, D. Leonardis, and A. Frisoli. 2018. “Design and Evaluation of a Novel 5 DoF Underactuated Thumb-Exoskeleton.” IEEE Robotics and Automation Letters 3 (3): 2322–2329. doi:https://doi.org/10.1109/LRA.2018.2807580.
- Galle, S., P. Malcolm, W. Derave, and D. De Clercq. 2014. “Enhancing Performance During Inclined Loaded Walking With a Powered Ankle-Foot Exoskeleton.” European Journal of Applied Physiology 114 (11): 2341–2351. doi:https://doi.org/10.1007/s00421-014-2955-1.
- Gams, A., T. Petric, T. Debevec, and J. Babic. 2013. “Effects of Robotic Knee Exoskeleton on Human Energy Expenditure.” IEEE Transactions on Bio-Medical Engineering 60 (6): 1636–1644. doi:https://doi.org/10.1109/TBME.2013.2240682.
- Gillette, J. C., and M. L. Stephenson. 2019. “Electromyographic Assessment of a Shoulder Support Exoskeleton During on-Site Job Tasks.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3-4): 302–310. doi:https://doi.org/10.1080/24725838.2019.1665596.
- Godwin, A. A., J. M. Stevenson, M. J. Agnew, A. L. Twiddy, M. Abdoli-Eramaki, and C. A. Lotz. 2009. “Testing the Efficacy of an Ergonomic Lifting Aid at Diminishing Muscular Fatigue in Women Over a Prolonged Period of Lifting.” International Journal of Industrial Ergonomics 39 (1): 121–126. doi:https://doi.org/10.1016/j.ergon.2008.05.008.
- Graham, R.B., M. J. Agnew, and J. M. Stevenson. 2009. “Effectiveness of an on-Body Lifting Aid at Reducing Low Back Physical Demands During an Automotive Assembly Task: Assessment of EMG Response and User Acceptability.” Applied Ergonomics 40 (5): 936–942. doi:https://doi.org/10.1016/j.apergo.2009.01.006.
- Grazi, L., E. Trigili, G. Proface, F. Giovacchini, S. Crea, and N. Vitiello. 2020. “Design and Experimental Evaluation of a Semi-Passive Upper-Limb Exoskeleton for Workers with Motorized Tuning of Assistance.” IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society 28 (10): 2276–2285. doi:https://doi.org/10.1109/TNSRE.2020.3014408.
- Han, M., B. J. Shi, S. J. Wang, T. J. Li, J. B. Feng, and T. Ma. 2020. “Parameter Optimization and Experimental Analysis of Passive Energy Storage Power-Assisted Exoskeleton.” Mathematical Problems in Engineering 2020: 1–11. doi:https://doi.org/10.1155/2020/5074858.
- Hefferle, M., M. Lechner, K. Kluth, and M. Christian. 2020. “Development of a Standardized Ergonomic Assessment Methodology for Exoskeletons Using Both Subjective and Objective Measurement Techniques.” In International Conference on Applied Human Factors and Ergonomics, 49–59. Springer, Cham. doi:https://doi.org/10.1007/978-3-030-20467-9_5.
- Hensel, R., and M. Keil. 2019. “Subjective Evaluation of a Passive Industrial Exoskeleton for Lower-Back Support: A Field Study in the Automotive Sector.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 213–221. doi:https://doi.org/10.1080/24725838.2019.1573770.
- Hoffmann, N., A. Argubi-Wollesen, C. Linnenberg, and R. Weidner. 2019. “Towards a Framework for Evaluating Exoskeletons.” In Production at the Leading Edge of Technology, edited by J. P. Wulfsberg, W. Hintze, and B. A. Behrens, 441–450. Berlin, Heidelberg: Springer. doi:https://doi.org/10.1007/978-3-662-60417-5_44.
- Huysamen, K., T. Bosch, M. de Looze, K. S. Stadler, E. Graf, and L. W. O’Sullivan. 2018. “Evaluation of a Passive Exoskeleton for Static Upper Limb Activities.” Applied Ergonomics 70: 148–155. doi:https://doi.org/10.1016/j.apergo.2018.02.009.
- Huysamen, K., M. de Looze, T. Bosch, J. Ortiz, S. Toxiri, and L. W. O’Sullivan. 2018. “Assessment of an Active Industrial Exoskeleton to Aid Dynamic Lifting and Lowering Manual Handling Tasks.” Applied Ergonomics 68: 125–131. doi:https://doi.org/10.1016/j.apergo.2017.11.004.
- Iranzo, S., A. Piedrabuena, D. Iordanov, U. Martinez-Iranzo, and J. M. Belda-Lois. 2020. “Ergonomics Assessment of Passive Upper-Limb Exoskeletons in an Automotive Assembly Plant.” Applied Ergonomics 87: 103120. doi:https://doi.org/10.1016/j.apergo.2020.103120.
- Ito, T., K. Ayusawa, E. Yoshida, and H. Kobayashi. 2018. “Evaluation of Active Wearable Assistive Devices with Human Posture Reproduction Using a Humanoid Robot.” Advanced Robotics 32 (12): 635–645. doi:https://doi.org/10.1080/01691864.2018.1490200.
- Jung, Y., and J. Bae. 2015. “Kinematic Analysis of a 5-DOF Upper-Limb Exoskeleton with a Tilted and Vertically Translating Shoulder Joint.” IEEE/ASME Transactions on Mechatronics 20 (3): 1428–1439. doi:https://doi.org/10.1109/TMECH.2014.2346767.
- Junius, K., N. Lefeber, E. Swinnen, B. Vanderborght, and D. Lefeber. 2018. “Metabolic Effects Induced by a Kinematically Compatible Hip Exoskeleton during STS.” IEEE Transactions on Bio-medical Engineering 65 (6): 1399–1409. doi:https://doi.org/10.1109/TBME.2017.2754922.
- Kang, I., H. Hsu, and A. Young. 2019. “The Effect of Hip Assistance Levels on Human Energetic Cost Using Robotic Hip Exoskeletons.” IEEE Robotics and Automation Letters 4 (2): 430–437. doi:https://doi.org/10.1109/LRA.2019.2890896.
- Kermavnar, Tjaša, Aijse W. de Vries, Michiel P. de Looze, and Leonard W. O’Sullivan. 2021. “Effects of Industrial Back-Support Exoskeletons on Body Loading and User Experience: An Updated Systematic Review.” Ergonomics 64 (6): 685–711. doi:https://doi.org/10.1080/00140139.2020.1870162.
- Kim, H.J., D. H. Lim, W. S. Kim, and C. S. Han. 2020. “Development of a Passive Modular Knee Mechanism for a Lower Limb Exoskeleton Robot and Its Effectiveness in the Workplace.” International Journal of Precision Engineering and Manufacturing 21 (2): 227–236. doi:https://doi.org/10.1007/s12541-019-00217-7.
- Kim, H. J., J. Noh, and W. Yang. 2020. “Knee-Assistive Robotic Exoskeleton (KARE-1) Using a Conditionally Singular Mechanism for Industrial Field Applications.” Applied Sciences 10 (15): 5141. doi:https://doi.org/10.3390/app10155141.
- Kim, S., S. Madinei, M. M. Alemi, D. Srinivasan, and M. A. Nussbaum. 2020. “Assessing the Potential for Undesired Effects of Passive Back-Support Exoskeleton Use during a Simulated Manual Assembly Task: Muscle Activity, Posture, Balance, Discomfort, and Usability.” Applied Ergonomics 89: 103194. doi:https://doi.org/10.1016/j.apergo.2020.103194.
- Kim, S., and M. A. Nussbaum. 2019. “A Follow-Up Study of the Effects of an Arm Support Exoskeleton on Physical Demands and Task Performance during Simulated Overhead Work.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 163–174. doi:https://doi.org/10.1080/24725838.2018.1551255.
- Kim, S., M. A. Nussbaum, M. I. Mokhlespour Esfahani, M. M. Alemi, B. Jia, and E. Rashedi. 2018. “Assessing the Influence of a Passive, Upper Extremity Exoskeletal Vest for Tasks Requiring Arm Elevation: Part II – “Unexpected” Effects on Shoulder Motion, Balance, and Spine Loading.” Applied Ergonomics 70: 323–330. doi:https://doi.org/10.1016/j.apergo.2018.02.024.
- Koopman, Axel S., Idsart Kingma, Michiel P. de Looze, and Jaap H. van Dieën. 2020. “Effects of a Passive Back Exoskeleton on the Mechanical Loading of the Low-Back During Symmetric Lifting.” Journal of Biomechanics 102: 109486. doi:https://doi.org/10.1016/j.jbiomech.2019.109486.
- Koopman, Axel S., Idsart Kingma, Gert S. Faber, Michiel P. de Looze, and Jaap H. van Dieën. 2019. “Effects of a Passive Exoskeleton on the Mechanical Loading of the Low Back in Static Holding tasks.” Journal of Biomechanics 83: 97–103. doi:https://doi.org/10.1016/j.jbiomech.2018.11.033.
- Koopman, A. S., M. Näf, S. J. Baltrusch, I. Kingma, C. Rodriguez-Guerrero, J. Babič, M. P. Looze de, and J. H. van Dieën. 2020. “Biomechanical Evaluation of a New Passive Back Support Exoskeleton.” Journal of Biomechanics 105: 109795. doi:https://doi.org/10.1016/j.jbiomech.2020.109795.
- Koopman, A.S., S. Toxiri, V. Power, I. Kingma, J. H. van Dieen, J. Ortiz, and M. P. de Looze. 2019. “The Effect of Control Strategies for an Active Back-Support Exoskeleton on Spine Loading and Kinematics During Lifting.” Journal of Biomechanics 91: 14–22. doi:https://doi.org/10.1016/j.jbiomech.2019.04.044.
- Kozinc, Z., S. Baltrusch, H. Houdijk, and N. Sarabon. 2020. “Reliability of a Battery of Tests for Functional Evaluation of Trunk Exoskeletons.” Applied Ergonomics 86: 103117. doi:https://doi.org/10.1016/j.apergo.2020.103117.
- Li, B., B. Yuan, S. Tang, Y. Mao, D. Zhang, C. Huang, and B. Tan. 2018. “Biomechanical Design Analysis and Experiments Evaluation of a Passive Knee-Assisting Exoskeleton for Weight-Climbing.” Industrial Robot: An International Journal 45 (4): 436–445. doi:https://doi.org/10.1108/IR-11-2017-0207.
- Li, H. X., W. M. Cheng, F. Liu, M. K. Zhang, and K. Wang. 2018. “The Effects on Muscle Activity and Discomfort of Varying Load Carriage With and Without an Augmentation Exoskeleton.” Applied Sciences 8 (12): 2638. doi:https://doi.org/10.3390/app8122638.
- Li, Y., C. Xu, X. R. Guan, Z. Li, and H. B. Li. 2019. “Experimental Verification of the Effect of Human Lower Extremity Exoskeleton.” Journal of Mechanical Science and Technology 33 (8): 3999–4004. doi:https://doi.org/10.1007/s12206-019-0744-9.
- Liu, C., H. B. Liang, N. Ueda, P. R. Li, Y. Fujimoto, and C. Zhu. 2020. “Functional Evaluation of a Force Sensor-Controlled Upper-Limb Power-Assisted Exoskeleton with High Backdrivability.” Sensors 20 (21): 6379. doi:https://doi.org/10.3390/s20216379.
- Looze, M. P., T. de Bosch, F. Krause, K. S. Stadler, and L. W. O’Sullivan 2016. “Exoskeletons for Industrial Application and Their Potential Effects on Physical Work Load.” Ergonomics 59 (5): 671–681. doi:https://doi.org/10.1080/00140139.2015.1081988.
- Lotz, C.A., M. J. Agnew, A. A. Godwin, and J. M. Stevenson. 2009. “The Effect of an on-Body Personal Lift Assist Device (PLAD) on Fatigue During a Repetitive Lifting Task.” Journal of Electromyography and Kinesiology: Official Journal of the International Society of Electrophysiological Kinesiology 19 (2): 331–340. doi:https://doi.org/10.1016/j.jelekin.2007.08.006.
- Luger, T., T. J. Cobb, R. Seibt, M. A. Rieger, and B. Steinhilber. 2019. “Subjective Evaluation of a Passive Lower-Limb Industrial Exoskeleton Used During Simulated Assembly.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 175–184. doi:https://doi.org/10.1080/24725838.2018.1560376.
- Madinei, S., M. M. Alemi, S. Kim, D. Srinivasan, and M. A. Nussbaum. 2020a. “Biomechanical Evaluation of Passive Back-Support Exoskeletons in a Precision Manual Assembly Task: "Expected" Effects on Trunk Muscle Activity, Perceived Exertion, and Task Performance.” Human Factors 62 (3): 441–457. doi:https://doi.org/10.1177/0018720819890966.
- Madinei, S., M. M. Alemi, S. Kim, D. Srinivasan, and M. A. Nussbaum. 2020b. “Biomechanical Assessment of Two Back-Support Exoskeletons in Symmetric and Asymmetric Repetitive Lifting with Moderate Postural Demands.” Applied Ergonomics 88: 103156. doi:https://doi.org/10.1016/j.apergo.2020.103156.
- Maurice, P., J. Camernik, D. Gorjan, B. Schirrmeister, J. Bornmann, L. Tagliapietra, C. Latella, D. Pucci, L. Fritzsche, S. Ivaldi, and J. Babic. 2020. “Objective and Subjective Effects of a Passive Exoskeleton on Overhead Work.” IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society 28 (1): 152–164. doi:https://doi.org/10.1109/TNSRE.2019.2945368.
- Nabeshima, C., K. Ayusawa, C. Hochberg, and E. Yoshida. 2018. “Standard Performance Test of Wearable Robots for Lumbar Support.” IEEE Robotics and Automation Letters 3 (3): 2182–2189. doi:https://doi.org/10.1109/LRA.2018.2810860.
- Nelson, A. J., P. T. Hall, K. R. Saul, and D. L. Crouch. 2020. “Effect of Mechanically Passive, Wearable Shoulder Exoskeletons on Muscle Output During Dynamic Upper Extremity Movements: A Computational Simulation Study.” Journal of Applied Biomechanics 36: 1–67. doi:https://doi.org/10.1123/jab.2018-0369.
- Otten, B., R. Weidner, and A. Argubi-Wollesen. 2018. “Evaluation of a Novel Active Exoskeleton for Tasks at or above Head Level.” IEEE Robotics and Automation Letters 3 (3): 2408–2415. doi:https://doi.org/10.1109/LRA.2018.2812905.
- Pacifico, I., A. Scano, E. Guanziroli, M. Moise, L. Morelli, A. Chiavenna, D. Romo, S. Spada, G. Colombina, F. Molteni, F. Giovacchini, N. Vitiello, and S. Crea. 2020. “An Experimental Evaluation of the Proto-MATE: A Novel Ergonomic Upper-Limb Exoskeleton to Reduce Workers’ Physical Strain.” IEEE Robotics & Automation Magazine 27 (1): 54–65. doi:https://doi.org/10.1109/MRA.2019.2954105.
- Pan, D.L., F. Gao, and Y. J. Miao. 2014. “Dynamic Research and Analyses of a Novel Exoskeleton Walking with humanoid gaits.” Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science 228 (9): 1501–1511. doi:https://doi.org/10.1177/0954406213509611.
- Pillai, M. V., L. van Engelhoven, and H. Kazerooni. 2020. “Evaluation of a Lower Leg Support Exoskeleton on Floor and below Hip Height Panel Work.” Human Factors 62 (3): 489–500. doi:https://doi.org/10.1177/0018720820907752.
- Pinto-Fernandez, D., D. Torricelli, Carmen del, M. Sanchez-Villamanan, F. Aller, K. Mombaur, R. Conti, N. Vitiello, J.C. Moreno, and J.L. Pons. 2020. “Performance Evaluation of Lower Limb Exoskeletons: A Systematic Review.” IEEE Transactions on Neural Systems and Rehabilitation Engineering : A Publication of the IEEE Engineering in Medicine and Biology Society 28 (7): 1573–1583. doi:https://doi.org/10.1109/TNSRE.2020.2989481.
- Ralfs, L., N. Hoffmann, and R. Weidner. 2021. “Approach of a Decision Support Matrix for the Implementation of Exoskeletons in Industrial Workplaces.” In Annals of Scientific Society for Assembly, Handling and Industrial Robotics, edited by T. Schüppstuhl, K. Tracht, and A. Raatz. Springer, Cham: Springer Nature.
- Rashedi, E., S. Kim, M. A. Nussbaum, and M. J. Agnew. 2014. “Ergonomic Evaluation of a Wearable Assistive Device for Overhead Work.” Ergonomics 57 (12): 1864–1874. doi:https://doi.org/10.1080/00140139.2014.952682.
- Sadler, E. M., R. B. Graham, and J. M. Stevenson. 2011. “The Personal Lift-Assist Device and Lifting Technique: A Principal Component Analysis.” Ergonomics 54 (4): 392–402. doi:https://doi.org/10.1080/00140139.2011.556259.
- Schmalz, T., J. Schandlinger, M. Schuler, J. Bornmann, B. Schirrmeister, A. Kannenberg, and M. Ernst. 2019. “Biomechanical and Metabolic Effectiveness of an Industrial Exoskeleton for Overhead Work.” International Journal of Environmental Research and Public Health 16 (23): 4792. doi:https://doi.org/10.3390/ijerph16234792.
- Schroeter, F., S. T. Kähler, Z. Yao, T. Jacobsen, and R. Weidner. 2020. “Cognitive Effects of Physical Support Systems: A Study of Resulting Effects for Tasks at and above Head Level Using Exoskeletons.” In Annals of Scientific Society for Assembly, Handling and Industrial Robotics, edited by T. Schüppstuhl, K. Tracht, and D. Henrich, 149–159. Berlin, Heidelberg: Springer Vieweg. doi:https://doi.org/10.1007/978-3-662-61755-7_14.
- Shamaei, K., M. Cenciarini, A. A. Adams, K. N. Gregorczyk, J. M. Schiffman, and A. M. Dollar. 2014. “Design and Evaluation of a Quasi-Passive Knee Exoskeleton for Investigation of Motor Adaptation in Lower Extremity Joints.” IEEE Transactions on Bio-Medical Engineering 61 (6): 1809–1821. doi:https://doi.org/10.1109/TBME.2014.2307698.
- Smets, M. 2019. “A Field Evaluation of Arm-Support Exoskeletons for Overhead Work Applications in Automotive Assembly.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 192–198. doi:https://doi.org/10.1080/24725838.2018.1563010.
- Spada, Stefania, Lidia Ghibaudo, Silvia Gilotta, Laura Gastaldi, and Maria Pia Cavatorta. 2017. “Investigation into the Applicability of a Passive Upper-Limb Exoskeleton in Automotive Industry.” Procedia Manufacturing 11: 1255–1262. doi:https://doi.org/10.1016/j.promfg.2017.07.252.
- Theurel, J., and K. Desbrosses. 2019. “Occupational Exoskeletons: Overview of Their Benefits and Limitations in Preventing Work-Related Musculoskeletal Disorders.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 264–280. doi:https://doi.org/10.1080/24725838.2019.1638331.
- Toxiri, S., A. S. Koopman, M. Lazzaroni, J. Ortiz, V. Power, M. P. de Looze, L. O’Sullivan, and D. G. Caldwell. 2018. “Rationale, Implementation and Evaluation of Assistive Strategies for an Active Back-Support Exoskeleton.” Frontiers in Robotics and AI 5: 53. doi:https://doi.org/10.3389/frobt.2018.00053.
- Ulrey, B. L., and F. A. Fathallah. 2013. “Effect of a Personal Weight Transfer Device on Muscle Activities and Joint Flexions in the Stooped posture.” Journal of Electromyography and Kinesiology: Official Journal of the International Society of Electrophysiological Kinesiology 23 (1): 195–205. doi:https://doi.org/10.1016/j.jelekin.2012.08.014.
- van Engelhoven, L., N. Poon, H. Kazerooni, D. Rempel, A. Barr, and C. Harris-Adamson. 2019. “Experimental Evaluation of a Shoulder-Support Exoskeleton for Overhead Work: Influences of Peak Torque Amplitude, Task, and Tool Mass.” IISE Transactions on Occupational Ergonomics and Human Factors 7 (3–4): 250–263. doi:https://doi.org/10.1080/24725838.2019.1637799.
- Wang, W.,. J. Y. Chen, Y. D. Ji, W. Jin, J. T. Liu, and J. J. Zhang. 2020. “Evaluation of Lower Leg Muscle Activities during Human Walking Assisted by an Ankle Exoskeleton.” IEEE Transactions on Industrial Informatics 16 (11): 7168–7176. doi:https://doi.org/10.1109/TII.2020.2974232.
- Wang, X., Q. Song, X. Wang, and P. Liu. 2018. “Kinematics and Dynamics Analysis of a 3-DOF Upper-Limb Exoskeleton with an Internally Rotated Elbow Joint.” Applied Sciences 8 (3): 464. doi:https://doi.org/10.3390/app8030464.
- Wei, W., W. Wang, Z. C. Qu, J. H. Gu, X. C. Lin, and C. F. Yue. 2020. “The Effects of a Passive Exoskeleton on Muscle Activity and Metabolic Cost of Energy.” Advanced Robotics 34 (1): 19–27. doi:https://doi.org/10.1080/01691864.2019.1707708.
- Weidner, R., A. Argubi-Wollesen, A. Karafillidis, and B. Otten. 2017. “Human-Machine Integration as Support Relation: Individual and Task-Related Hybrid Systems in Industrial Production.” i-com 16 (2): 143–152. doi:https://doi.org/10.1515/icom-2017-0019.
- Weidner, R., and N. Hoffmann. 2020. “Technische Unterstützungssysteme – Menschen gewollt.” In Mensch und Technik – Perspektiven einer zukunftsfähigen Gesellschaft, edited by S. Hartard, A. Schaffer, 225–246. Marburg: Metropolis.
- Weidner, R., and A. Karafillidis. 2018. “Distinguishing Support Technologies. A General Scheme and Its Application to Exoskeletons.”. In Developing Support Technologies. Biosystems & Biorobotics, edited by A. Karafillidis and R. Weidner, 85–100. Springer, Cham: Springer. doi:https://doi.org/10.1007/978-3-030-01836-8_8.
- Weidner, R., N. Kong, and J. P. Wulfsberg. 2013. “Human Hybrid Robot: A New Concept for Supporting Manual Assembly Tasks.” Production Engineering 7 (6): 675–684. doi:https://doi.org/10.1007/s11740-013-0487-x.
- Weidner, R., C. Linnenberg, N. Hoffmann, G. Prokop, and V. Edwards. 2020. “Exoskelette für den industriellen Kontext: Systematisches Review und Klassifikation.” In 66. Frühjahrskongress der Gesell-schaft für Arbeitswissenschaft e.V., Berlin.
- Weston, E. B., M. Alizadeh, G. G. Knapik, X. Wang, and W. S. Marras. 2018. “Biomechanical Evaluation of Exoskeleton Use on Loading of the Lumbar Spine.” Applied Ergonomics 68: 101–108. doi:https://doi.org/10.1016/j.apergo.2017.11.006.
- Whitfield, B. H., P.A. Costigan, J. M. Stevenson, and C. L. Smallman. 2014. “Effect of an on-Body Ergonomic Aid on Oxygen Consumption During a Repetitive Lifting Task.” International Journal of Industrial Ergonomics 44 (1): 39–44. doi:https://doi.org/10.1016/j.ergon.2013.10.002.
- Yan, Q., J. Zhang, and K. Qi. 2018. “Structure Design and Kinematics Analysis of a Novel Unpowered Load-Carrying Lower Extremity Exoskeleton with Parallel Topology.” Mathematical Problems in Engineering 2018: 1–10. doi:https://doi.org/10.1155/2018/4128520.
- Yang, L., P. Yin, and S. G. Qu. 2020. “Effects of an Unpowered Exoskeleton Device for Assisting Manual Workers.” Basic & Clinical Pharmacology & Toxicology 126: 068.
- Yin, P., L. Yang, S. G. Qu, and C. Wang. 2020. “Effects of a Passive Upper Extremity Exoskeleton for Overhead Tasks.” Journal of Electromyography and Kinesiology: Official Journal of the International Society of Electrophysiological Kinesiology 55: 102478. doi:https://doi.org/10.1016/j.jelekin.2020.102478.
- Yong, X., Z. Yan, C. Wang, C. Wang, N. Li, and X. Wu. 2019. “Ergonomic Mechanical Design and Assessment of a Waist Assist Exoskeleton for Reducing Lumbar Loads during Lifting Task.” Micromachines 10 (7): 463. doi:https://doi.org/10.3390/mi10070463.
- Yu, S., H. Lee, W. Kim, and C. Han. 2016. “Development of an Underactuated Exoskeleton for Effective Walking and Load-Carrying Assist.” Advanced Robotics 30 (8): 535–551. doi:https://doi.org/10.1080/01691864.2015.1135080.
- Zhang, H.H., A. Kadrolkar, and F. C. Sup. 2016. “Design and Preliminary Evaluation of a Passive Spine Exoskeleton.” Journal of Medical Devices 10 (1): 1–8. doi:https://doi.org/10.1115/1.4031798.