Is a structured work task application for the assessment of work performance in a constructed environment, useful for patients with attention deficits?

Abstract

Background

The purpose of our study was to develop a Structured Work Task application for the Assessment of Work Performance for patients with attention deficits.

Material and methods

We developed a computer-based registration task titled the Attention-demanding Registration Task. It had a structured administrative procedure with additional scoring regarding time and accuracy, also linked to the original scoring of the Assessment of Work Performance. We evaluated the Attention-demanding Registration Task for content validity. Furthermore, we investigated it concerning sensitivity and specificity in patients with attention deficits due to acquired brain injury (n = 65) against a comparison group of healthy people (n = 47).

Results

Our investigation on content validity using the Assessment of Work Characteristics confirmed that the Attention-demanding Registration Task sets high demands on process skills, especially on energy, temporal organization, and adaptation. The Attention-demanding Registration Task showed high sensitivity and specificity in differing between patients with attention deficits and a healthy working group; nine out of ten participants were placed in the correct group.

Conclusions

To assess work performance, the use of a Structured Work Task application, the Attention-demanding Registration Task, linked with the Assessment of Work Performance, proved to be sensitive to attention deficits.

Implications for rehabilitation

• A Structured Work Task application for the Assessment of Work Performance was developed for use in people with attention deficits and showing a high degree of sensitivity and specificity.

• Linking performance time and accuracy to the Assessment of Work Performance scoring and providing a guide for linking task performance to the Assessment of Work Performance skills in addition to the usual observations performed, may increase scoring accuracy.

• Reference data for a comparison group of healthy subjects are provided.

• The use of the Attention demanding Registration Task, while using the Assessment of Work Performance within clinical practice ensures a more accurate description of process skills in performance.

Keywords:

• Vocational rehabilitation
• registration task
• occupational therapy
• traumatic brain injury
• attention process training
• work assessment
• stroke

Introduction

Most people of working age consider employment a vital area of participation. Therefore, returning to work is often a significant issue for rehabilitation and the goal in many cases [1–5]. Work ability is defined as being in good health and having work-related competence required for specific work-tasks [6–8], and work performance as “our ability to fulfill a worker role” [9]. Assessment of work ability requires consideration of numerous aspects such as personal, environmental, and temporal factors as well as individual capacity. Furthermore, work ability may be affected by illness or injury.

In Sweden, about 50 000 persons survive acquired brain injury (ABI) annually [10]. The two largest diagnostic groups with ABI are traumatic brain injury [11] and stroke [12]. Only about 40% of people working before ABI return to work within two years [3]. ABI typically affects neurological, cognitive, emotional, and behavioral functions [1,3,13,14].

Attention deficits are among the most common cognitive sequelae after ABI [15–18]. Attention can be defined as the ability to process the constant inflow of external and internal information and the capacity to maintain selective or sustained concentration [19]. Therefore, attention deficits commonly lead to difficulties in ignoring distractions and in paying attention to more than one thing at a time [20]. As a result, attention deficits often affect the ability to perform activities of daily living, such as work [21,22].

In many disabilities such as ABI, attention deficit hyperactivity disability, burnout syndrome, schizophrenia, and major depressive disorder, cognitive ability has important implications for performance in work environments and job proficiency [21,23–28]. In line with this, research supports a positive relationship between performance skills and cognitive functions in patients with stroke [29,30], dementia [31], bipolar disease [32] and Alzheimer´s disease [33,34]. In particular, visual attention and visuo-contextual memory show evident correlations with process performance skills, indicating that these are essential underlying functions for activities of daily living [29].

Attention is one cognitive function, that has been successfully improved through systematic training with an evidence-based method such as Attention Process Training (APT). This method combines cognitive training with metacognitive strategies for everyday difficulties after ABI [15,35–37]. Few studies, however, examine the effectiveness of intense attention training on return to work or its effect on work performance [38,39]. Currently, data on the direct relationship between rehabilitation and vocational outcomes after ABI are insufficient [40,41].

In general, a growing body of evidence indicates that performance-based assessments are valid and accurate means of determining work ability, capacity, and body functions [42–44]. Performance-based assessments provide detailed information about skills that limit or support performance [30,31,45]. A potential weakness of performance-based assessments is that they require a high degree of inter-consistency rater judgment [46]. Inconsistency among raters has been studied extensively [47–52].

Our study was a pre-study within a more extensive and prospective randomized controlled trial of 120 consecutive patients with stroke or traumatic brain injury. This extensive randomized trial aimed to examine intensively targeted cognitive rehabilitation of attention in the acute (<4 months) and subacute rehabilitation phases (4–12 months) after ABI. It also aimed at evaluating the effects on function, activity, and participation [53]. The randomized controlled trial was registered at ClinicalTrials.gov Protocol NCT02091453. We evaluated progress using several methods, such as Statistical Process Control [54] and a qualitative interview study [55]. Also, pre-, post- and follow-up measurements of function [56], activity, and participation were conducted. Here, the assessment of work performance and return to work are two focus areas.

The process of finding a suitable instrument for assessing work performance started with a database search performed in 2011. The focus of this search was to investigate different assessments used for assessing work performance after ABI within occupational therapy. We used the keywords “work assessment,” “work ability,” “brain injury,” “skills,” “cognition,” “attentional function” and “functional assessment” in our database searches of PubMed, Web of Science and Psyc Info publications.

Depending on the keyword combinations used, up to 303 hits were received on possible relevant articles. Articles whose abstracts appeared to correspond to our purpose were thoroughly examined, and from their reference lists, additional relevant articles were found. A total of 38 articles were finally selected that met the purpose of our literature search. These articles described 31 types of work ability assessments on functional or activity levels for different diagnostic groups.

However, most identified instruments evaluated physical and mental functioning or physical capacity in performing tasks [57–60]. Others were based on specific written or oral questions to assess general work ability or to identify factors that were potential barriers for work return [42,58]. For instance, we found a couple of vocational evaluations that assessed performance in a variety of non-standardized simulated work tasks for a period of two to five days [4,43]. When it came to instruments with the capacity to assess an individual`s performance in a single work task, which was our primary goal, we only found three instruments; The Assessment of Work Performance (AWP) [61], The Assessment of Motor and Process Skills [62] and The Perceive, Recall, Plan and Perform System for task analysis [4]. Only one of these, AWP, was primarily designed to assess work performance, while the other two were developed primarily for assessing performance in personal care and instrumental activities.

The observation method AWP [61], was selected in order to investigate the effect of intensive attention training on post-ABI work performance, in a constructed environment such as a clinical setting in a hospital. The AWP evaluates how efficiently and appropriately a person performs a task in terms of motor, process, communication, and interaction skills (Table 1) [63]. The AWP is a frequently used observation assessment in vocational rehabilitation and has been chosen as the national assessment instrument to be used by occupational therapists [63,64] of the Swedish Social Insurance Administration. It can be used in various settings and tasks, as it aims to assess a person`s skills when performing a specific task. Initially, the AWP [61] was developed for use in tasks most relevant for the patient, preferably in a real work environment and lacked a structured work task application for assessment in a constructed environment.

Table 1. Work skills assessed in the AWP (n = 14).

Motor skills
Posture	Ability to stabilize and position oneself in relation to environment and task
Mobility	Ability to move the body and body parts in relation to the environment
Coordination	Ability to coordinate body parts movements with each other and the environment
Strength	Ability to use strength/handle objects in an appropriate manner
Physical Energy	Ability to perform and complete a work task within a reasonable time and without becoming physically exhausted
Process skills
Mental Energy	Ability to perform and complete the work with maintained attention and without becoming fatigued
Knowledge	Ability to acquire, learn and use knowledge and tools and perform a work task according to aim and goal
Temporal Organization	Ability to initiate, continue, finish and perform task moments in a logical sequence
Organization of Space and Objects	Ability to organize workspace and tools
Adaptation	Ability to note/react, adjust behavior and adapt the environment as a reaction to perceptual or environmental performance cues
Communication and Interaction Skills
Physicality	Ability to physically communicate and interact with other people
Language	Ability to use language for communication and interaction
Relations	Ability to provide communication and social fellowship with other persons
Information Exchange	Ability to exchange information with others

As the relative difficulty of a selected task will affect an individual´s performance, linear measurements of work performance can neither be generated between testing sessions nor between different individuals [42]. However, no suitable alternative of structured tasks was found in the literature search [4,38,42–44] during the planning or data collection of this study.

The purpose of the present study was to develop a Structured Work Task application for the AWP, aimed for people with attention deficits following ABI.

Hence, the present aims were

1. to develop a Structured Work Task application for the AWP in a constructed environment that requires a high level of attention, for the more extensive randomized controlled trial

2. to investigate the content validity of the Structured Work Task application by using the Assessment of Work Characteristics to establish the demands on specific skills relying on attention

3. to examine how well the Structured Work Task application discriminates between patients on sick-leave with attention deficits due to ABI and a healthy comparison group without attention deficits

4. to investigate the sensitivity and specificity of the Structured Work Task application for the AWP

Materials and methods

Design

The design of this study is methodological and descriptive.

Measure

The AWP has shown sound psychometric properties concerning face, content, construct, and social validity, internal consistency, and utility [61,63–65]. The result of a questionnaire (n = 21) with open-ended questions concerning adequacy, the content of variables, clarity and comprehensibility, manageability, learning time, and layout indicated a satisfactory face validity and utility for use within vocational rehabilitation [61]. Further, in the following study on content validity and utility, most respondents (63%) thought that the AWP covered all possible aspects of observable working skills at least “to a great extent” in a sample of expected users of the AWP in Sweden (n = 67) [65]. Sandqvist et al. [64], using Principal Component Analysis with 365 clients, found that the AWP consisted of two dimensions; one involved motor skills and the other a combination of the process skills together with communication and interaction skills [64]. In a study using Rasch analysis, the 4-point rating scale was confirmed. All items of the AWP loaded on a unidimensional construct of “work performance,” i.e., the item set met the criteria of unidimensionality. The item separation reliability was 0.99, person separation reliability 0.83, and the item functioning was found to be differential across gender and diagnoses [63]. A recent study investigating the psychometric properties of work task application for the AWP indicated social and content validity, as well as utility, for used work tasks [66].

Assessment with the AWP is conducted by observing the individual´s performance during a work task and then summarizing the observations in groups of skills. The following groups were established: motor skills, process skills, and communication and interaction skills. Motor skills are defined as observable operations used to move an object or the performing person. Process skills are defined as observable operations used to organize sensibly and adapt sequences of actions to complete an activity. Lastly, communication and interaction skills are defined as observable operations used to communicate intentions and needs as well as to coordinate social behavior in order to interact with people (Table 1). Several sub-skills are used as guiding concepts in the summary of each skill. The 14 skills are numerically and individually rated on a four-point Likert-type scale where 1 = deficient performance, 2 = inefficient performance, 3 = uncertain performance, and 4 = competent performance. Users who find an item irrelevant, or impossible to assess due to lack of information, mark it as lacking information or not relevant [61]. In more detail, a skill is assessed as being “deficient” if all its parameters are clearly limited; “inefficient” if the performance is ineffective, not purposeful or gives an unacceptable result; “uncertain” if the performance is not fully competent, but the parameters are not clearly limited, and “competent” if the performance is adequate, purposeful and leads to an acceptable result.

Study participants

A cohort of patients on sick-leave with ABI, participating in cognitive rehabilitation and having identified attention deficits was compared with a healthy working comparison group. The patients were part of a more extensive randomized controlled trial [53]. The participants were categorized into occupational groups using the International Classification of Occupation (ISCO-08) [67]. The demographic data are presented in Table 2.

Table 2. Main demographic and clinical characteristics of the study participants.

	Patient group	Comparison group
Participants (n)	65	47
Sex (n:F/M)	36/29	31/16
Mean age (range) (years)	42.7 (24–60)	47.7 (22–60)
Diagnosis (TBI/Stroke)	15/50
Occupation ISCO-08 (n)
1. Managers	6
2. Professionals	20	32
3. Technicians and Associate Professionals	14	11
4. Clerical Support Workers	2
5. Services and Sales Workers	8	1
6. Skilled Agricultural, Forestry, Fishery Workers		1
7. Craft and Related Trades Workers	5	1
8. Plant and Machine Operators and Assemblers	1
9. Elementary Occupations	9	1

Occupation is classified using International Standard Classification of Occupations (ISCO-08); Major Groups.

Patient group

The patient group (PG) (n = 65) comprised a consecutive series of patients with mild-to-moderate stroke or traumatic brain injury. In this study, they had been referred to the clinic for outpatient rehabilitation on average, eight months after ABI (4–29 months). They were in the age range of 24–60 years and understood Swedish well. The PG included managers, medical professionals, teachers, engineers, administrators, preschool teachers, construction workers, and unskilled workers.

Inclusion criteria were: attention deficit, as shown by the diagnostic test in the Attention Process Training method [35]. Cut-off scores were 70% or less on at least two of the five subtests. Scores on the lower average and above (standard scores of seven and above) regarding abstract thinking and reasoning skills were measured with Matrix reasoning from the WAIS-III [68].

Exclusion criteria: aphasia, ongoing psychiatric illness, severe pain, and ongoing substance abuse. Severe bilateral motor impairments or visual impairments that made participation impossible (i.e., carry out assessments and interventions for attention deficits). Neglect as measured with Albert’s test/Line crossing [68] with a cutoff score of ≥2.

One patient lacked data due to severe difficulties performing the task and was therefore excluded from further performance analysis. Also, one patient's AWP evaluation was missing due to an administrative error. Thus, sixty-three patients were included in the analysis. Data from all participants in the CG were included.

Comparison group without ABI

The comparison group (CG) (n = 47) was a convenience sample recruited through the distribution of information about the study on the hospital webpage. The aim was to compare the work performance of a patient group with a sample of active people, working and without ABI using the AWP and a Structured Work Task application. The CG included medical and rehabilitation professionals, administrators, as well as unskilled workers, such as kitchen aid workers, gardeners, and transporters.

Procedure

Development of a Structured Work Task application for the AWP

The following criteria were selected for the Structured Work Task application:

• High demands on the individual's attention span

• Possible to carry out by most individuals regardless of, e.g., age, gender, background or interests

• Relevant to most workplaces in post-industrial countries

• Sensitive enough to distinguish among patients with different work performance levels

• Reliable administration by several different observers

Since we found no Structured Work Task application for use with the AWP in the literature review, the clinically available work tasks used within the rehabilitation program at the clinic were compared with the requirements of our study. High demands on the individual´s attention span was a priority since the intervention project focused on the cognitive function attention and the effect of attention training [53]. In standard clinical practice, a computer-based registration task, the Attention-demanding Registration Task (AdRT), was used during the later phase of rehabilitation. The task was primarily used with a focus on work retraining and rarely together with the AWP.

The AdRT met the criteria above, and the task also seemed to be a suitable simulated work task as computer-based tasks occur at most workplaces today [69,70]. For example, retail sales use database applications to manage their client bases, healthcare visits are electronically recorded, and in agriculture production, farmers monitor production and adjust planting and cultivation to maximize yield with the help of computers [69]. Therefore, the AdRT was chosen for further development for use as a Structured Work Task application with the AWP in this study.

The focus of the more extensive, prospective, randomized, and controlled trial was attention training, requiring a task relying profoundly on attention skills. The AdRT requires visual scanning, accuracy, persistence, focus, and the ability to follow written instructions, all of which skills are important for work performance [5,13,71–73]. The AdRT consists of a transfer of information from 37 handwritten nameplate orders to a data file. Each order is containing the name of customer, order and delivery dates and number of nameplates. When entered into the data file, each order is defined as “a registration.” In clinical practice, the patient works with the task during one or more sessions until it is completed or performs as much of the task as possible during a defined timeframe, usually 60 min.

The task material includes a patient´s task instruction defining what should be registered in the data file and what format. A recurring element of the task is arithmetic. It involves calculating the price of each order based on a current price list and then entering the sum. If needed, the subject can use a calculator, pencil, and ruler.

The flow chart of the developmental process of the AdRT is presented in Figure 1.

Figure 1. The flow chart of the developmental process of the AdRT.

Work task analysis using the assessment of work characteristics

The Assessment of Work Characteristics [74] is an observational instrument to estimate the demands of specific work skills when performing a task and to what degree a person needs to use different skills to perform a task appropriately and efficiently [74]. Initial testing of the Assessment of Work Characteristics has shown good content validity and utility [75,76].

The Assessment of Work Characteristics evaluates the same skills as included in the AWP, but from the point of the work task. Thus, the four-point Likert scale refers to the needs of the actual work task: [1] indicates that the skill is never needed for effective and appropriate performance of the target work task, [2] that the skill is needed occasionally, [3] that it is needed frequently and [4] indicates that this skill is needed continuously [74].

Pilot testing with patients, task modification and quality improvement

During development, the AdRT was initially tested by the first author with two patients who had attention deficits after ABI. The patients could terminate the task before it was completed and if necessary, take breaks. Observation of the pilot patients’ performances revealed that they had difficulties such as keeping focused on the task, searching for information on the nameplate orders, and remembering the task instructions. Moreover, the patients made many errors on the computer recording sheet, and they needed different amounts of time to perform the task. Therefore, the research group concluded that the dimensions accuracy and performance time should be registered more specifically than recommended in the AWP manual in order to complement the observations of skills and thus increase the accuracy of the task analysis.

Based on the pilot patients’ experiences, minor adjustments included text color enhancement and clarifying some information in the patients’ task instruction to avoid misinterpretation. An additional scoring sheet was created to document performance time, number of nameplate orders performed within sixty minutes, and errors made during the performance.

To ensure that the assessment situation and procedure would be standardized, a manual was created consisting of step-by-step information about appropriate assessment environment and instructions to be given before and during the performance. Finally, the computer’s spelling correction file was disabled.

A guide was created to exemplify how difficulties in performing the AdRT could be linked to various skills in the AWP. For example, motor limitations in two-handed tasks affect the ability to use tools (motor skill: Coordination). Being distracted by noise and movement may affect the ability to maintain attention to relevant aspects of the task (process skill: Energy) or to initiate and continue until the task is completed (process skill: Temporal Organization).

The rationale for the revised AWP scoring was to provide support and guidance for the scoring of the skills. The use of objective performance measures, i.e., performance time and the number of errors, was assumed suitable since patients with attention deficits are reported to have difficulties in this area [77–79]. The rationale for collecting data from a healthy comparison group was to obtain an estimate for normal variations in the performance of this task [6].

Assessment of the CG and PG performances on the structured work task application AdRT for the AWP

The first author assessed the performance of the CG (n = 47). Ten occupational therapists trained in the AWP [61] and familiar with the AdRT evaluated the PG (n = 65) performance. Before the studies began, the occupational therapists participated in a seminar where common difficulties during task performance were discussed. Ways in which these difficulties could be linked to different skills assessed in the AWP were also considered. The occupational therapists also received a guide exemplifying how difficulties in performing the AdRT could be linked to various AWP skills.

A strict protocol was used in all data collection. Before initiating the AdRT, the examiner gave written and oral instructions about the task. The participants were instructed to follow the task instructions carefully and perform the task as accurately and as fast as possible. They could take breaks when needed or terminate the task prematurely if necessary. The task needed to be completed within 60 min. The examiner remained present, but the participants were instructed to consult the task instruction first if help was needed. Two short practice trials were given before the start of the AdRT.

Statistics

The SPSS for Windows version 22.0 was used for all statistical analysis. p value of 0.05 or smaller was considered as significant. Data were checked for skewness and kurtosis. Skewed data were analyzed with non-parametric tests. Descriptive statistics such as frequencies, mean, standard deviations, percentiles, and confidence intervals were calculated.

Cronbach’s alpha for the subscales of the Assessment of Work Performance was used to evaluate the internal consistency [80].

The comparison between the PG and the CG was calculated using the non-parametric Mann Whitney U test for the ordinal AWP measures based on the performance of AdRT. In order to describe if achieved treatment effects have a relevant magnitude, the distributions of the test statistics were transformed into effect sizes using an effect size calculator for non-parametric tests [81,82]. Cohen [82] reports the following intervals for effect size magnitude: 0.1–0.3: small effect; 0.3–0.5: intermediate effect; 0.5 and higher: strong effect.

The accuracy of the assessment to discern disease cases from typical cases was evaluated using Receiver Operating Characteristics curve analysis [83]. These analyses were used to investigate whether the assessment could correctly identify those “with disease”/”presence of a characteristic” (sensitivity) and discern them from those “without disease”/”absence of a characteristic” (specificity). A Receiver Operating Characteristics curve informs the trade-off between sensitivity and specificity for different cutoff points and criteria. The area summarizes the overall quality of an assessment under the Receiver Operating Characteristics curve, which ranges between 0 and 1. The larger the area under the Receiver Operating Characteristics, the more accurately the assessment predicts the disease in terms of sensitivity and specificity [84]. Assessments with perfect discrimination (no overlap in the two dimensions) have a Receiver Operating Characteristics curve that passes through the upper left corner (100% sensitivity and 100% specificity).

Ethical considerations

The Regional Ethical Review Board approved the study protocol in Stockholm, Sweden. Participants received oral and written information according to the Declaration of Helsinki regarding their participation in the study, and they all gave written consent.

Results

Content validity and reliability of the AdRT using the AWP

Table 3 presents the result of the AdRT being analyzed with the task analysis instrument Assessment of Work Characteristics. The evaluation confirmed that the AdRT sets high demands on process skills, especially the process skills Energy, Knowledge, Temporal organization and Adaptation, as well as the motor skill Coordination. Competent performance of these skills requires the ability to (a) maintain focus, (b) read and follow instructions, (c) search for information, (d) use elementary arithmetic skills or a calculator, (e) use keyboard and computer in an appropriate way, (f) effectively initiate and complete operations, (g) monitor and if necessary correct the performance of the task, and (h) maintain upright posture.

Table 3. Result of evaluation of the AdRT using the Assessment of Work Characteristics.

Assessment of Work Characteristics Domains	Skills	Assessment of Work Characteristics rating level
Motor skills	Posture	3
	Mobility	2
	Coordination	4
	Strength and handling of object	3
	Physical energy	2
Process skills	Mental energy	4
	Knowledge	4
	Temporal organization	4
	Organization of workplace	2
	Adaptation	4
Communication and	Physical communication and interaction	1
interaction skills	Language	1
	Social contact	1
	Information exchange	1

Level of skills needed to efficient and effective perform the AdRT. Assessment of Work Characteristics rating levels: 1 = never needed, 2 = occasionally needed, 3= frequently needed, 4= always needed.

AdRT is sedentary and therefore, does not place high demands on motor skills except for fine motor skills, such as the coordination required during keyboard work. As the AdRT mainly consists of individual work at a computer, the Assessment of Work Characteristics also shows that this task sets no requirements on the ability to interact and communicate with other people.

Cronbach’s alfa for the subscales of the AWP, using the AdRT task was acceptable (0.74).

The revised AWP scoring

In addition to the observational data, performance time, the number of errors, and the number of registrations during one hour of performance were documented separately (Table 4). The CG performed the 37 registrations in an average time of 43 min and with an average of 6 errors. Data from the CG were further used to establish a reference value for clinical use in the following way: Percentiles for performance time, the total number of registrations and number of errors made were used as guidance in the AWP scoring based on data from the CG (Table 5). Performance within the first 25 percentiles received score 4, performance between the 26–50 percentiles received score 3, performance between the 51–75 percentiles received score 2 and above 75 percentiles score 1. The guidance in scoring AWP was added to the rest of the observation conducted.

Table 4. Reference data for the CG (n = 47): Performance time and types of errors when performing the AdRT.

			95% Confidence Interval of the Difference
	Mean	Median	Lower	Upper	IQR
Performance time (min)*	42.7	42	39.5	45.9	34–50
Performance time per reg (sec)**	69	68	63.83	74.17	55–81
Total numbers of errors***	6	4	4.46	7.54	2–9

*Total time in minutes needed to complete the task.

**Time in seconds needed to complete one nameplate registration.

***Total numbers of errors consist of primary errors (factual inaccuracies that cannot be traced back to earlier incorrectly inserted information), secondary errors (inaccuracies made because of a previously performed primary error) and format errors (the criteria given in the instruction are not followed consistently).

Table 5. Benchmarks for assessing the AdRT with the AWP assessment levels based on the performance of the healthy CG.

AWP Assessment Levels*	Performance time	Total numbers of error
4	≤42	0–4
3–2	43–50	5–9
1	>51	>10

*1 = deficient performance, 2 = inefficient performance, 3 = uncertain performance and 4 = competent performance.

Comparison between PG and CG regarding performance time and number of errors when performing the structured work task AdRT

Comparisons by t-test between PG and CG showed that the patients were significantly slower in their performance (t= −5.1, p < 0.001) and committed significantly more errors of all types (t = 3.8, p < 0.001) than the controls (Table 6). Only 46,2% of the PG:s was able to complete the AdRT within sixty minutes compared to 91.5% of the CG:s. Six patients (9.2%) had to terminate performance before the sixty-minutes limit, due to fatigue or due to severe difficulties performing the task.

Table 6. Comparison of performance during 60 min on the AdRT between CG and PG for performance time, the total number of registrations and errors.

				95% Confidence Interval of the Difference
	n	Mean	Median	Lower	Upper	IQR
Total performance time in minutes
CG	47	42.2	42.0	8.4	15.9	34.0–50.0
PG	60	54.4	60.0			50.5–60.0
Average performance time per reg^a
CG	47	69.2	68.4	42.4	99.1	55.2–81.0
PG	64	140	100.2			84.3–153.3
Total number of registrations
CG	47	36.6	37.0	−7.4	3.2	37.0–37.0
PG	60	31.4	36.5			24.5–37.0
Total number of errors
CG	47	5.9	4.0	3.2	8.9	2.0–9.0
PG	60	12.0	10.0			5.0–16.5
Average number of errors per reg^b
CG	47	0.2	0.1	0.1	0.5	0.1–0.2
PG	64	0.5	0.3			0.1–0.6

All comparisons were significant on p < 0.001 level.

^aAverage time in seconds needed to complete one nameplate registration.

^bAverage number of errors per completed registration despite the type of error.

Comparison of work performance between PG and CG using the structured work task application AdRT for assessment with the AWP

Comparisons for the AWP skills were made by Mann-Whitney U-test. Statistically significant differences were found in a number of process and motor skills (Table 7). The results indicated small effect sizes for three process skills, Energy, Temporal Organization, and Adaptation. Lower performance on Energy (z = 6.3, p < 0.001) indicated less ability to pay attention to, search for, and remember task-relevant information. Impaired performance on Temporal Organization (z = 5.8, p < 0.001) indicated difficulties in initiating and maintaining performance without being distracted and lower scores on Adaptation (z = 5.0, p < 0.001) resulted from impaired ability to notice errors and change methods of performance if needed to increase accuracy and speed.

Table 7. Comparison between PG and CG on performance on AWP subscales motor and process skills, showing median values, interquartile range (IQR), and effect size.

AWP Motor and Process Skills	n	Mean	Median	IQR		Exact Sig. (2-taled)	Effect Size (η^2)
Posture
CG	47	4.0	4	4	4	0.019	0.01
PG	63	4.0	4	4	4
Coordination
CG	47	4.0	4	4	4	<0.001	0.09
PG	63	3.4	4	3	4
Strength and Effort
CG	47	4.0	4	4	4	0.049	0.01
PG	63	3.9	4	4	4
Energy (Motor)
CG	47	4.0	4	4	4	<0.001	0.08
PG	63	3.4	4	3	4
Energy (Process)
CG	47	3.2	3	3	4	<0.001	0.34
PG	63	1.8	1	1	3
Knowledge
CG	47	2.5	2	1	4	<0.001	0.16
PG	63	1.3	1	1	1
Temporal Organization
CG	47	3.9	4	4	4	<0.001	0.24
PG	63	2.8	3	2	4
Organizing Space and Object
CG	47	4.0	4	4	4	0.002	0.02
PG	63	3.8	4	4	4
Adaptation
CG	47	3.1	3	2	4	<0.001	0.21
PG	63	1.9	1	1	3

Sensitivity and specificity of the AWP using the structured work task AdRT

Analyzing the sensitivity and specificity resulted in a prediction area under the curve of 0.91 when using the AWP Process scale only and 0.97 when using both the AWP Motor and Process scale. Thus, the AWP using the Structured Work Task application AdRT, showed very high sensitivity and specificity differentiating between the two participating groups; more than nine out of ten participants belonging to either PG or CG were placed in the correct group based on their performances on the task. The analysis also revealed that the sensitivity (to correctly identify those with ABI) of the AWP for the AdRT was slightly higher than the specificity of the task (to identify correctly those without ABI – Figure 2).

Figure 2. Sensitivity and Specificity for the AdRT assessed with the AWP Motor and Process Skills (A) Sensitivity and Specificity using the AWP Process and Motor Skills, (B) Sensitivity and Specificity using the AWP Process Skills.

Discussion

Statistically significant differences between patients and controls were found in a number of motor – and process skills. The differences were most pronounced in three process skills, Energy, Temporal Organization, and Adaptation. More than nine out of ten participants belonging to either the PG- or the healthy CG were placed in the correct group based on their performance on the AWP Structured Work Task application AdRT. Therefore, the AdRT seemed useful when assessing the work performance of people with attention deficits in a constructed environment. However, the small effect sizes are signaling caution at the interpretation of data by indicating a substantial overlap between patients and controls. AWP scoring is based on a Likert scale, where the numbers lack intrinsic meaning. In such cases, effects sizes are considered more useful to describe possible differences between groups.

The Structured Work Task application for the AWP, AdRT, simulates a common computer-based work task. Today, such tasks are regular at many workplaces [69,70]. The AdRT has many similarities with what is called document literacy tasks [85]. These tasks often include instructions and require abilities such as locating information, processing numbers, or relating different pieces of information to each other. Literacy tasks require process skills dependent on cognitive functions such as visuomotor function, visual attention, processing speed, working memory, and long-term memory [45,86].

Parallel to our work, one of the coworkers, had been engaged in developing the AWP in a more general direction meant for clients with various disabilities [66]. It includes three work tasks: an administrative task, an assembly task, and a sort- and distributing task not focusing on ABI and attention deficits or repetitive measurement. A previous study investigated the content validity of differently structured task applications for the AWP. This study showed that an administrative computer task was reported challenging regarding process skills, but was also perceived as monotonous. However, it provided information regarding physical and mental endurance [66].

The AWP was developed to be used assessing the most relevant work tasks of the patient [61]. However, to be able to measure the quality of a person´s performance objectively, for instance, before and after a period of training, the same task and work environment is needed [42]. One of the aims of the present study was to develop such a Structured Work Task application for the AWP for use regarding people with attention deficits. The evaluation confirmed that the Structured Work Task application AdRT sets high demands on process skills, especially the process skills Energy, Knowledge, Temporal organization and Adaptation and all these skills are essential for attention [29,30].

In order to increase inter-rater consistency [47–52], we decided to create a structured administrative scoring in addition to the observations of AWP. Linking performance time and accuracy of performance to the AWP scoring, and providing a guide for linking task performance to the AWP skills, is supposed to satisfy the need noted in previous research for example of skilled performance related to the four steps in the AWP rating scale [65].

We had to be sure that the Structured Work Task application for AWP made enough demands on attention to be able to discriminate between individuals with or without attention deficits. For the PG with attention deficits participating in our study, the analysis of the process scale of the AWP alone would have been enough to obtain sufficiently high specificity and sensitivity. Adding the AWP motor scale to the analysis resulted in somewhat increased specificity and sensibility values due to problems of fine motor coordination functions in the PG.

Regarding the limitations of the study, first, the restricted range of Likert type scores and the considerable overlap between PG and CG recommend a cautionary interpretation of the results in a clinical context. Further methodological studies are required to increase the discriminatory power on an individual level. The CG was a convenience sample recruited from people working in different areas at one hospital. They were assumed to be neurologically healthy, but this was not explicitly tested. The distribution of professions in both groups was wide. According to Eurostat [70], computer literacy in Sweden is very high (97%); thus it was reasonable to assume that the Structured Work Task application, AdRT, for the AWP will be suitable for all professional categories in working age.

Conclusion and clinical application

The Structured Work Task application AdRT, for AWP with the addition of the measurement of accuracy and performance time gives valuable information regarding the ability to perform an attention-demanding work task of patients with attention deficits after ABI. The use of quantified performance measures, i.e., performance time and accuracy and a structured manual increase the clinical utility. The Structured Work Task application for the AWP enables comparison between patients and the evaluation of treatment effects. Based on these results, the Structured Work Task application for the AWP, AdRT, was considered suitable for the evaluation of the ability to perform an attention-demanding work task and will, therefore, be used in parts of the randomized controlled trial.

Acknowledgements

The authors want to express their sincerest thanks to the participants. The authors would also like to thank the occupational therapists at the Department of Rehabilitation Medicine for their valuable contributions to the assessment of the participants and to Lisbet Broman who helped with statistical analysis.

Disclosure statement

The authors have declared no conflicts of interest.

Additional information

Funding

This work was supported by the grant Karolinska Institutet Foundation for Graduate Students, by grants from Promobilia Foundation (12059, 13009, 13061, 14040, 14094, 17046) and by grants Regional Agreement on Medical Training and Clinical Research between Stockholm County Council and Karolinska Institutet (20120248, 20140057).