Test-retest reliability of the assessment of time management skills (ATMS-S) in adults with neurodevelopmental disorders

Abstract

Background

Time-management skills are essential in handling daily life, and adults with neurodevelopmental disorders often have difficulty with these skills. Therefore, interventions targeting such skills are common in occupational therapy. The Assessment of Time-Management Skills (ATMS) is a self-rated instrument for measuring time-management skills.

Aim

This study aims to evaluate the test-retest reliability of the Swedish version of the ATMS (ATMS-S).

Materials and methods

A total of 33 participants with neurodevelopmental disorders and difficulty with time management completed the test twice, approximately 1 week apart. The test-retest reliability for the three subscales in the ATMS-S was analyzed using intraclass correlation coefficients. The smallest detectable change was calculated to determine the precision of individual ATMS units.

Results and conclusion

The results showed overall moderate to good stability for the measures. The intraclass correlation coefficients were 0.79 (time management), 0.82 (organization and planning), and 0.50 (regulation of emotions) for the three subscales, and the smallest detectable changes were 9.5, 6.9, and 15.7 ATMS units for the respective subscales. These results suggest that the ATMS-S is a sufficiently stable tool for measuring time management and organization and planning skills in adults with neurodevelopmental disorders, but may be less reliable for measuring emotional regulation.

Keywords:

• ADHD
• autism spectrum disorder
• emotional regulation
• occupational therapy
• planning and organization
• psychometrics
• time management

Introduction

Time management includes behaviours aimed at the efficient use of time in goal-directed activities and involves planning and organizing actions into a sequence [1,2]. It is a part of executive functioning, which also includes working memory, inhibitory control (i.e. self-control and the ability to resist temptation), and cognitive flexibility (i.e. the ability to think about something in more than one way) [3]. Time-management skills are essential in handling the demands of society today [4], and good time-management skills are associated with lower stress, increased well-being, and better work and school performance [1,4]. People with limitations in time management encounter difficulties in carrying out the activities they want and must be able to perform in everyday life and their participation in social life can be negatively affected as a result.

Limitations in time-management skills and in organization and planning skills are common problems for people with neurodevelopmental disorders such as attention deficit hyperactivity disorder (ADHD) and autism spectrum disorder (ASD) [5–8]. In ADHD and ASD, the pattern of co-occurring symptoms changes over an individual’s lifespan, but impairments in executive functioning and thus in time-management skills are likely to continue across all ages [9]. The consequences of impaired executive functioning include problems in handling studies, work life, and home life because such activities require several abilities related to executive functioning [4,10]. Specific consequences in relation to time management in daily life include not handing in assignments on time, being late for work, or missing important appointments. In addition, people with ADHD and ASD can have poor self-control and emotional regulation [11,12], which can negatively affect tasks such as time management and planning and organizing activities [13]. Considering the consequences of activity performance and participation time management is important to intervene within occupational therapy.

To evaluate the benefits of occupational therapy, psychometrically sound instruments that can evaluate changes over time – that is, before and after an intervention – are essential. However, few self-rating instruments are available today that can measure different aspects of time-management skills in adults. The available instruments include Time-S for adults [14], the Zimbardo Time Perspective Inventory (ZTPI) [15], the Time Organization and Participation Scale (TOPS) [16], and the Assessment of Time-Management Skills (ATMS) [17]. Three of these – namely, the ATMS, ZTPI, and Time-S – are available in Swedish. The ZTPI focuses on assessing the client’s perspective of time in the past, present, and future, whereas the ATMS and Time-S focus on daily time management – that is, the ability to handle time in everyday life. Unlike Time-S, the ATMS also measures organization and planning and emotional regulation in relation to time management.

The ATMS was developed in the United States as a standardized way to measure how clients assess their own use of strategies, tools, and self-awareness to resolve practical situations involving time management [17]. The psychometric properties of the original version of ATMS have been evaluated in a general population, and the analysis revealed good internal consistency and very good stability over time [17]. The ATMS has been translated into Swedish (referred to herein as the ATMS-S) and adapted to a Swedish context [18]. Janeslätt et al. evaluated the psychometric properties of the ATMS-S through a Rasch analysis and showed that the ATMS-S consists of three subscales: time management, organization and planning, and regulation of emotions [18]. Janeslätt et al. also found that the ATMS-S is useful for clients with cognitive disabilities associated with neurodevelopmental disorders such as ADHD and ASD [18]. Furthermore, the researchers concluded that the ATMS-S gives occupational therapists a structured way to assess time-management skills in clients with these difficulties [18].

For the ATMS-S to be useful for measuring change over time and thus be useful for evaluating occupational therapy interventions, this instrument must be evaluated for test-retest reliability and the smallest detectable change (SDC) [18]. Such results will provide information about the magnitude of the measurement error under stable conditions and the score change needed for an individual client to exceed the measurement error [19]. Thus, the aim of the present study was to evaluate the test-retest reliability and SDC of the ATMS-S for persons with neurodevelopmental disorders.

Material and methods

Recruitment and participants

The inclusion criteria were as follows: age 18 years or over; a diagnosis of ASD, ADHD, or a combination of the two; and the ability to complete the ATMS-S with verbal instructions in combination with visual cues. Intellectual disability was an exclusion criterion. A total of 36 participants were recruited from 12 clinics, which included three outpatient psychiatric clinics, six adult habilitation centres (which provide needs-tested support services for adults with lifelong disabilities), one daily activity centre, one work rehabilitation centre, and one student support centre at a university. A total of 66 people at the 12 clinics met the inclusion criteria during the period of February 2019 to March 2020. These individuals were verbally asked by their treating occupational therapist – or, in one case, by a specialist education teacher – about their interest in participating in the study. Interested individuals (n = 50) were given written and oral information about the study and were offered time to consider their participation before deciding. Out of the 50 interested individuals, 14 declined or dropped out of participation once they became aware that their participation would require two visits to the clinic, and 36 agreed to participate. The characteristics of the participants are provided in Table 1.

Table 1. Demographic characteristics of the participants (n = 33).

Sex, n (%)
Women	18 (55)
Men	15 (45)
Age (years)
Range	19–49
Mean (SD)	31 (9.0)
Median	31
Living arrangement, n (%)
Flat	20 (61)
House	12 (36)
Other	1 (3)
Family, n (%)
Single	13 (39)
Living apart from partner	2 (6)
Living with somebody else	18 (55)
Education, n (%)
Compulsory school	6 (18)
Secondary school	22 (67)
Higher education	5 (15)
Work, n (%)
Not employed	20 (60)
Employed	13 (40)

This study was conducted in accordance with the Declaration of Helsinki [20], and ethical approval was obtained from the Regional Ethical Review Board in Uppsala, Sweden (Dnr 2013/323/1, amended on 2018-11-14).

The assessment of time-management skills – Swedish version

The ATMS-S is a self-rated questionnaire with 27 items intended to measure the client’s experience of how different practical situations related to time management are resolved in that individual’s life. The instrument also includes items describing possible views on the client’s own organizational and time-management skills (17). The questionnaire, which takes approximately 10 min to complete, consists of three subscales: the time-management subscale (11 items); the organization and planning subscale (11 items); and the regulation of emotions subscale (5 items) [18]. The items include statements describing how everyday situations regarding time management are resolved and statements regarding how the client perceives his or her own time-management skills. Every item is rated on a 4-point rating scale from 1 = none of the time to 4 = all of the time. The subscale sum scores can be transformed into ATMS units at the interval level using Rasch measurement analysis [18]. The probabilistic Rasch measurement model is used to evaluate the subscales by positioning items and people along a common measurement line, estimating the intervals between the scores of all items, and using a logarithmic transformation to produce measures in the unit of logits (log-odds probability units). These logits are transformed into the more easily interpreted ATMS units, which range from 0 to 100, where a higher score indicates better time-management skills.

A study-specific questionnaire was used to collect demographic information on gender, age, family, living arrangements, highest level of education, and work situation.

Procedure

After giving informed consent, 36 participants completed the ATMS-S and the study-specific questionnaire with demographic questions. If necessary, the participants received support in reading the ATMS-S items and the demographic questions. After completing the ATMS-S, the participants scheduled a new appointment to complete it a second time. A time period of approximately 7–14 days was chosen between the two tests because the participants’ time management was deemed to remain reasonably stable during such an interval, while the timing was long enough to cause the participants to forget their answers from the first occasion.

Three participants were excluded from the analysis because they did complete the second test outside the clinic. The demographics of the excluded participants did not differ significantly from those of the included ones. Thus, 33 participants were included in the analysis.

Data analysis

The data were analyzed using IBM SPSS version 26. A Shapiro-Wilks test of normality with an analysis of outliers showed that outliers might be present. To detect and manage possible outliers, the recommendations of Aguinis et al. [21] were followed. An outlier was defined as a participant whose results in terms of the total units of each measure were unusually large or small (±2.24 standard deviation, SD) in comparison with other values for the same construct.

Intraclass correlation coefficients (ICCs) for the ATMS were calculated with a single-rater, absolute-agreement, two-way mixed-effects analysis of variance (ANOVA; 3.1) with a 95% confidence interval (CI) [19,22,23]. An ICC greater than 0.90 was considered to be excellent, 0.75–0.90 was considered to be good, 0.50–0.75 was considered to be moderate, and below 0.50 was considered to be poor [23]. A paired t-test was used to calculate the statistical significance of the differences between the tests.

The SDC with a 95% CI was calculated to establish what changes in units were clinically relevant in individual cases – that is, what changes could be classified as true changes and not as changes due to measurement error. To calculate the SDC, the standard error of measurement (SEM)-based method was used [19].

Results

Outliers

The data were found to be normally distributed, except for one participant who fulfilled the criteria for being an outlier. This participant was classified as a single-construct outlier – that is, as a person whose test scores were extremely large or small compared with other scores [21]. This outlier was excluded from further analysis of reliability, so the results are based on 32 participants.

Test-retest reliability

The ICCs of each of the three subscales ranged from moderate to good. Both the time-management subscale and the organization and planning subscale had good reliability coefficients (0.79 and 0.82, respectively), while the regulation of emotions subscale showed moderate reliability (0.50). The 95% CIs are reported in Table 2.

Table 2. ICCs for the ATMS-S and subscales with 95% CI, SEM, and SDC in ATMS units (n = 32).

	ICC (CI 95%)	SEM	SDC
Time-management subscale	0.79 (0.60–0.90)	3.4	9.5
Organization & planning subscale	0.82 (0.67–0.91)	2.5	6.9
Regulation of emotions subscale	0.50 (0.19–0.72)	5.6	15.7

ICC: intraclass correlation coefficient; ATMS-S: Assessment of Time-Management Skills, Swedish version; CI: confidence interval; SEM: standard error of measurement; SDC: smallest detectable change.

Smallest detectable change

There were no systematic changes in units between the tests in the subscales of organization and planning and regulation of emotions. However, in the time-management subscale, the mean retest units were significantly higher than the test units (Test 1: 42.3 and Test 2: 44.1; p = 0.023). The mean units, SD, and 95% CIs for Test 1 and Test 2 are presented in Table 3. The SDC was 9.5 ATMS units for the time-management subscale, 6.9 ATMS units for the organization and planning subscale, and 15.7 ATMS units for the regulation of emotions subscale. This result indicates that, for example, in order to measure the change in an individual client, the change in the client’s ATMS units on the time-management subscale between two test occasions must be 9.5 or greater to be considered a real change.

Table 3. Scores from the two test occasions in ATMS units, mean, and standard deviation (SD) (n = 32).

	Test 1	Test 2
	Mean (SD)	Mean (SD)	p value
Time-management subscale	42.3 (7.6)	44.1 (7.3)	0.023
Organization & planning subscale	54.4 (5.9)	54.2 (5.9)	0.435
Regulation of emotions subscale	47.0 (7.7)	48.7 (8.3)	0.239

ATMS: Assessment of Time-Management Skills; SD: standard deviation.

Discussion

This study explored the test-retest reliability of the three subscales in the ATMS-S. In summary, the evaluation showed mixed results. The time-management and organization and planning subscales’ reliability coefficients showed good stability but had wide 95% CIs, indicating that the stability likely varied between participants. The stability of the regulation of emotions subscale was moderate, with an ICC bordering on poor and a very wide 95% CI. The SDC showed that a change in units of 6%–16%, depending on the subscale, needed to be considered a true change for an individual client – that is, to exceed possible measurement error and not be attributed to daily variations.

The analysis showed good ICCs for the time-management and organization and planning subscales, indicating that they may be a useful assessment for repeated measurements and evaluating change. The ATMS-S is unique among the available instruments measuring time management in Sweden today in that it measures both time management and organization and planning in separate scales; therefore, the ATMS-S is a useful tool for client work with people with impaired executive functioning. Important aspects of human occupation, such as mental ability, habits, and emotional regulation, can be highlighted with the ATMS-S.

The poorer stability of the regulation of emotions subscale compared with the other two subscales has important implications. A probable reason for the low ICC and high SDC is the small number of items in the regulation of emotions subscale. It is generally accepted in classical test theory that a greater number of items increases reliability because the mean random error approaches zero with an increasing number of items [24]; thus, when there are fewer items in a scale, a random error has more influence on the sum score. Therefore, this short subscale is less reliable than the two longer ones, as assessed by the ICC. Due to the poorer stability of the regulation of emotions subscale, a relatively large unit change is required between test occasions in an individual to measure a true change that exceeds the measurement error. On the other hand, because this 5-item subscale has a 0–100 ATMS unit measurement range, a raw score change in only a couple of items results in a relatively large change in ATMS units, which must be taken into account when interpreting the SDC. Despite its moderate reliability, this subscale is valuable. Questions on emotional regulation and its impact on daily time-management skills are important, as many individuals with ADHD experience emotional regulation as a serious obstacle in their daily occupational performance [25]. However, this subscale is likely to be less sensitive to change in an individual over time than the other two subscales.

The ATMS was developed to encourage clients to assess and thereby reflect on their abilities. A possible problem with taking these types of tests more than once is that the test is ‘reactive’ in the sense that a process of self-awareness leads to lower scores the second time (in what is known as ‘response shift’) [22]. This can explain some of the variances that were found between the two tests in this study, but it does not explain why the second measurement of time management showed a significantly higher result than the first – since the opposite result, if anything, was to be expected. Another explanation might be the type of item on this subscale. The items within the time-management subscale are more abstract than those on the other subscales and require the participants to judge their own performance; however, such judgement may vary more from day to day compared with the responses to the planning and organization subscale, in which the items are more concrete and focus on what the client does or does not do. From this perspective, the items in the time-management subscale are more demanding and thus may allow for more interpretation. This possible issue requires further investigation and must be considered when using the ATMS-S in research; when using the ATMS-S for individual clients, however, this issue can be taken into account by applying the SDC.

As a limitation of this study, one participant was defined as an outlier. This participant had extremely high units on both tests and thus could be a typical case in which self-awareness was low [26,27]. Self-awareness is crucial in order to be motivated to change, and interventions for people with low self-awareness should focus on understanding their problems [28]. The ATMS-S can be used to develop clients’ self-awareness of their own difficulties, identify appropriate strategies for use in intervention, and evaluate interventions. A study of adults who went through a neuropsychiatric investigation before receiving an ADHD and/or ASD diagnosis in combination with immediate and integrated services from a multi-modal team found that it is important to involve the clients in such a process [29]. For example, the participants in that study received occupational therapy recommendations on how to organize and structure their daily activities and participated in close dialogue with a professional when evaluating their own limitations. The results of that study showed that the supportive assessment process allowed the participants to gain purposeful insights into their strengths and limitations. The clients also felt understood and listened to as individuals, which confirms the importance of self-assessments [30].

If possible, clinicians should combine questionnaires with other reliable instruments to complement their assessment [19,22]. Self-awareness can be difficult for people with ADHD or ASD; therefore, observations are a relevant addition to the ATMS-S [30,31]. A central aspect of a successful intervention is the client’s self-awareness and ability to adjust to cognitive changes [32]. According to Toglia [33], there are important differences between a client’s self-awareness, capability, and execution, which is why both self-assessment and observation should be used.

Methodological considerations

Of the 66 people invited to participate in this study, 36 agreed to participate, and 32 were included in the final analysis. A larger sample would have been desirable, as the large CIs of the ICCs are likely due to the small sample size making the estimates less precise; nevertheless, the sample size was sufficient to make conclusions about reliability [19,34]. An alternative could have been to include the participants who were initially interested but declined due to having to complete both test occasions at the clinic. Perhaps these participants could have been allowed to complete the second test at home and return the questionnaires by post. However, that would have introduced additional issues, as it would not have been possible to control for the risk of the participants completing the ATMS-S with influence from other people with opinions about how the participant functions.

The relatively large proportion of invited individuals that declined participation before receiving information is not surprising and illustrates a clinical issue faced by occupational therapists and other health professionals in this field. People with cognitive limitations such as ADHD and ASD do not prefer activities in which they are unsure about what the activity entails and what is expected from them. Participating in research may be perceived as an unknown activity, which may have caused some invited individuals to decline participation. For ethical reasons, the invited individuals were not asked why they declined, and their demographic information (e.g. level of ASD or type of ADHD) was not obtained, although it would have been valuable for identifying differences between participants and non-participants. It is possible that some individuals refused participation because they felt stigmatized, in contrast to others who might have viewed participation in the study as an opportunity to obtain support or self-knowledge. Another possible reason is that this population is already subjected to numerous diagnostic tests or administrative tasks, which can be perceived as an ethical dilemma.

In conclusion, this study contributes to earlier psychometric evaluations of the ATMS-S [17] by showing that measurements of time-management skills and planning and organization skills done with the ATMS-S are sufficiently stable in people with neurodevelopmental disorders, whereas measurements of emotional regulation are less stable and thus less likely to be useful for detecting change. The ATMS-S provides occupational therapists with a self-rating instrument for evaluating time-management skills and organization and planning skills before and after interventions. Further studies are still required for the ATMS-S to be used in other populations. Future research could also include studies on using the ATMS-S in a general population, in order to establish criteria for what should be considered typical time-management skills for adults. As mentioned earlier, it would also be interesting to compare individuals with ADHD and those with ASD to determine whether difficulties within the subscales of the ATMS-S differ between these two groups. It will also be important to continuously validate the instrument, such as through cultural validity, because the ways in which we organize, plan, and manage time change with new technologies. Thus, the items in the ATMS-S must correspond with the strategies clients are currently using. Another suggestion for future studies is to develop proxy versions of the ATMS-S. A proxy version would allow the occupational therapist to make an assessment of the client’s time-management skills for comparison with the client’s self-reporting of those skills. This could give the occupational therapist valuable information about the client’s self-awareness and lay a foundation for providing the best possible care.

Acknowledgements

The authors thanks the occupational therapists that performed the data collection. This study was financially supported by Örebro University and the University Health Care Research Center, Region Örebro County.

Disclosure statement

The authors report no conflicts of interest.

Data availability statement

Data can be obtained from the corresponding author upon request.