Psychometric properties of the Montreal Cognitive Assessment (MoCA) in healthy participants aged 18–70

Abstract

Objectives: The Montreal Cognitive Assessment (MoCA) is a cognitive screen, available in three alternate versions. Aims of the current study were to examine the effects of age, education and intelligence on MoCA performance and to determine the alternate-form equivalence and test–retest reliability of the MoCA, in a group of healthy participants.

Method: In 210 participants, two MoCA versions and an estimator for premorbid intelligence were administered at two time points.

Results: Age, education and estimated premorbid intelligence correlated significantly with the total score (MoCA-TS) and the Memory Index Score (MoCA-MIS). Systematic differences between MoCA version 7.1 and alternate versions 7.2 and 7.3 were only found for the items animal naming, abstract reasoning and sentence repetition. Test–retest reliability of the MoCA-TS was good between 7.1 and 7.2 (ICC: 0.64) and excellent between 7.1 and 7.3 (ICC: 0.82). For the MoCA-MIS, coefficients were poor (ICC: 0.32) to fair (ICC: 0.48), respectively.

Conclusion: Adequate norms are needed that take the effects of age, education and intelligence on MoCA performance into account. All three MoCA versions are largely equivalent based on MoCA-TS and the test–retest reliabilities show that this score is suitable to monitor cognitive change over time. Comparisons of the domain-specific scores should be interpreted with caution.

Key points

• The MoCA total score is a reliable cognitive measure.

• All three MoCA versions are largely equivalent.

• Age, education and intelligence are predictors of MoCA performance in healthy participants.

• Future studies should focus on collecting normative data for age, education and intelligence for use in clinical practice.

Keywords:

• Montreal Cognitive Assessment
• cognitive screener
• test–retest reliability
• alternate-form reliability

Introduction

The Montreal Cognitive Assessment (MoCA; Nasreddine et al. 2005) is one of many available cognitive screening instruments. The screener contains 12 items measuring seven cognitive domains: executive functioning; visuospatial abilities; language; attention, concentration and working memory; abstract reasoning; memory and orientation. Recently, the Memory Index Score (MoCA-MIS) was developed as an additional clinical outcome measure to assess the severity of memory impairments (Julayanont et al. 2014). The MoCA was developed to be more sensitive to mild cognitive impairment in geriatric populations than other screeners, like the Mini Mental State Examination (MMSE; Folstein et al. 1975), and has been translated into nearly 100 languages. Besides geriatrics, the MoCA is widely used in, for instance, substance use, HIV and hepatitis C (Bassiony et al. 2015; Bruijnen et al. 2016; Copersino et al. 2009; Janssen et al. 2015). Although the MoCA has gained wide support, research focussing on its psychometric properties, such as alternate-form-, test–retest- and inter-rater reliability, has yielded mixed findings. Moreover, the influence of educational attainment, age and sex on the test performance has been under debate. A short overview of these mixed findings is discussed below (see also www.mocatest.org or Julayanont et al. 2013).

With respect to educational attainment, Nasreddine et al. (2005) found that years of education affect MoCA performance. As a result, they suggested to add one ‘correction point’ to scores of individuals with 12 years of education or less. Bruijnen et al. (2019) recently proposed a more fine-grained correction method based on the level of education, rather than its duration, where individuals with a low level of education receive two additional points and those with an average level of education receive one additional point. Only one study correlated MoCA scores with general intellectual abilities and found a correlation of 0.64 (Sugarman and Axelrod 2014). With respect to age, Nasreddine et al. (2005) in their study in healthy adults between 55 and 85, did not find this characteristic to be of influence on MoCA performance. However, recent studies have demonstrated a negative correlation between age and MoCA scores in a slightly wider age range of 50–100 years (Apolinario et al. 2018; Malek-Ahmadi et al. 2015; Oren et al. 2014; Yancar Demir and Özcan 2015). Also, an interaction effect between age and education was found, in which younger participants with higher education levels had higher MoCA scores, compared to older participants with lower education levels (Zheng et al. 2012). Normative data correcting for age, education and sex have been presented for several translations (Borland et al. 2017; Larouche et al. 2016; Ojeda et al. 2016; Thomann et al. 2018), while others found no effect of sex on MoCA performance (Apolinario et al. 2018; Kopecek, Stepankova, et al. 2017; Santangelo et al. 2015; Zheng et al. 2012). Although there are some studies presenting normative data, these are currently not as widely used as the aforementioned correction method by Nasreddine et al. (2005).

With respect to alternate-form reliability, relevant for repeated assessment in individuals with the aim to overcome material-specific learning effects, Chertkow et al. (2011) developed two alternate versions of the original version (MoCA alternate versions 7.2 and 7.3). The alternate versions have been translated into nearly 40 languages, including Dutch. The MoCA Total Score (MoCA-TS) of the English, French and several other translations was found to be systematically equivalent across versions (Chertkow et al. 2011; Nasreddine and Patel 2016). However, looking at item level one study in geriatric patients showed systematic differences between the original and both alternate versions for the items figure copy, animal naming and abstract reasoning (Lebedeva et al. 2016), but studies in healthy individuals are lacking.

Finally, the test–retest and inter-rater reliability are important for repeated assessments. The test–retest reliability of the English-language versions, and the German and Czech translations of MoCA 7.1 and 7.2 range from 0.42 to 0.81, for MoCA-TS (Costa et al. 2012; Feeney et al. 2016; Kopecek, Bezdicek, et al. 2017; Wu et al. 2017). The lowest reliability was found after an administration interval of three years (Kopecek, Bezdicek, et al. 2017). The inter-rater reliability of the items alternating trail making, figure copy and clock drawing of MoCA version 7.1 was recently investigated in a large multicentre trial, including 1119 participants (Cumming et al. 2018). Kappa coefficients ranging from 0.46 (hands of the clock) to 0.94 (alternating trail making) were found, where a higher coefficient means better agreement between raters.

As can be seen from this overview, differences in MoCA properties exist between populations and translations, and mixed results have been found for participants with different sociodemographic characteristics. As the MoCA is increasingly being administered to individuals younger than 50, it is important to determine the psychometric properties of the Dutch translations of the MoCA in a wide age range. The aims of the current study are, therefore, to assess psychometric properties of the MoCA in healthy participants. We will investigate: (1) whether demographic variables such as age, sex, educational level and estimated premorbid intelligence are associated with performance on the MoCA; (2) whether systematic differences between MoCA version 7.1 and alternate versions 7.2 or 7.3 can be found and (3) what the test–retest reliability between MoCA version 7.1, and alternate versions 7.2 and 7.3 is.

Methods

Design

A repeated-measures within-subject design was used in which MoCA version 7.1 and one of the alternate versions 7.2 or 7.3 were administered, with an interval of two to four weeks. Data were collected between March 2012 and December 2016, as part of a larger research project on the applicability of the MoCA in addiction care.

Participants

The main inclusion criterion was an age between 18 and 70. Exclusion criteria were: (a) a current diagnosis of substance or behavioural abuse/dependence according to DSM-5 (American Psychiatric Association 2013) criteria, excluding nicotine; (b) self-reported presence or history of neurological disorders (e.g., stroke, dementia, traumatic brain injury, Korsakoff’s syndrome); (c) self-reported presence or history of an otherwise defined psychotic, psychiatric and/or medical condition that, in the view of this study, would interfere with administration of the MoCA and/or otherwise compromise participation. In total, 218 healthy participants were recruited, one of whom was excluded from further analyses for not meeting the inclusion criterion (i.e., aged 17). Of the remaining 217 participants, seven had completed MoCA version 7.2 and 7.3, and not MoCA version 7.1. As this group was too small to make reliable comparisons between these two alternate versions, these participants were also excluded. Participants were recruited via the personal network of the assessors, by contacting companies and associations, via mouth to mouth, or via social media.

Materials

Montreal Cognitive Assessment (MoCA)

The authorised Dutch translations of three alternate versions of the MoCA were used in this study (www.mocatest.org). MoCA version 7.1 was translated in 2010 by Dautzenberg and De Jonghe (2010) and MoCA alternate versions 7.2 and 7.3 were translated in 2012 by Wester and Kessels. The MoCA consists of 12 items: alternating trail making (0–1 point), figure copy (0–1 point), clock drawing (0–3 points), animal naming (0–3 points), digit span (0–2 points), sustained attention (0–1 point), serial subtraction (0–3 points), sentence repetition (0–2 points), verbal fluency (0–1 point), abstract reasoning (0–2 points), memory (delayed recall, 0–5 points) and orientation (0–6 points).

All items add up to MoCA-TS, with a maximum of 30 points, where a higher score represents better cognitive functioning. In this study, the unadjusted raw MoCA-TS was used in all analyses (i.e., not adding points for individuals with lower educational levels). Seven domain scores (MoCA-DS) were calculated: executive functioning (alternating trail making and verbal fluency: 0–2 points), visuospatial abilities (figure copy and clock drawing: 0–4 points), attention, concentration and working memory (digit span, sustained attention and serial subtraction: 0–6 points), language (animal naming and sentence repetition: 0–5 points), abstract reasoning (0–2 points), memory (0–5 points) and orientation (0–6 points). Finally, the MoCA-MIS was calculated, in which freely recalled words receive three points, words recalled after a category cue receive two points (cued recall) and correct identification after a multiple-choice cue (recognition) receives one point, with a maximum of 15 points (Julayanont et al. 2014).

The first author thoroughly checked all scores and corrected scoring errors when needed. This check revealed ambiguities in the scoring, mainly for the figure copy, for which the scoring instructionswere not fully specified. Therefore, all figures were scored in a consistent manner according to strict criteria by the first author. This procedure also eliminated inter-rater differences that were found to influence results (Cumming et al. 2018).

National Adult Reading Test (NART)

The Dutch version of the National Adult Reading Test (NART; Nelson 1982), an important tool for estimating premorbid levels of intelligence (Bright et al. 2018), was administered. The test consists of 50 words with an uncommon pronunciation. The participant is instructed to read these words aloud in the correct pronunciation. Each answer is awarded 0–2 points, where 0 = false, 1 = questionable, 2 = correct pronunciation, with a maximum of 100 points. Norm scores based on age and sex are added to the total score to determine the estimated premorbid intelligence (NART-IQ; Schmand et al. 1992).

Procedure

MoCA and NART administration and scoring were performed by four assessors with a background in psychology who received extensive training by the first author. With respect to ethical clearance, the study design was approved by the ethics review committee of the Faculty of Social Sciences of Radboud University and the Institutional Review Board of Vincent van Gogh Institute for Psychiatry. After written informed consent was obtained, an appointment was made for the administration of the tasks. Relevant demographic data were recorded via a self-report questionnaire. Level of education was classified on a seven-point scale ranging from 1 = less than primary school to 7 = a university master’s degree or higher (Duits and Kessels 2014), based on the Dutch educational system. This is a classification system that is comparable to the International Standard Classification of Education (ISCED; United Nations Educational Scientific and Cultural Organization 2012). Next, the NART was administered followed by one of the MoCA versions. A second appointment was then made in which another version of the MoCA was administered. Assessments took place in a quiet room at the office or at the participant’s home for logistic reasons, that is, to reduce recruitment bias, make flexible appointments possible, and to reduce travel time for participants.

Data sets from three smaller studies were combined. In the first study, the NART and MoCA version 7.1 were administered in 34 participants. In the second study, MoCA version 7.1 and 7.2 were administered in this fixed order in 74 participants. In the third study, 103 participants completed the NART, and two versions of the MoCA were administered in a counterbalanced order: MoCA version 7.1 and 7.2 (51 participants), and MoCA version 7.1 and 7.3 (52 participants).

Analyses

Participant characteristics and MoCA scores

The participant characteristics for age, age category (for which age was categorised into two groups, i.e., 18–54 and 55–70 years), sex, educational level and estimated premorbid intelligence are presented for the total sample. Next, a description of MoCA scores on all items, and the mean MoCA-TS, -DS and -MIS were provided for each version.

Demographic factors

In the analyses examining the influence of demographic factors on MoCA performance, only participants were included in whom MoCA version 7.1 was administered first, eliminating possible learning effects. The influence of age, educational level and estimated premorbid intelligence on MoCA-TS and -MIS were determined using Pearson’s correlations or Spearman’s rho (if assumptions of normality were not met). The influence of sex and age category on MoCA-TS was determined using independent t-tests and to determine the influence of these variables on MoCA-MIS, Mann–Whitney tests were used.

Systematic differences

The equivalence of the MoCA versions was determined by using all participants in whom MoCA 7.1 and one of the alternate versions were administered in a counterbalanced order, to reduce the chances of the order of administration adversely influencing the results. Equivalence was determined for the MoCA-TS, -DS, -MIS and for each item by examining possible systematic differences using Wilcoxon signed-rank two-related-samples tests or McNemar’s test (for all dichotomous scores).

Test–retest reliability

The test–retest reliability was determined using all participants in whom administration of MoCA 7.1 was followed by one of the alternate versions. By using two-way mixed intra-class correlations (ICC) with absolute agreement, the test–retest reliability for the MoCA-TS, -DS, -MIS and each item was determined. An ICC of less than 0.40 is indicative of poor reliability, between 0.40 and 0.59 is considered fair, between 0.60 and 0.74 good, and an ICC of 0.75 and above indicates excellent reliability (Cicchetti 1994). All data were computed and analysed with IBM SPSS version 25.0 (Armonk, NY: IBM Corp.).

Results

Participant characteristics and MoCA scores

Table 1 provides an overview of participant characteristics for the total sample. Table 2 presents an overview of the mean MoCA-TS, -DS and -MIS, and a frequency distribution of all item scores, for all three versions.

Table 1. Participant characteristics, frequency of administration and administration interval of the Montreal Cognitive Assessment (MoCA), for the total sample.

	Total sample (n = 210)
Mean age in years (SD)	35.3 (16.4)
18–54 years (%)	170 (81.0)
55–70 years (%)	40 (19.0)
Sex (%)
Male	78 (37.1)
Female	132 (62.9)
Level of education (%)
2: primary school	1 (0.5)
3: more than primary school, no other diploma’s	6 (2.9)
4: lower secondary education	16 (7.6)
5: average secondary education	120 (57.1)
6: higher secondary education/a university bachelor’s degree	55 (26.2)
7: a university master’s degree/a post-doc degree	12 (5.7)
NART-IQ (%; n = 121)	98.1 (11.6)
Range	67–135
MoCA administration (%)
7.1	44 (21.0)
7.1–7.2	84 (40.0)
7.2–7.1	27 (12.9)
7.1–7.3	29 (13.8)
7.3–7.1	26 (12.4)
Mean administration interval in days (SD; n = 166)	22.1 (10.1)
Range	5–65

NART-IQ, National Adult Reading Test; estimated premorbid intelligence.

Table 2. Means and standard deviations (SD) for the Montreal Cognitive Assessment (MoCA) total, domain and memory index scores, and frequency distribution of scores per item, for each version in the total sample.

	MoCA 7.1 (n = 210)	MoCA 7.2 (n = 111)	MoCA 7.3 (n = 55)
Mean total score	25.50 (2.27)	25.13 (2.67)	26.16 (1.96)
Scoring range (0–30)	19–30	15–30	21–30
Mean domain scores
Executive functioning	1.50 (0.61)	1.67 (0.49)	1.67 (0.47)
Visuospatial abilities	2.88 (0.89)	2.84 (0.78)	2.85 (0.97)
Attention	5.70 (0.55)	5.55 (0.78)	5.76 (0.47)
Language	4.48 (0.67)	4.17 (0.84)	4.04 (0.58)
Abstract reasoning	1.45 (0.56)	1.61 (0.62)	1.91 (0.29)
Memory	3.63 (1.13)	3.36 (1.31)	4.07 (1.14)
Orientation	5.86 (0.36)	5.93 (0.26)	5.85 (0.36)
	(n = 159)	(n = 65)	(n = 55)
Mean memory index score	13.15 (1.80)	13.57 (1.08)	13.76 (1.48)
Scoring range (0–15)	6–15	11–15	9–15
Item score frequency distribution
Alternating trail making (0/1)	29/181	10/101	3/52
Copy figure (0/1)	135/75	87/24	33/22
Clock drawing (0/1/2/3)	0/16/68/126	0/6/30/75	0/7/16/32
Naming (0/1/2/3)	0/1/24/185	0/0/17/94	0/0/1/54
Digit span (0/1/2)	1/25/184	2/27/82	0/6/49
Sustained attention (0/1)	6/204	4/107	1/54
Serial subtraction (0/1/2/3)	0/5/20/185	0/5/5/101	0/1/4/50
Sentence repetition (0/1/2)	7/69/134	15/45/51	7/38/10
Verbal fluency (0/1)	77/133	27/84	15/40
Abstract reasoning (0/1/2)	7/101/102	8/27/76	0/5/50
Memory (0/1/2/3/4/5)	3/5/22/61/64/55	4/7/14/28/36/22	0/2/5/6/16/26
Orientation (0/1/2/3/4/5/6)	0/0/0/0/1/28/181	0/0/0/0/0/8/103	0/0/0/0/0/8/47

Demographic factors

MoCA-TS was negatively related to age (ρ = −0.21, p = .009) and age category (t(155) = 3.63, p < .001), and positively correlated with level of education (ρ = 0.47, p < .001) and estimated premorbid intelligence (r = 0.51, p ≤ .001). Sex was not related to MoCA-TS (t(155) = 0.59, p = .557).

MoCA-MIS was negatively related to age (ρ = −0.26, p = .003) and age category (U = 255.50, z = −3.96, p < .001, r = −0.35). Younger participants (Mdn = 14.00) scored significantly higher (and near-ceiling) than older participants (Mdn = 11.00). A positive correlation was found between MoCA-MIS and level of education (ρ = 0.32, p < .001), and estimated premorbid intelligence (ρ = 0.34, p = .016). With respect to sex, men (Mdn = 13.00) scored significantly lower than women (Mdn = 14.00; U = 1520.50, z = −2.14, p = .032, r = −0.19).

Systematic differences

Systematic differences were found across alternate versions for the items animal naming, sentence repetition and abstract reasoning. For animal naming, scores on version 7.1 were lower than on version 7.3 (T = 1, z = −2.53, p = .011, r = −0.34). For sentence repetition, scores on version 7.1 were higher than those on both version 7.2 (T = 10, z = −2.52, p = .012, r = −0.34) and version 7.3 (T = 3, z = −5.09, p < .001, r = −0.69). Both animal naming and sentence repetition are part of the language domain, for which scores on version 7.1 were found to be significantly higher than on both version 7.2 (T = 9, z = −2.59, p = .010, r = −0.35) and version 7.3 (T = 6, z = −3.74, p < 0.001, r = −0.50). As for the item abstract reasoning, which is also a MoCA-DS, scores on version 7.1 were lower than those on both version 7.2 (T = 6, z = −2.69, p = .007, r = −0.36) and version 7.3 (T = 4, z = −2.56, p = .011, r = −0.34). MoCA-TS, -MIS and the other MoCA-DS and items, did not differ between versions.

Test–retest reliability

Tables 3 and 4 show the ICCs with means and standard deviations for the MoCA-TS, -DS and -MIS, and for all items, between versions 7.1–7.2 and 7.1–7.3, respectively. The test–retest reliability for both MoCA-TS and -MIS was higher between version 7.1–7.3 (MoCA-TS: ICC = 0.82; MoCA-MIS: ICC = 0.48), than between version 7.1–7.2 (MoCA-TS: ICC = 0.64; MoCA-MIS: ICC = 0.32). For MoCA-DS of version 7.1–7.2, the ICCs ranged from poor (0.18 for language) to good (0.60 for visuospatial abilities). For version 7.1–7.3, the ICCs ranged from poor (0.38 for attention, concentration and working memory) to fair (0.57 for memory), excluding two negative ICCs (abstract reasoning, −0.29 and orientation, −0.32).

Table 3. Means and standard deviations (SD) for the Montreal Cognitive Assessment (MoCA) total, domain, memory index and item scores for both version 7.1 and alternate version 7.2, and test–retest reliability (intra-class-correlation; ICC) between versions.

	MoCA 7.1	MoCA 7.2	ICC	95% CI	p Value
	(n = 84)
Mean total score	25.10 (2.15)	25.58 (2.28)	0.64	0.44–0.76	<.001
Mean domain scores
Executive functioning	1.44 (0.63)	1.70 (0.49)	0.44	0.14–0.63	.002
Visuospatial abilities	2.85 (0.87)	2.99 (0.69)	0.60	0.38–0.74	<.001
Attention	5.52 (0.69)	5.51 (0.83)	0.49	0.21–0.67	.001
Language	4.38 (0.68)	4.19 (0.86)	0.18	−0.26 to 0.46	.183
Abstract reasoning	1.49 (0.55)	1.68 (0.58)	0.31	−0.04 to 0.55	.037
Memory	3.49 (1.04)	3.56 (1.19)	0.30	−0.08 to 0.55	.054
Orientation	5.93 (0.26)	5.95 (0.21)	0.54	0.29–0.70	<.001
	(n = 60)
Mean memory index score	13.20 (1.48)	13.67 (1.00)	0.32	−0.10 to 0.59	.060
Mean item scores^a
Alternating trail making	0.82 (0.39)	0.92 (0.28)	0.56	0.32–0.71	<.001
Copy figure	0.29 (0.45)	0.24 (0.43)	0.34	−0.02 to 0.57	.031
Clock drawing	2.56 (0.63)	2.75 (0.46)	0.45	0.17–0.64	.002
Animal naming	2.88 (0.36)	2.83 (0.38)	−0.35	−1.09 to 0.13	.912
Digit span	1.80 (0.43)	1.71 (0.48)	0.22	−0.20 to 0.49	.130
Sustained attention	0.98 (0.15)	0.95 (0.21)	0.48	0.20–0.66	.002
Serial subtraction	2.75 (0.54)	2.85 (0.50)	0.41	0.09–0.61	.009
Sentence repetition	1.50 (0.55)	1.36 (0.71)	0.34	−0.00 to 0.57	.026
Verbal fluency	0.62 (0.49)	0.79 (0.41)	0.45	0.16–0.64	.002

^a Item scores for abstract reasoning, memory and orientation are not shown, as they are the same as the domain scores.

Table 4. Means and standard deviations (SD) for the Montreal Cognitive Assessment (MoCA) total, domain, memory index and item scores for both version 7.1 and alternate version 7.3, and test–retest reliability (intra-class-correlation; ICC) between versions.

	MoCA 7.1	MoCA 7.3	ICC	95% CI	p Value
	(n = 29)
Mean total score	26.97 (2.01)	26.45 (2.25)	0.82	0.62-0.92	<.001
Mean domain scores
Executive functioning	1.79 (0.41)	1.90 (0.31)	0.53	0.03–0.78	.022
Visuospatial abilities	2.93 (1.00)	2.83 (1.00)	0.52	−0.04 to 0.78	.031
Attention	5.86 (0.35)	5.76 (0.44)	0.38	−0.30 to 0.71	.104
Language	4.59 (0.63)	4.14 (0.58)	0.44	−0.09 to 0.73	.026
Abstract reasoning	1.76 (0.44)	1.93 (0.26)	−0.29	−1.56 to 0.38	.761
Memory	4.17 (0.85)	4.00 (1.13)	0.57	0.08–0.80	.016
Orientation	5.86 (0.35)	5.90 (0.31)	−0.32	−1.96 to 0.39	.761
Mean memory index score	13.62 (1.50)	13.72 (1.49)	0.48	−0.14 to 0.76	.051
Mean item scores^a
Alternating trail making	0.97 (0.19)	0.97 (0.19)	1.00	–	–
Copy figure	0.38 (0.49)	0.41 (0.50)	0.13	−0.93 to 0.60	.367
Clock drawing	2.55 (0.69)	2.41 (0.73)	0.62	0.20–0.82	.006
Animal naming	2.86 (0.35)	2.97 (0.19)	−0.13	−1.33 to 0.46	.629
Digit span	1.97 (0.19)	1.86 (0.35)	0.54	0.07–0.78	.016
Sustained attention	1.00 (0.00)	0.97 (0.19)	0.00	−1.13 to 0.53	.500
Serial subtraction	2.90 (0.31)	2.93 (0.26)	0.52	−0.03 to 0.78	.030
Sentence repetition	1.72 (0.46)	1.17 (0.54)	0.35	−0.20 to 0.68	.032
Verbal fluency	0.83 (0.38)	0.93 (0.26)	0.35	−0.34 to 0.69	.123

^a Item scores for abstract reasoning, memory and orientation are not shown, as they are the same as the domain scores.

Discussion

The current study assessed the psychometric properties of the original version and two alternate versions of the MoCA in a group of healthy participants with an age range of 18–70 years. We found that older participants performed worse on MoCA-TS and -MIS than younger participants, while participants with higher educational levels and a higher estimated premorbid intelligence also obtained higher MoCA scores. Significant but small sex differences were found for MoCA-MIS, with women outperforming men. Systematic differences between versions were identified for the items animal naming, sentence repetition and MoCA-DS language and abstract reasoning. The test–retest reliability for the MoCA-TS was good (7.1–7.2) to excellent (7.1–7.3). For the MoCA-MIS, the test–retest reliability was poor (7.1–7.2) to fair (7.1–7.3).

The current results show that performance on the MoCA is moderated by educational attainment, intelligence and age, and that these factors should be taken into account when interpreting results on this screener (Shulman 2000). The effect of intelligence on MoCA performance was expected as it is known that intelligence typically correlates highly with level of education (Lezak et al. 2012), albeit that in older adults educational attainment and intelligence may not always correspond well because of limited access or possibilities to advanced schooling. Comparing our results of participants from a large age range with other studies shows they are in agreement with previous findings in adults aged 50 and older (Apolinario et al. 2018; Malek-Ahmadi et al. 2015; Oren et al. 2014; Sugarman and Axelrod 2014; Yancar Demir and Özcan 2015; Zheng et al. 2012). Next, like several other studies, this study did not find an effect of sex on MoCA-TS (Apolinario et al. 2018; Kopecek, Stepankova, et al. 2017; Santangelo et al. 2015; Zheng et al. 2012). However, an effect of sex on the MoCA-MIS was found, in favour of women. Ojeda et al. (2016) also found an effect of sex on the delayed recall of the MoCA in Spanish participants aged ≥18. The difference between men and women in memory functioning has also been objectified in childhood ages (Gur et al. 2012) and in particular working memory tasks in adults (Saylik et al. 2018).

No systematic differences between MoCA-TS of the alternate versions compared to version 7.1 were found, indicating that all versions are essentially equivalent. This was also found for the recently developed MoCA-MIS, replicating the findings of Chertkow et al. (2011) and Nasreddine and Patel (2016) in older individuals. Taking a closer look at the items, however, it was found that animal naming, sentence repetition and abstract reasoning did systematically differ between versions, with some items of the alternate versions being more difficult and others less compared to MoCA version 7.1. Lebedeva et al. (2016) reported similar findings for both animal naming and abstract reasoning. They also found a performance difference for the item figure copy across the different versions. The fact that we did not identify any differences for the latter could be explained by the thorough checking of scores and the conservative scoring method that was used, eliminating ambiguities in scoring and possible inter-rater differences influencing results (Cumming et al. 2018). Similarly strict scoring criteria are embedded in the instructions of the recently published MoCA version 8 (i.e., ‘all lines meet with little or no space’ and ‘the figure’s orientation in space must be preserved’). The systematic differences found between the sentences may be the outcome of the adaptation and translation process into Dutch resulting in somewhat longer sentences for alternate versions 7.2 and 7.3 compared to the English-language alternate versions. Nasreddine and Patel (2016) completely changed the sentences in the French versions instead of translating them from the original English-language versions. The systematic differences that were found on item level cancelled each other out when looking at the MoCA-TS, resulting in three equivalent versions. Focussing on the MoCA-DS, it should be taken into account that individuals may perform better on language (which includes the items animal naming and sentence repetition) and worse on abstract reasoning on version 7.1 compared to version 7.2 and/or 7.3 due to the abovementioned systematic differences, rather than actual changes in cognitive functioning over time.

A good to excellent test-retest reliability was found for MoCA-TS between versions 7.1–7.2 and 7.1–7.3, respectively, in line with findings of other studies (Costa et al. 2012; Feeney et al. 2016; Kopecek, Bezdicek, et al. 2017; Nasreddine and Patel 2016; Wu et al. 2017). The test–retest reliabilities can be used to compute Reliable Change Indices (RCI; Chelune et al. 1993) making the MoCA scores useful for monitoring change over time. The MoCA-MIS had poor to fair test–retest reliability, possibly due to a strong negative skewness of scores with ceiling performances on both versions in 31 participants (as opposed to 0 ceiling performances for MoCA-TS on both versions). However, in memory impaired individuals, the MoCA-MIS has been found to be a useful index of monitoring change over time. For instance, Julayanont et al. (2014) showed that the MoCA-MIS was a good predictor for conversion from mild cognitive impairment to Alzheimer’s disease. More research into the applicability of the MoCA-MIS in other clinical samples is needed, as compared to this sample of healthy individuals. For the other MoCA-DS and item scores, reliabilities ranged from poor to good, strengthening the fact that the MoCA is reliable and applicable as a global screen for cognitive functioning rather than as a tool to assess individual cognitive domains.

Some strengths of the study are firstly, that it is the first one to assess the psychometric properties of all three versions of the Dutch translations of the MoCA. Secondly, we included the MoCA-MIS for which not much is known yet about the applicability of this score in clinical practice. Next, it was possible to include a large sample of healthy participants with a wide age range (including adults aged 18–50), and educational background, being representative for the general population. And finally, although we used the Dutch version of the MoCA, our results are clearly also relevant for the use and interpretation of other-language versions of the MoCA. Limitations to the current study are, firstly, the non-orthogonal design, resulting in a relatively small group for comparing version 7.1 with version 7.3, and making it unable to compare version 7.2 with version 7.3, and secondly, that only self-reported exclusion criteria were used rather than an objective measure of cognitive impairment.

With respect to the clinical implications of our results, it is clear that an adjustment for education (either for level or years) is essential, as originally proposed by Nasreddine et al. (2005), and recently fine-grained by Bruijnen et al. (2019). However, further research is needed in order to examine if a better adjustment method can be generated. For instance, more elaborate stratified or regression-based normative data could be constructed, like those of Borland et al. (2017) who proposed normative Swedish data that adjust the total score for age, education and sex. Furthermore, it should be noted that a ceiling performance was found for a substantial number of healthy participants for the MoCA-MIS, indicating that this index may be insensitive to small cognitive decrements in some clinical populations.

Some of the shortcomings of MoCA version 7 might be overcome in the recently published version 8. When comparing the English-language versions 7 to versions 8, no changes were seen in version 1, while in versions 2 and 3 only about half of the items remained the same (i.e., the figures are all ‘cube’-like, using only straight lines, and the digits are randomised rather than changed). For the Dutch translation, only version 8.1 has been made available yet, for which one word of the memory subtest was replaced. As for the scoring and administration instructions, changes have been made by clarifying some of the ambiguities in instructions that lead to personal interpretation in the scoring of version 7 (i.e., the possibility to repeat instructions, clarifications for scoring the executive/visuospatial items, simplified instructions for the verbal fluency, the adding of multiple-choice cues), but also the MoCA-MIS is included as a stand-alone score. If these changes/additions are, in fact, overcoming the shortcomings should be examined in more detail in future research, when MoCA version 8 becomes more widely available.

Based on both MoCA-TS and -MIS the Dutch translations of the MoCA are comparable to the English-language versions in their equivalence across versions and their test–retest reliability. Comparisons of the MoCA-DS should, however, be interpreted with caution. Although performance is affected by age, education and intelligence, adequate psychometric properties were found. The test-retest reliability can be used to determine change over time by calculating a reliable change index, adding to the clinical usability of the MoCA. After some training and following strict instructions the screener is easy to score, and was also reported as being acceptable in terms of clarity and difficulty for those undergoing it.

Acknowledgements

We would like to thank Josette Westhoff, Tom Jacobs and Sharon Walhout for collecting the data.

Disclosure statement

The authors report no conflicts of interest.

Additional information

Funding

This research was funded by the Nijmegen Institute for Scientist-Practitioners in Addiction (NISPA), Nijmegen, The Netherlands and Vincent van Gogh for Mental Health, Venray, The Netherlands.