Comparative Study of Two Short-Form Versions of the Montreal Cognitive Assessment for Screening of Post-Stroke Cognitive Impairment in a Chinese Population

Abstract

Purpose

Cognitive impairment (CI) is one of the most significant post-stroke complications. The Montreal Cognitive Assessment (MoCA) is widely applied to the early screening of post-stroke CI (PSCI), and has good sensitivity and specificity, but needs a long time to administer. Clinicians and researchers need shorter, more effective cognitive testing tools. The purpose of this study was to detect the sensitivity and specificity of two different short-form versions of the MoCA (SF-MoCA) for screening of PSCI in a Chinese population.

Methods

A total of 2,989 stroke participants were included from 14 hospitals in northern and southern China between June 2011 and September 2013. The sensitivity and specificity of the two SF-MoCA versions were compared.

Results

Using an MoCA score <26 as the critical value, the National Institute of Neurological Disease and Stroke–Canadian Stroke Network SF-MoCA showed sensitivity of 91% and specificity of 63% (PPV 71%, BPV 87%) with scores ≤10 points. The sensitivity and specificity of the Bocti SF-MoCA were 92% and 69% (PPV 75%, BPV 89%) with scores ≤7, respectively. The area under the curve was 0.885 (95% CI 0.873–0.897) and 0.912 (95% CI 0.902–0.922), respectively.

Conclusion

The Bocti SF-MoCA can be used as a briefer and more effective screening tool for PSCI in Chinese.

Keywords:

• cognitive dysfunction
• Montreal Cognitive Assessment
• MoCA
• stroke
• sensitivity
• specificity

Plain-Language Summary

The Montreal Cognitive Assessment (MoCA) is widely applied to the early screening of post-stroke cognitive impairment (PSCI), but it is time-consuming and not routinely carried out. Clinicians and researchers need shorter, more effective cognitive testing tools. National Institute of Neurological Disease and Stroke–Canadian Stroke Network once proposed a 5-minute protocol, which has been verified as brief, valid, reliable, and feasible for telephone administration in detecting PSCI. The Bocti SF-MoCA has also demonstrated reliability in detecting PSCI. However, there have been few published studies reporting the sensitivity and specificity of the two SF-MoCAs in the Chinese post-stroke population. In this study, we included 2,989 Chinese stroke patients and compared the sensitivity and specificity of the two SF-MoCAs. Our results showed that both SF-MoCAs had good reliability. Interestingly, Bocti SF-MoCA was better on both sensitivity and specificity, which is worth further research, and we conclude that the Bocti SF-MoCA could be used as a briefer and more effective screening tool for PSCI in Chinese.

Introduction

Cognitive impairment (CI), one of the most significant post-stroke complications, occurs at any time after stroke, even after incident stroke,¹ up to 10 years of follow-up.² A third of stroke survivors may be troubled by post-stroke CI (PSCI) or post-stroke dementia.³ Evidence has indicated that first-ever mild stroke might affect cognition and daily functioning, consequently affecting participation, quality of life, and return to work,⁴ and incidence of dementia being 50 times that in the year after a major stroke compared with the general population.⁵ Although there is little evidence for specific therapeutic strategies for preventing cognitive decline after stroke, researches have shown that starting a rehabilitation course for CI as early as possible has a pivotal role in the outcome of patients.^6–8 Therefore, early screening and recognition of PSCI is very important for patients’ rehabilitation treatment.

To date, the Mini–Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA) have been the most reasonable instruments for identifying PSCI.⁹ Researches has shown that the MoCA has better sensitivity in detecting PSCI,^10–15 especially in the Chinese population.^16,17 Our preliminary study¹⁸ proved that the Beijing version of the MoCA executive tasks were more sensitive in detecting post-stroke executive dysfunction compared with MMSE executive tasks. In all, the MoCA is the most widely used and reliable scale in identifying PSCI.

An important concern is that completion of the MoCA can take much longer in practice¹⁹ than 10 minutes. A shorter form of the MoCA (SF-MoCA) is attractive for both patients and clinicians, especially in China, considering stroke burden has increased over the last 30 years in China and the country bears the biggest stroke burden in the world (1,596 per 100,000 people).²⁰ Therefore, many studies have been devoted to SF-MoCAs, since they could improve clinical feasibility, shorten diagnosis time, and reduce assessment burden on patients. A recent study²¹ evaluated the comparative performance of seven SF-MoCAs in identifying mild CI or dementia, and the results showed that all seven had acceptable performance in participants aged ≥50 years. Another cross-sectional test study proposed a brief four-item MoCA that could be a good alternative in nonspecialty clinics.²² Moreover, a recent systematic review²³ containing 13 SF-MoCAs suggested that the choice of SF-MoCAs should be varied by clinical population.

All in all, it is very important to conduct comparative studies of SF-MoCAs in different populations to choose the optimal SF-MoCA. As early as in 2006, the National Institute of Neurological Disease and Stroke–Canadian Stroke Network VCI Harmonization Working Group proposed a 5-minute protocol (NINDS-CSN SF-MoCA),²⁴ which was subsequently verified as brief, valid, reliable, and feasible for telephone administration in detecting PSCI.²⁵ The Bocti SF-MoCA has also demonstrated reliability in detecting PSCI.^26,27 However, there have been few published studies reporting on the sensitivity and specificity of the two SF-MoCAs in the Chinese post-stroke population. Our study focused on the sensitivity, specificity, and comparison of the two SF-MoCAs by a cross-sectional large-sample screening in a Chinese population. The purpose of this study was to provide shorter and more effective time-saving screening tools in detecting PSCI for clinicians.

Methods

Study Design and Participants

Data were obtained from the “Study on Traditional Chinese Medicine (TCM) prevention, treatment and administration in community of post-stroke cognitive impairment” project, which is supported by the TCM Special Research Projects Program, China Ministry of Science and Technology (201007002). As our pervious published study,¹⁸ patients (2 weeks to 6 months after stroke) were recruited from 14 hospitals in northern and southern China (ten top-grade hospitals and four community hospitals) between June 2011 and September 2013, and their cognitive status was scored with the Beijing version of the MoCA.

PSCI emphasizes the importance of stroke as a vascular factor leading to CI. The diagnosis of PSCI was conducted by neurologists using the NINDS-CSN VCI standard²⁴ on the basis of neuropsychological examination together with other available clinical information, such as medical history, prestroke cognitive functioning, and cognition-related functional decline. A MoCA score <26 was taken to indicate CI in stroke patients according to recommendations from published data.^14,28 Also, another stroke cross-sectional study in China has cited this standard.²⁹ The two SF-MoCA scores were derived retrospectively from the full MoCA.

Inclusion criteria were complaining of CI or CI reported by caregiver(s), conscious without communication problems hindering the performance of cognitive tests (eg, aphasia, apraxia, visual deprivation, hearing loss), and cooperative during neuropsychological examination. Exclusion criteria were Hamilton Depression Scale score ≥17 point, acute mental changes (eg, delirium), CI before stroke recorded in medical documents or reported by patients and families, history of head trauma, brain tumor, epilepsy, schizophrenia, or other psychiatric disorders, drug abuse, substance abuse, or alcohol addiction, and moderate–severe dementia. All participants signed informed consent. The study was approved by the Ethics Committee on Clinical Research of Dongfang Hospital (2011123004).

Procedure

General data on all subjects — sex, birth date, education, history of smoking or alcohol intake, as well as history of hypertension, diabetes mellitus, coronary heart disease, previous stroke/transient ischemic attack, and peripheral vascular diseases — were obtained. All staff designated for assessment underwent strict professional training courses. Participants were instructed to complete the Beijing version of the MoCA and received clinical and functional assessment, which took approximately 30 minutes. Each participant was instructed to complete the MoCA first, and after at least an hour they were instructed to complete the MMSE.²⁴

The Beijing version of the MoCA ^30,31 (http://www.mocatest.org) contains seven cognitive domains (visuospatial/executive function, naming, attention, abstraction, language, delayed recall, and orientation) translated from the original English version literally, with the exception of the following modifications. For the visuospatial/executive function domain, alphabetic letters are replaced by Chinese characters (甲/乙/丙/丁/戊), which denote the same as “A/B/C/D/E” in English. For the attention domain, numbers are used instead of alphabetic letters in the auditory vigilance task. For the language domain, in the verbal fluency task, the phonemic fluency task that requires participants to generate words beginning with the letter F is replaced by the semantic fluency task, requiring participants to produce as many animals as possible in 60 seconds. When education of the participants was <12 years (high middle school), their MoCA score would have 1 point added. Activities of daily living (ADL) including basic ADL and instrumental ADL, National Institute of Health Stroke Scale (NIHSS), and Hamilton Depression Scale were also assessed.

Statistical Analysis

All statistical analysis was performed using SPSS 20.0 software (IBM, Armonk, NY, USA). Continuous and categorical variables are presented as means ± SD and frequency (%), respectively. Nonparametric and χ² tests were used to compare two groups for continuous and categorical variables, respectively.The diagnostic accuracy of the two SF-MoCAs for prediction of clinical diagnosis of PSCI was assessed through receiver- operating characteristic (ROC)–curve analysis, with larger area under the curve (AUC) indicating better diagnostic accuracy. ROC curves were compared according to the AUC-comparison method of Hanley and McNeil.³² Optimal cutoff points for each screening instrument that yielded the highest Youden index values were selected, with higher values indicating maximum sensitivity and specificity. To analyze the predictive value of the tests, for each cutoff point we calculated the sensitivity (probability of subjects with CI having a positive result), specificity (probability of subjects without CI having a negative test), positive predictive value (PPV; probability of disease in subjects with a positive result), negative predictive value (NPV; probability of a “lack of disease” classification in subjects with a negative result), and classification accuracy (probability of correct classification of subjects, with or without CI).³³ All P-values were considered statistically significant if P<0.05.

Results

A total of 3,000 participants were recruited for this study, of which eleven were excluded due to incomplete data (six did not complete the MoCA, five lacking imaging data), and so 2,989 were eventually included for statistical analysis. There were 1,651 males of mean age 62.50±10.56 years (range 33–92 years) and 1,338 females of mean age 63.56±9.92 years (range 33–90 years). Median stroke duration was 1–2months. Mean NIHSS score was 1.16±2.54, ADL 25.67±11.23, Hamilton Depression Scale 3.68±3.88, and MMSE 27.24±2.83. The range of MMSE scores was 17–30 (Table 1). These patients had different lesion regions, lesion sides, lesion size, and types of stroke, as shown in Table 2. The severity of these patients was examined by the NIHSS and ADL scale, which were 1.16±2.54 and 25.67±11.23 respectively. Vascular risk factors are shown in Table 3.

Table 1 Demographic Characteristics and Test Scores

	Overall (n=2,989)
Age (years)	63.03±10.28 (33–92)
Sex
Male (%)	1,651 (55.2)
Female (%)	1,338 (44.8)
Education (%)
<6 years	716 (24.0)
6–12 years	826 (27.6)
≥12 years	1,447 (48.4)
BMI (kg/m^2)	24.28±3.01
NIHSS	1.16±2.54
ADL scale	25.67±11.23
HAMD	3.68±3.88
MMSE	27.24±2.83 (17–30)

Abbreviations: BMI, Body Mass Index; NIHSS, National Institutes of Health Stroke Scale; ADL, Activities of Daily Living; HAMD, Hamilton Depression (17 items); MMSE, Mini–Mental State Examination.

Table 2 Lesion Regions, Lesion Sides, and Types of Stroke

	Overall (n=2,989)
Lesions involving structure
Cortex	333 (11.1%)
Subcortical white matter	800 (26.8%)
Basal ganglia	1,796 (60.1%)
Thalamus	204 (6.8%)
Brain stem	215 (7.2%)
Cerebellum	99 (3.3%)
Lesions involving lobe
Frontal	570 (19.1%)
Parietal	507 (17.0%)
Temporal	350 (11.7%)
Occipital	153 (5.1%)
Insular	42 (1.4%)
Lesion side
Left	679 (22.7%)
Right	667 (22.3%)
Bilateral	1,456 (48.7%)
Type of stroke
Ischemic	2,867 (95.9%)
Hemorrhagic	122 (4.1%)

Table 3 Vascular Risk Factors

	Yes	No
Smoking	1,043 (34.9%)	1,946 (65.1%)
Alcohol	961 (32.2%)	2,028 (67.8%)
Hypertension	1,989 (66.5%)	1,000 (33.5%)
Diabetes	695 (23.3%)	2,294 (76.7%)
Dyslipidemia	691 (23.1%)	2,298 (76.9%)
Heart diseases	535 (17.9%)	2,454 (82.1%)
Previous stroke/TIA	545 (18.2%)	2,444 (71.8%)
Peripheral vascular diseases	37 (1.2%)	2,952 (98.8%)

Abbreviation: TIA, transient ischemic attack.

Table 4 shows descriptive statistics for each test of cognition, including the MMSE, MoCA total, and every MoCA subset. After division of patients into two groups by cutoff score for the MoCA of 26, the prevalence of PSCI was 50.4% (1,506 of 2,989), and all cognitive domains in the MoCA score <26 group (visuospatial/executive, naming, attention, language, abstraction, delayed recall, and orientation) scores were lower than MoCA score ≥26 group. As shown in Table 5, when the NINDS-CSN SF-MoCA score was 10/12 or less, sensitivity was 91% and specificity 63% (PPV 71%, NPV 87%), and when the Bocti SF-MoCA score was 7/10 or less, sensitivity was 92% and specificity was 69% (PPV 75%, NPV 89%). The AUC was 0.885 (95% CI 0.873–0.897) and 0.912 (95% CI 0.902–0.922), respectively. Table 6 is a cross-classification of patients by each SF-MoCA.

Table 4 Descriptive Statistics for Each Test Factor of the MoCA

Domain	All Patients, n=2,989	MoCA <26, n=1,506	MoCA ≥26, n=1,483	Z	P
Visuospatial/executive (VE)
Trail-making	0.47±0.50	0.28±0.45	0.67±0.47	21.096	0*
Cube copy	0.63±0.48	0.46±0.50	0.79±0.41	18.792	0*
Clock-drawing	2.45±0.84	2.10±0.97	2.80±0.46	23.152	0*
Subtotal (VE)	3.55±1.36	2.84±1.40	4.26±0.85	28.680	0*
Naming	2.76±0.52	2.59±0.65	2.93±0.25	18.269	0*
Attention
Digit span	1.84±0.41	1.71±0.52	1.98±0.14	19.325	0*
Letter A	0.83±0.38	0.74±0.44	0.92±0.28	12.797	0*
Serial 7	2.60±0.72	2.32±0.85	2.89±0.38	23.439	0*
Subtotal (attention)	5.27±1.11	4.76±1.30	5.79±0.48	26.494	0*
Language
Sentence repeat	1.46±0.71	1.15±0.76	1.76±0.48	23.818	0*
Fluency	0.85±0.36	0.76±0.43	0.94±0.24	13.746	0*
Subtotal (language)	2.30±0.83	1.91±0.88	2.70±0.52	26.589	0*
Abstraction	1.44±0.68	1.16±0.73	1.73±0.47	22.834	0*
Delayed recall	3.04±1.44	2.19±1.38	3.91±0.86	33.417	0*
Orientation	5.62±0.88	5.34±1.13	5.91±0.32	19.619	0*
MoCA total	24.26±4.27	21.10±3.83	27.46±1.18	47.569	0*

Note: Data are means ± SD, *P<0.05.

Abbreviation: MoCA, Montreal Cognitive Assessment.

Table 5 Sensitivity, Specificity, and Positive and Negative Predictive Values of the Two SF-MoCAs

	Sensitivity	Specificity	PPV	NPV
NINDS-CSN SF-MoCA ≤10/12	91%	63%	71%	87%
NINDS-CSN SF-MoCA ≤9/12	71%	93%	91%	76%
NINDS-CSN SF-MoCA ≤8/12	46%	99%	97%	64%
Bocti SF-MoCA ≤8/10	98%	34%	60%	93%
Bocti SF-MoCA ≤7/10	92%	69%	75%	89%
Bocti SF-MoCA ≤6/10	77%	90%	89%	80%

Notes: NINDS-CSN SF-MoCA score (12 points) includes orientation (6 points), five-word recall (5 points), and word fluency (1 point); Bocti SF-MoCA score (10 points) includes verbal fluency (1 point), cube copy (1 point), trail-making (1 point), abstraction (2 points), and five-word recall (5 points).

Abbreviations: NINDS-CSN, National Institute of Neurological Disorders and Stroke–Canadian Stroke Network; MoCA, Montreal Cognitive Assessment; SF-MoCA, short-form MoCA.

Table 6 Cross Classification of Patients Screening Positive or Negative for Cognitive Impairment on the Two SF-MoCAs

	MoCA Score
	<26	≥26
NINDS-CSN SF-MoCA
≤10	1,363 (45.5%)	556 (18.6%)
>10	143 (4.8%)	927 (31.0%)
Bocti SF-MoCA
≤7	1,381 (46.2%)	467 (15.6%)
>7	125 (4.2%)	1,016 (34.0%)

Abbreviations: NINDS-CSN, National Institute of Neurological Disorders and Stroke–Canadian Stroke Network; MoCA, Montreal Cognitive Assessment; SF-MoCA, short-form MoCA.

Discussion

To our knowledge, this is the first study to explore the utility and accuracy of these two SF-MoCAs in a Chinese post-stroke population. The main findings of our study were that sensitiviy and specificity of the NINDS-CSN SF-MoCA were 91% and 63% and AUC 0.885 (95% CI 0.873–0.897), sensitivity and specificity of the Bocti SF-MoCA were 92% and 69% and AUC 0.912 (95% CI 0.902–0.922), both the SF-MoCAs had high sensitivity and low specificity, and the Bocti SF-MoCA was superior in detecting PSCI in this Chinese population.

Sensitivity and Specificity of NINDS-CSN SF-MoCA

The NINDS-CSN SF-MoCA, known as the NINDS-CSN 5-minute protocol, consists of only verbally conducted tests: orientation, memory, and verbal fluency tasks. As such, it has been recommended for stroke patients for whom paper-and-pencil tests were inconvenient or who needed telephone follow-up.^24,25 In our study, a cutoff of ten scores was identified as optimal (sensitivity 91%, specificity 63%, AUC 0.885) based on ROC-curve analysis, which was different from a previous study³⁴ that applied the NINDS-CSN 5-minute neuropsychology protocol in acute stroke for detecting post-stroke dementia, with an acceptable value was seven scores. Possible reasons for the difference may be that our subjects were at different stages of stroke (2 weeks to 6 months vs within 2 weeks) and the coverage of disease (PSCI vs post-stroke dementia) was different. Also, this difference reminds us that the sensitivity and specificity of the NINDS-CSN SF-MoCA may vary according to the clinical population and research purposes, which was similar to the results of existing researches.^27,33,35,36

Sensitivity and Specificity of Bocti SF-MoCA

The Bocti SF-MoCA includes verbal fluency, cube copy, trail-making, abstraction, and five-word recall. A previous study²⁶ took Canadian stroke patients after 3 months as subjects and sensitivity was 91%, similar to our result. Considering the median stroke duration in our study was 1–2 months, this is not surprising. Also, there is no “best” value. For high sensitivity, we choose eight scores as the cutoff so as not to miss patients who needed further in-depth assessment. All in all, the Bocti SF-MoCA is also suitable for detecting PSCI in Chinese.

Comparison of NINDS-CSN SF-MoCA and Bocti SF-MoCA

SF-MoCAs provide a viable alternative for detection of PSCI when it is not feasible to complete the full MoCA in clinical practice. SF-MoCAs are especially useful in nonspecialty clinics (such as primary-care clinics)²¹ for early detection of CI, since they are derived from the full MoCA, time-saving, and less resource-consuming. Screen positives mean further in-depth professional neuropsychological evaluations are needed, and also provide the possibility of timely and early interventions, which is in line with the International Association of Gerontology and Geriatrics call³⁷ on the need for validated short screening tests and timely support for early CI. As shown in Table 5, both the Bocti SF-MoCA and NINDS-CSN SF-MoCA had lower specificity and higher sensitivity, with corresponding low positive predictive value and high negative predictive value. This was consistent with a newly published systematic review.²³ These results were not surprising, since the MoCA was designed to detect mild CI and our participants included people with mild dementia. High sensitivity suggests potential utility of the two SF-MoCAs as initial cognitive screening tools, but low specificity suggests many false positives and further assessment is necessary.

Compared with the NINDS-CSN MoCA, the Bocti SF-MoCA includes executive function (ie, cube copy, trail-making) besides memory and word fluency. Therefore, the Bocti SF-MoCA picked up substantially more cognitive deficits in patients with stroke than the NINDS-CSN MoCA. Moreover, in the years following an incident stroke, executive function declines significantly faster than it did before the stroke,¹ and 40.6% of post-ischemic stroke patients had executive dysfunction.³⁸ That is to say, executive dysfunction has become one of the most significant PSCI complications,^24,39,40 and the detection of executive function is of great significance after stroke. In the present study, we found that the sensitivity and specificity of the Bocti SF-MoCA were higher than the NINDS-CSN SF-MoCA. From Table 6, we can also see that the Bocti SF-MoCA had greater true-positive and -negative rates. Te AUC of the Bocti SF-MoCA was also more accurate. These findings showed that including more cognitive domains, especially executive, may be one advantage of the Bocti SF-MoCA.

Limitations

In this study, several limitations existed. Firstly, we used just MoCA scores <26 as a cutoff for diagnosis, and did not administer an independent and more extensive neuropsychological battery as a gold standard. Therefore, we cannot be certain that lower scores on the MoCA were detecting meaningful CI. This might bring a risk of circularity. More accurate assessments of PSCI should be added in future studies (such as the 60-minute protocol).²⁴ Secondly, NIHSS scores of subjects in this study were 1.16±2.54 on average, and may not be applicable to patients with severe stroke, who may be more concerned about survival rate.⁴¹ Thirdly, patients with moderate and severe dementia were excluded from the present study, because of the ceiling effect. Finally, the choice of cutoff points should depend on whether the test is used as a screen (high sensitivity required) or as a diagnostic tool (high specificity required).⁴² Also, we conducted this cross-sectional study on only two SF-MoCAs, both of which were derived retrospectively from the full MoCA.Further investigations with prospective design are needed to clarify more reliable cutoff points for SF-MoCAs according to different clinical environments.

Conclusion

The Bocti SF-MoCA — trail-making (1 point), cube copy (1 point), verbal fluency (1 point), abstraction (2 points), and five-word recall (5 points) — can be used as a briefer and more effective screening tool for Chinese PSCI patients.

Acknowledgments

This study was funded by National Adiministration of TCM (NATCM) TCM Inheritance and Innovation "Hundred-Thousuand-Ten Thousand" Talents Project (Qihuang Scholar)-National TCM Leading Personnel Support Program (NATCM Personnel and Education Department[2018]12), the TCM Special Research Projects Program, State Administration of TCM (201007002, 201407001-8), and National Key R&D Program of China (2018YFC1704300). We are also grateful to the participants in this study for their generous participation and their clinicians for recruiting them in this study.

Disclosure

The authors report no conflicts of interest in this work.