http://orcid.org/0000-0002-2662-6023
Abstract
Background: Under conditions of high demand for primary care services in a setting of low financial resources, there is need for brief, easily administered cognitive screening tools for use in the primary care setting, especially in rural areas. However, interpretation of these cognitive tests’ results requires knowledge on their susceptibility to cultural, educational and demographic patient characteristics.
Objectives: To assess the clinical validity of the ‘Test Your Memory’ (TYM) and ‘General Practitioner assessment of Cognition’ (GPCog) which was specifically designed for primary care practice, in a rural primary care setting in Greece, utilizing the ‘Mini Mental State Examination’ (MMSE) as a reference standard.
Methods: The MMSE, TYM, and GPCog were administered to a random sample of 319 community dwelling Greek adults aged 60 to 89 years in 11 rural Primary Healthcare Centres of the Prefecture of Heraklion on the island of Crete, Greece. Analyses examined (a) The association of each instrument with demographic factors and MMSE and (b) optimal cut-off scores, sensitivity and specificity against MMSE-based cognitive impairment risk using ROC analyses with the MMSE 23/24 point cut-off as a reference standard.
Results: We found a sensitivity of 80% and a specificity of 77% for TYM (35/36 or 38/39 cut-off, depending on education). Corresponding values were 89% and 61% for GPCog (7/8 cut-off), respectively.
Conclusion: The TYM and GPCog instruments appear to be suitable for routine use in the primary care setting as tools for cognitive impairment risk detection in elderly rural populations.
KEY MESSAGES
Tools for cognitive screening in rural primary care must be brief, easily scored and relatively unaffected by sociodemographic factors.
TYM and GPCog scales display fair concordance with MMSE-based determination of cognitive decline risk and may be used with ease in clinical practice for cognitive screening of older adults in Greek rural settings.
Introduction
Detection of early signs of dementia is crucial and may be missed in a busy primary care practice until functional impairment becomes apparent [Citation1]. The Mini-Mental State Examination (MMSE) remains the most widely used screening tool for cognitive impairment [Citation2,Citation3]. Perhaps the most serious obstacle for incorporating the MMSE into primary care practice concerns the time and special training it requires for administration and scoring [Citation3] However, in primary care, MMSE may not be the best option since an ideal cognitive screening test in this setting should be brief, easily administered and scored, while maintaining adequate sensitivity and specificity [Citation3]. Such tools, which were specifically designed for general medical and primary care settings, are the Test Your Memory Test (TYM) and General Practitioner assessment of Cognition (GPCog) [Citation4,Citation5].
The TYM consists of eleven tasks or scales, assessing specific domains of memory related cognitive function, takes 5–10 min to complete with minimal supervision, and can be easily scored by non-medical personnel [Citation4]. The GPCog-Patient scale is a much shorter instrument that can be completed in less than 4 min by a health professional [Citation5]. Both instruments have been translated into several languages and their clinical validity has been assessed for detecting possible cognitive impairment against other screening instruments (such as the MMSE) and/or clinical diagnosis of dementia [Citation1,Citation4,Citation6]. Since the vast majority of validation studies have been conducted with clinical samples recruited primarily in urban settings, it is crucial to assess the validity of these new instruments for detecting risk for cognitive impairment in rural elderly populations, which are also characterized by limited formal education [Citation7]. This need is supported by considerable suggested adjustments to MMSE cut-offs [Citation7–10], potentially increasing false-negative rates in detecting dementia.
Whereas information on the validity of the Greek version of TYM has been reported previously (in a neurology clinic sample) [Citation11], such data is lacking for GPCog. Populations, such as those living in rural regions in Greece, include a very high proportion of persons who have completed fewer than six years of formal education and their capacity to cope with complex instructions and unfamiliar tasks may lead to an overestimation of cognitive impairment risk [Citation12]. The present study extends previous reports from populations with similar characteristics in other European countries regarding the development of primary care and struggle from the economic crisis (e.g. Portugal and Italy) [Citation7,Citation10], which were based primarily on the MMSE, and was designed to assess the clinical validity of two relatively novel cognitive screening tools (TYM and GPCog) in a largely rural, community dwelling sample of Greek elders.
Methods
General design, participants and data collection
The sample included 319 community dwelling elders aged 60 to 89 years who were randomly selected from a larger epidemiological cohort (n = 3140) of visitors in 14 PHC units (11 located in rural and semi-urban areas) in the Prefecture of Heraklion on the island of Crete, Greece (Cretan Aging Cohort). The overreaching project objective was to assess the burden of cognitive impairment among Cretan community dwelling elders and establish demographic, lifestyle, genetic, psycho-emotional, sleep, and inflammatory correlates of age-related cognitive decline and took place between March 2013 and May 2014. All PHC visitors aged ≥60 years were invited to participate by their GP. The reason for visit was mainly (90%) prescription renewal and response rate was 92.2%. Other than their willingness to participate in the study, no other exclusionary criteria were implemented.
The present sample included 64.6% women, retirees from manual labour professions (73%), and persons who had not attended high school (88.8%; and ) and was comparable to the overall cohort on the percentage of persons performing manual labour professions (74.8%, P = 0.2) and persons who did not attend secondary education (83.1%, P = 0.09), but included a higher percentage of women (56.2%; P = 0.007). A specially trained study nurse assessed sociodemographic factors and medical history in the participants’ native language (Greek) during a face-to-face interview, who also administered the MMSE, TYM, and GPCog instruments in random order across participants.
Table 1. Community sample demographics by cognitive impairment risk group.
Table 2. Community sample clinical characteristics and performance on MMSE, TYM, and GPCog by cognitive impairment risk group.
The study was approved by the Ethics Committee of the University Hospital of Heraklion, Crete and all participants provided written consent following detailed briefing on study purpose and procedures.
Instruments
TYM Test
The TYM Test consists of eleven tasks or scales evaluating specific memory functions. The TYM takes 5–10 min to complete with minimal supervision [Citation4]. The maximum score is 50 points (indicating perfect performance). Inter-rater agreement for scoring in the original English version was excellent (99%). The original validation study reported excellent sensitivity (93%) and specificity (86%) using a score of 42 points to differentiate between patients diagnosed independently with mild probable AD and controls [Citation4]. The procedure adopted for the translation and cultural adaptation of TYM in Greek is described in Iatraki et al. [Citation11].
GPCog scale
The GPCog-Patient section, which has a maximum score of nine points (indicating perfect performance), was translated and culturally adapted into Greek according to the Minimal Translation Criteria developed by the Scientific Advisory Committee of the Medical Outcomes Trust with written permission from the original developers of the instrument. In the original validation study, sensitivity and specificity against clinical diagnosis of dementia was 85% and 86%, respectively [Citation5].
Mini-mental state examination
The MMSE assesses five cognitive domains: orientation, registration, attention and calculation, recall, and language, yielding a total score of 30. In elderly patients’ samples from several Western countries including Greece displaying relatively high levels of literacy, sensitivity and specificity values ranged between 0.70 and 0.90 using the standard cut-off of 23/24 points [Citation2–4].
Data analysis
The contribution of age, education and gender on TYM, GPCog and for comparison, MMSE test scores was examined through multiple linear regression analyses (all variables were force-entered in the model). Given the low average education level of the sample and the distribution of MMSE scores, the originally suggested cut-off of 23/24 points was employed, resulting in 81 persons (25%) identified as at risk for cognitive impairment [Citation2,Citation13]. This cut-off was used to create the grouping variable against which sensitivity, specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV) of each test were estimated through ROC analyses. Optimal cut-off points for TYM and GPCog were determined using the Youden Index defined as the TYM or GPCog score associated with J = max {Sensitivity + Specificity -1} [Citation14].
In addition, the Bland–Altman method was used to assess the compatibility of TYM or GPCog with MMSE [Citation15]. Given that the three tools possess different measurement scales, raw scores were first z-transformed in the entire sample.
Results
Internal consistency estimates (Cronbach’s alpha) for MMSE, TYM, and GPCog were 0.80, 0.77, and 0.79, respectively. Preliminary analyses confirmed the significant effects of age (negative) and education level (positive) with TYM, GPCog, and MMSE scores. Interestingly, the effect of gender (higher age and education-adjusted scores for men) reached significance only for MMSE.
Associations between TYM and GPCog with reference standard MMSE scores
The associations between TYM and MMSE (R2 = 0.52, βlinear = 0.785, t = 3.48, P = 0.001, βquadratic = –0.066, t = –0.29, P = 0.77) and between GPCog and MMSE approached linearity (R2 = 0.43, βlinear = 0.761, t = 3.51, P = 0.001, βquadratic = –0.108, t = –0.50, P = 0.61; ). Moderated regression analyses further established that the magnitude of associations between TYM-MMSE and GPCog-MMSE did not vary with education level.
Table 3. Associations between demographic characteristics, TYM, MMSE and GPCog total scores. Zero-order Pearson correlations are shown above the diagonal and partial correlations controlling for age and education below the diagonal.
Clinical Validity of TYM and GPCog
The at-risk group (based on MMSE) included more women and older persons who had attained fewer years of formal education () and had lower average scores on TYM and GPCog ().
The results of the ROC analyses, conducted separately for persons with low (i.e. ≤5 years) and persons with greater education level (≥ 6 years), are summarized in . According to the J index, the optimal cut-off value on TYM is 35/36 points for low-educated persons, ensuring a minimum acceptable sensitivity of 80%. The corresponding cut-off for persons with higher education is 37/38 points (associated with 75% sensitivity). To ensure a minimum sensitivity of 80%, we suggest that a slightly higher cut-off value of 38/39 should be adopted.
Figure 1. Receiver Operating Characteristic Curves of TYM (upper panel) and GPCog scores (lower panel) for detecting risk of cognitive impairment based on MMSE performance.
For GPCog a universal cut-off of 7/8 points was associated with the maximum J index and 80% sensitivity. Overall sensitivity, specificity, PPV, and NPV values in the entire sample using these cut-off scores for each test are displayed in . The impact of education level on the clinical validity of both tests is further indicated by area under the curve values approaching 0.90 for persons with six or more years of education (indicating high discrimination accuracy) and 0.75–0.79 for persons with lower education (indicating moderate accuracy; ) [Citation16].
Table 4. Sensitivity and specificity for identifying at risk individuals using TYM and GPCog scores against MMSE-defined cognitive impairment risk. Optimal cut-off values associated with maximum J index (Sensitivity + Specificity –1).
The average difference of normalized scores between TYM and MMSE (0.0099) and between GPCog and MMSE (–0.0028) was not significantly different from 0 (P > 0.7 and P > 0.9, respectively), as per Bland–Altman technique. Inspection of the Bland–Altman plots in , reveals that the standardized difference for each pair of tests rarely exceeded one standard deviation from the mean (5/314 cases on TYM and 2/314 cases on GPCog). It was further noted that the greatest dispersion of difference scores was found among persons scoring in the low average range on both tests (i.e. within approximately 1 SD below the mean).
Figure 2. Bland–Altman plots illustrating the distribution of standardized score differences for each pair of tests (MMSE minus TYM in the left-hand panel and MMSE minus GPCog in the right-hand panel) as a function of corresponding average scores.
Discussion
Main findings
Data from this random, rural sample of community dwelling elders indicate that both TYM and GPCog display comparable internal consistency properties to the MMSE, and had fair concordance with the MMSE-based determination of risk for cognitive decline, as indicated by sensitivity of 80% and specificity of 77% for TYM, and corresponding values of 89% and 61% for GPCog.
Discussion in light of the literature
Comparison of TYM and GPCog with MMSE
Sensitivity and specificity estimates were virtually identical to the original TYM standardization study and to those obtained in a clinical sample of Greek elders [Citation11]. Internal consistency estimates for GPCog were also very like the original standardization study and somewhat higher than in other cultures (e.g. French and Chinese) [Citation5,Citation17,Citation18].
Sensitivity and specificity values for each instrument varied noticeably with educational level, highlighting the contribution of additional factors to TYM and GPCog scores, especially for persons with minimal formal education (e.g. literacy level associated with personal interests, occupational experiences and engagement in reading and other cultural activities).
Demographic effects
Demographic effects on TYM, GPCog and MMSE were similar with the exception of gender: although men and women displayed comparable scores on TYM and GPCog, age and education-adjusted MMSE scores were higher among men. In our sample age effects were modest (r = –0.41 to –0.45). The magnitude of the effect of educational level on the two instruments was comparable to MMSE (r = 0.38 to 0.49) and tended to be stronger for persons with fewer years of schooling. The effect of education level on TYM appeared to be more pronounced (further stressed by the need to implement a substantial correction of cut-off scores for persons with ≤5 years of formal education in order to maintain acceptable clinical validity). This finding may have been largely neglected in previous validation studies.
Educational level and literacy are known determinants of performance on screening instruments and a well-documented risk factor for dementia [Citation19,Citation20], although causality has not been firmly established [Citation21]. Previous reports have been inconclusive regarding the dependence of TYM and GPCog upon demographic variables [Citation22–24]. However, both the range of demographic variables sampled by each study cohort and the scarcity of persons with very few years of formal education may have biased earlier reports.
Strengths and limitations
The key limitation of the present study is that the clinical validity of TYM/GPCog was assessed through comparison with a screening test that was used to establish risk for cognitive impairment instead of clinical diagnosis of dementia. Although, MMSE is widely used and considered by many as the gold-standard instrument for dementia screening, it has far from perfect sensitivity or specificity [Citation3] In mixed urban/rural samples the Greek version of MMSE (using a 23/24 point cut-off) was associated with 0.70–0.80 sensitivity and 0.62–0.90 specificity [Citation11,Citation13]. Specificity for detecting clinically diagnosed dementia may be considerably lower, however, (40%) in rural elderly populations as suggested by unpublished data (n = 505) from the Cretan Aging Cohort (Basta et al., personal communication), in agreement with data from low literacy and/or rural elderly samples in a variety of cultures [Citation7–10]. Future studies should directly compare the three instruments against clinical diagnosis of dementia in the same rural elderly population.
Implications for clinical practice
TYM and GPCog can be administered by GPs or other health personnel with minimal training and appear to be suitable for primary care settings serving elders with no or minimal formal education [Citation3,Citation25]. They may prove particularly useful in rural areas where there are limited resources, expertise in state-supported primary care systems and accessibility to specialists, such as the case in Greece where, in addition, formal recommendations for screening tool use by primary care physicians are lacking. Only recently general guidelines have been published in Greece in the context of the National Action Plan for Dementia and Alzheimer’s Disease 2015–2020, stressing the importance of developing and using such tools and associated screening procedures in primary care [Citation26].
The limited gender and education-level susceptibility of GPCog render it preferable over TYM for routine use in primary care settings in accordance with a growing number of studies in other cultures [Citation17,Citation18,Citation23]. Additional desirable attributes of GPCog include the following [Citation1]: it requires 5 min or less to administer; it is validated in a primary care or community setting; it is easily administered by medical staff members who are not physicians; it has good to excellent psychometric properties; it is relatively free from educational, language, and/or culture bias; it can be used in a clinical setting without copyright cost [Citation27]. Its capacity to detect cognitive impairment may be further improved if supplemented by the GPCog Informant Scale [Citation28]. However, the final choice of instrument is left to the clinical judgment of the GP.
The generalizability of the current results to other, especially, southern European populations is supported by important historic and sociocultural similarities. Comparable characteristics include similar social structure featuring a prominent role of extended family and severely limited access to adequate quality, systematic basic education during WWII and the following years (due to severe poverty, internal conflicts and devastated state infrastructure and services). Additional similarities in current conditions of rural areas in, especially Mediterranean, European countries include higher percentage of older residents; lower per capita income and higher poverty rate, who face greater obstacles in accessing health services and high quality, specialized care [Citation29,Citation30].
Conclusion
The TYM and GPCog instruments demonstrate desirable properties for cognitive impairment risk detection in primary care settings, although the former test requires substantial adjustment of cut-off scores according to education level.
Acknowledgements
This work has been supported by a grant from the European Union (European Social Fund—ESF) and Greek national funds through the Operational Program ‘Education and Lifelong Learning’ of the National Strategic Reference Framework (NSRF)—Research Funding Program: THALES entitled ‘UOC-Multidisciplinary network for the study of Alzheimer’s Disease’ (Grant Code: MIS 377299).
The authors should like to thank Professors H. Brodaty and J. Brown for their valuable input during the translation and cultural adaptation procedure of the examining tools and their scientific support. The authors should also like to acknowledge the significant contribution of the THALIS Primary Health Care Research Team/Network for their contribution to the acquisition of normative and clinical data.
Part of the research described in this article was carried out in partial fulfilment of the dissertation requirements of one of the authors (E.I.).
Disclosure statement
The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.
Additional information
Funding
References
- Bradford A, Kunik ME, Schulz P, et al. Missed and delayed diagnosis of dementia in primary care: prevalence and contributing factors. Alzheimer Dis Assoc Disord. 2009;23:306–314.
- Folstein M, Folstein SE, McHugh P. ‘Mini-Mental State’: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12:189–198.
- Milne A, Culverwell A, Guss R, et al. Screening for dementia in primary care: a review of the use, efficacy and quality of measures. Int Psychogeriatr. 2008;20:911–926.
- Brown J, Pengas G, Dawson K, et al. Self administered cognitive screening test (TYM) for detection of Alzheimer's disease: cross sectional study. BMJ. 2009;338:b2030.
- Brodaty H, Pond D, Kemp NM, et al. The GPCOG: a new screening test for dementia designed for general practice. J Am Geriatr Soc. 2002;50:530–534.
- Holtzman NA, Watson MS. Final report of the task force on genetic testing. Promoting safe and effective genetic testing in the United States. Baltimore: Johns Hopkins University Press; 1999.
- Nunes B, Silva RD, Cruz VT, et al. Prevalence and pattern of cognitive impairment in rural and urban populations from northern Portugal. BMC Neurol. 2010;10:42.
- Ishizaki J, Meguro K, Ambo H, et al. A normative, community-based study of mini-mental state in elderly adults: the effect of age and educational level. J Gerontol B Psychol Sci Soc Sci. 1998;53:P359–P363.
- Pandav R, Fillenbaum G, Ratcliff G, et al. Sensitivity and specificity of cognitive and functional screening instruments for dementia: the Indo-U.S. dementia epidemiology study. J Am Geriatr Soc. 2002;50:554–561.
- Laks J, Batista EM, Guilherme ER, et al. Prevalence of cognitive and functional impairment in community-dwelling elderly: importance of evaluating activities of daily living. Arq Neuropsiquiatr. 2005;63:207–212.
- Iatraki Ε, Simos PG, Lionis C, et al. Cultural adaptation, standardization and clinical validity of the Test Your Memory dementia screening instrument in Greek. Dement Geriatr Cogn Disord. 2014;37:163–180.
- Schneider BC, Lichtenberg PA. Influence of reading ability on neuropsychological performance in African American elders. Arch Clin Neuropsychol. 2011;26:624–631.
- Fountoulakis ΚΝ, Tsolaki Μ, Chantzi H, et al. Mini Mental State Examination (MMSE): A validation study in Greece. Am J Alzheimers Dis Other Demen. 2000;15:342–345.
- Fluss R, Faraggi D, Reiser B. Estimation of the Youden Index and its associated cut-off point. Biom J. 2005;47:458–472.
- Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327:307–310.
- Fischer JE, Bachman LM, Jaeschke R. A readers’ guide to the interpretation of diagnostic test properties: clinical example of sepsis. Intensive Care Med. 2003;29:1043–1051.
- Thomas P, Hazif-Thomas C, Viéban F, et al. The GPcog for detecting a population with a high risk of dementia. Psychol Neuropsychiatr Vieil. 2006;4:69–77.
- Li X, Xiao S, Fang Y, et al. Validation of the general practitioner assessment of cognition - Chinese version (GPCOG-C) in China. Int Psychogeriatr. 2013;25:1649–1657.
- Cordell CB, Borson S, Boustani M, et al. Alzheimer’s Association recommendations for operationalizing the detection of cognitive impairment during the Medicare Annual wellness visit in a primary care setting. Alzheimers Dement. 2013;9:141–150.
- Stern Y, Gurland B, Tatemichi TK, et al. Influence of education and occupation on the incidence of Alzheimer’s disease. JAMA. 1994;271:1004–1010.
- Scarmeas N, Albert SM, Manly JJ, et al. Education and rates of cognitive decline in incident Alzheimer’s disease. J Neurol Neurosurg Psychiatr. 2006;77:308–316.
- Pirani A, Brodaty H, Martini E, et al. The validation of the Italian version of the GPCOG (GPCOG-IT): a contribution to cross-national implementation of a screening test for dementia in general practice. Int Psychogeriatr. 2010;22:82–90.
- Szczesniak D, Wojtynska R, Rymaszewska J. Test Your Memory (TYM) as a screening instrument in clinical practice – the Polish validation study. Aging Mental Health. 2013;17:863–868.
- Hanyu H, Maezono M, Sakurai H, et al. Japanese version of the Test Your Memory as a screening test in a Japanese memory clinic. Psychiatry Res. 2011;190:145–148.
- Yokomizo JE, Simon SS, Botino CM. Cognitive screening for dementia in primary care: a systematic review. Int Psychogeriatr. 2014;26:1783–1804.
- National Action Plan for dementia and Alzheimer’s disease. Greek Ministry of Health; 2014.
- Lorentz WJ, Scanlan JM, Borson S. Brief screening tests for dementia. Can J Psychiatry. 2002;47:723–733.
- Borson S, Scanlan JM, Watanabe J, et al. Improving identification of cognitive impairment in primary care. Int J Geriatr Psychiatry. 2006;21:349–355.
- European Commission: Directorate-General for Employment, Social Affairs and Equal Opportunities. Poverty and social exclusion in rural areas: Final Study Report. European Communities, 2008.
- Soler JK, Lionis C, Kaloeidas M, et al. Developing a Mediterranean family medicine group. Eur J Gen Pract. 2002;8:69–70, 74.