Examination of the validity of the ‘Papadum test’: An alternative to the clock drawing test for people with low levels of education

Abstract

Objectives: The clock drawing test (CDT) is a widely used cognitive screening test. However, CDT performance is affected by education. This study examined an alternative, the Papadum test, designed for people with low levels of education/literacy. The association between education and test performance, correlation between CDT and Papadum test, and diagnostic accuracy of both CDT and Papadum tests were examined. Method: 89 healthy literate adults and 59 literate adults (all Bengali speaking) with a diagnosis of mild-moderate dementia from hospitals in Kolkata, India undertook the CDT and the Papadum test. Results: Education had a significant association with the CDT but not with the Papadum test. Across the whole sample there was a significant correlation between CDT and Papadum, but not within separate groups of healthy controls and patients. Diagnostic accuracy for the Papadum test was similar to that for CDT. Conclusions: Results highlight the strong influence that education has on CDT performance indicating that it is not suitable for those with low levels of literacy. The Papadum test could provide a viable alternative as a screening tool to the CDT for use with people who are illiterate or have low levels of education. Further validation studies are required.

Keywords:

• Clock drawing test
• cognitive screening
• dementia
• education

Introduction

An increase in life expectancy is occurring across the world, with the greatest increase being seen in low-middle income countries (LMICs) (World Health Organization & Alzheimer’s Disease International, 2012). It is expected that the number of over 80 s will grow significantly by 2050 in several countries. Additionally, there is expected to be an increase in the number of people living with age related diseases, including dementia (Ferri et al., 2005; Prince et al., 2013). It is estimated that by 2040, 71% of 81.1 million dementia cases will be in lower income countries (Ferri et al., 2005).

Dementia is considered as one of the leading causes of disability among older people and typically contributes to many systemic and socioeconomic difficulties (World Health Organization & Alzheimer’s Disease International, 2012). In 2015, the total global societal cost of dementia was estimated to be $818 billion. The World Health Organization (World Health Organization & Alzheimer’s Disease International, 2012) recognises dementia as a public health priority. There is a clear need for effective neuropsychological testing that can be used to help screen for dementias to allow for timely diagnosis and intervention within low-middle income populations.

The Clock Drawing Test (CDT) was devised in the 20^th century and has been widely used in research protocols and in clinic for screening individuals suspected of having cognitive impairment or dementia (Critchley, 1953; Royall et al., 1998). In the last thirty years, an evidence base has formed for its use as an early screening tool of cognitive impairment, especially in Alzheimer’s disease (Scanlan et al., 2002). The CDT makes demands on a range of cognitive domains; memory, attention, visual memory, reconstruction, planning, motor skills, visuospatial abilities, concentration, and auditory comprehension (Royall et al., 1998). It is a brief tool that has been widely applied internationally and involves asking the client to draw or copy the face of a clock and then to draw the hands to indicate a particular time. The CDT is included as a subtest in the Addenbrooke’s Cognitive Examination (ACE III; Hsieh et al., 2013), which is a widely used cognitive screening tool that assesses five cognitive domains; memory, attention, language, visuospatial and perceptual abilities.

The CDT is a very popular screening tool; however, it was developed to be used in Western, educated populations. One of the challenges in screening for dementia in LMICs is related to performance variances due to educational and cultural influences. It is widely documented that neuropsychological test performance may be considerably influenced by aspects such as culture, language, education, and literacy (Ardila, 2005; Ardila et al., 1989). Strong evidence has further developed over the last decade indicating that CDT results can be influenced by education (Kim & Chey, 2010; Nielsen & Jorgensen, 2013) which can lower the specificity of the test. Studies have discussed significant limitations in the use of the CDT with people with low levels of education or literacy. Lieberman et al. (1999) found that poorer performance on the CDT has also been observed in those who do not speak English.

De Noronha et al. (2018) found that CDT performance was significantly impacted in illiterate individuals. Illiteracy refers to the inability to read or write a simple message (UNESCO Institute of Statistics, 2017). Literacy rates in LMICs vary among older generations and amongst individuals in rural areas who often receive limited or no formal education. Nielsen and Jorgensen (2013) further found that healthy illiterate individuals may experience problems with graphomotor construction when asked to engage in the CDT task. Another commonly used screening tool for cognitive impairment, the Mini-Mental State Examination (MMSE) has also been shown to be influenced by schooling, displaying a lack of validity in illiterate populations (Kalafat et al., 2003). As well as potentially having an influence on performance, low education and illiteracy are considered major risk factors for developing Alzheimer’s disease (Stern et al., 1994) in LMICs.

Levels of literacy in India are particularly low; although the country has made significant progress in improving literacy over the years, it continues to be home to 313 million illiterate people (UNESCO Institute of Statistics, 2017). Tripathi et al. (2020) recently examined the usefulness of the CDT in screening Indian older adults for cognitive impairment and found that education and language are significant variables that correlate with CDT performance. Apart from difficulties with screening tools, other difficulties can exist regarding a general lack of awareness of dementia in the general population in India, in particular attributing common symptoms to “normal aging” (Khan, 2011).

The present study examined the validity of a task designed to examine similar cognitive skills to the CDT, but which does not require the ability to read or write and therefore may serve as an alternative to the CDT for illiterate or low educated populations. The task we designed is referred to as the Pizza test (or Papadum test, depending on the cultural context in which it is used), and involves a person being provided with an 18 cm diameter paper circle and being asked to imagine it is a pizza (or papadum). They are told that they should divide the pizza between six people equally, so they must fold and divide the paper into six equal pieces. We hypothesise that this test makes demands on cognitive abilities such as attention, planning, problem solving, visuo-spatial and praxis skills and should therefore perform in a similar way to the CDT. It can be adapted culturally and was designed to be an educationally unbiased alternative to the CDT. A study carried out in NHS Greater Glasgow and Clyde, Scotland previously examined the validity of the Pizza test (Manoj et al., 2015) – twenty-nine adults who underwent a cognitive screening as part of a dementia assessment were given the Addenbrookes Cognitive Examination (ACE III) which includes the CDT, and the Pizza test. Results showed that scores on the Pizza test was not associated with level of education (though all participants had at least high school levels of education), and performance correlated with the CDT (rho = 0.627). The Pizza test also showed good diagnostic accuracy (AUC 0.809). It was concluded that the Pizza test may be a good alternative to the CDT in those with low levels of education or literacy. However, as this preliminary study was carried out in a cultural context with relatively high minimum levels of education it was important to examine its validity in a context where educational and literacy levels vary much more.

The data for this present study were collected in Kolkata, India, where the “Pizza” test was referred to as the “Papadum” test. The study included the use of a paper “papadum” and an actual papadum. Literate participants were included as they were able to attempt both the CDT and papadum to examine correlations between performance on the two tests. The impact of education on the CDT and the Papadum test performance was examined. In addition, the diagnostic accuracy of the CDT and Papadum test was examined.

Hypothesis

It was hypothesised that education would have a significant association with the CDT but not with the Papadum test. Performance on the Papadum test (both paper and actual papadum versions) was predicted to be significantly correlated with the performance on the CDT.

Exploratory analyses were conducted to assess diagnostic accuracy of the Papadum test.

Method

Participants

A total of 59 literate patients clinically diagnosed with Alzheimer’s disease (AD) or Vascular dementia (VaD) participated in the study. As AD and VaD are the most common causes of dementia, these two types were included in the study. Patients with AD and VaD in the mild and moderate stages of the illness as evident from their scores on the Mini Mental State Examination (MMSE) (Folstein et al., 1975)/Bengali Mental State Examination (BMSE) (Das et al., 2006) as well as their Clinical Dementia Rating (Hughes et al., 1982) scores were invited to participate in the study. The patients were seen at Duttanagar Mental Health Centre and at Apollo Gleneagles Hospitals, Kolkata, India. Patients were assessed by a neurologist or a psychiatrist and underwent an MRI or CT scan. Each patient underwent a comprehensive neuropsychological examination by an experienced neuropsychologist which included a range of neuropsychological tests. Clinicians making the diagnosis of dementia based on clinical history, imaging and neuropsychological test results were blind to Papadum test scores. Patients with AD with a history of cerebrovascular disease or significant changes in the brain suggestive of additional cerebrovascular pathology were excluded from the study. None of the patients included in the study had other neurological illnesses, history of psychiatric illness, head injury, major medical illness, or substance abuse.

A total of 89 cognitively healthy literate adults were also included in this study who all resided in Kolkata and were in the age range of 40 years and above. These participants were a convenience sample of a) relatives or friends of patients attending the Neuropsychology and Clinical Psychology unit at Duttanagar Mental Health Centre, b) family members of other patients attending the hospitals, c) volunteer hospital staff, or d) members in the community (acquaintances of other participants who volunteered for the study). Inclusion criteria included having a minimum of one year of education, with a MMSE/BMSE score of above 25. The Hospital Anxiety and Depression Scale (HADS) was also administered. Exclusion criteria were applied; individuals with cognitive complaints, hearing or vision problems or any history of neurological or psychiatric illnesses were excluded from this study.

The primary language of all participants was Bengali. Demographic details of age, sex and years of education were gathered by a structured interview for each participant.

Ethical approval

Ethical approval for the study was obtained from the Institutional Ethics Committees of Apollo Gleneagles Hospitals, Kolkata and Duttanagar Mental Health Centre, Kolkata. Informed consent was sought and provided by all participants.

Materials

Addenbrooke’s cognitive examination (ACE) III – Bengali version (Radley, Evans and Dutt, in preparation)

The ACE III – Bengali Version was administered to all participants. The ACE III is a brief cognitive screening tool that assesses five cognitive domains: attention, memory, verbal fluency, language, and visuospatial abilities. The maximum total score is 100. The ACE III includes the Clock Drawing Test as a subtest to assess visuospatial abilities. For the purposes of this paper, only scores of the CDT component will be reported.

Participants were asked to draw a clock including all the numbers and set the hands to ten past five.

The clocks were scored quantitatively according to two scoring systems based on the clockface, numbers and hands using the ACE III 5-point scoring system and the Rouleau 10-point scoring system (Rouleau et al., 1992) (supplementary material 1 – available online only). This paper will focus on the analysis of the Rouleau system scores only.

Some of the scoring criteria were modified while using the Rouleau quantitative scoring system to capture culturally different information, for example, clock numbers written partly in English script and partly in Bengali. Each clock was scored independently by two members of the research team.

Papadum test

The Papadum test was administered to all the participants. Two versions of this test were used;

1. Paper papadum: The individual was given a circular piece of paper measuring 18cm in diameter. The instructions were: “Imagine that this is a papadum which you have to share amongst six members in your family. Could you kindly show me how you will tear the paper so that the six members in your family get an equal share”.

2. Actual papadum: A dried/unfried papadum was given to each participant. They were asked to divide it into six equal slices. The instructions were: “Imagine that you have to share this papadum amongst six members in your family. Could you kindly show me how you will tear the papadum so that the six members in your family get an equal share”.

See supplementary material 2 (available online only) for scoring criteria. The maximum score was 18. Each papadum was scored independently by two members of the research team.

Additional clinical data

Mini-mental state examination (MMSE) or Bengali Mental State Examination and Clinical Dementia Rating scores were obtained from clinical records.

Statistical analysis

Scoring reliability for the CDT and Papadum tests was examined. Two members of the research team rated all the CDT and Papadum performances. Where discrepancies occurred between raters, a third senior member of the research team was consulted, and a final score was determined. An example of where there were discrepancies between raters was in relation to interpretation of the threshold for what constituted a “rotation” of numbers on the clock face.

We examined whether there was a correlation between scores on the Papadum Test with the CDT scores within the whole sample, healthy controls and patients. As the data were not normally distributed, Spearman’s Correlations were carried out. The effects of education, age and sex on performance on both the CDT and Papadum tests were examined to determine whether demographic factors are associated with test performance. Regression analyses were also carried out.

We examined the relationship between ACE III scores and papadum test scores. In the patients we also examined the association between the CDT/Papadum tests and the MMSE/CDR scores. Given the correlation between CDT and education (and between MMSE and education) we used regression models that used MMSE/CDR as dependent variables and included education and CDT/Papadum as predictors.

An exploratory comparison of the AD and VaD patients on the CDT and Papadum tests was carried out using a Mann Whitney test.

The diagnostic accuracies of the tests were examined using Receiver Operating Characteristic (ROC) curves for the data from the patients and healthy controls.

Results

Demographic characteristics

One hundred and forty eight adults (73 males and 75 females) were included in the study and analysed. Of this total, 89 were healthy controls and 59 were patients with a diagnosis of dementia (30 had AD, whilst 29 had VaD). Mean age (years) and education (years) of the total sample were 63.24 (SD = 11.46; max 87, min 40) and 11.22 (SD = 5.01, max 21, min 1) respectively. With a total of 148 participants, the correlation analysis had 80% power to detect correlation as small as r = 0.225.

Participant characteristics for patients and healthy controls are provided in Table 1. The healthy controls were significantly younger than the patients and also had a significantly lower level of education than the patients. There was a significant difference in terms of numbers of males/females between the groups with more females than males in the healthy group but more males than females in the patient group.

Table 1. Participant characteristics.

	Healthy Sample (n = 89)	Patient Sample (n = 59)	Statistical comparison
Age (years) – Mean (SD; Range)	58.97 (SD 11.12) (Range 40-86)	69.69 (SD 8.65) (Range 48-87)	t(142.2) = −6.579, p < 0.001, d = 1.05
Female/Male	50/39	23/36	(χ2 (1) = 4.20, p < 0.04
Education (years) – Mean (SD; Range)	9.58 (SD 5.16) (Range 1- 21 yrs)	13.68 (SD 3.62) (Range 4-21 yrs).	t(145.5)=-5.674 p < 0.001, d=.89
MMSE/BMSE^1 – Mean (SD)	27.79 (1.47) (Min 26 Max30)	18.88 (5.28) (Min 9; Max 28)	–
CDR^2 – Median (IQR; Min; Max)	–	1.0 (IQR 1.0; Min 0.5; Max 2.0)	–
ACE III^3 – Median (IQR; Min; Max)	76.0 (IQR 22.5; Min 24; Max 99)	45.0 (IQR 30.0; Min 17; Max 80)	U = 666.5; Z=-7.673, p < 0.001; d = 1.652

1 Mini-Mental State Examination/Bengali Mental State Examination.

2 Clinical Dementia Rating.

3 Addenbrookes Cognitive Examination III (Bengali version).

The data for the Papadum and CDT tests for all participants as well as patients and healthy controls separately are included in Table 2.

Table 2. Actual Papadum, Paper Papadum and Clock Drawing Test (CDT) test data for all participants, patients, and healthy controls.

Group	Tests	Mean	SD	Minimum score obtained	Maximum score obtained	Max possible score
All Participants (n = 148)	Actual Papadum	12.47	4.10	0	18	18
	Paper Papadum	12.8	5.08	0	18	18
	CDT	6.76	2.68	0	10	10
Patients (n = 59)	Actual Papadum	10.03	3.85	0	18	18
	Paper Papadum	9.85	4.97	0	18	18
	CDT	5.08	2.52	0	10	10
Healthy Controls(n = 89)	Actual Papadum	14.09	3.41	6	18	18
	Paper Papadum	14.81	4.12	0	18	18
	CDT	7.87	2.17	1	10	10

Inter-rater reliability

Regarding inter-rater reliability of papadum and CDT scoring, there was 76% agreement between raters for the papadum scores, 86% agreement for the paper papadum scores and 80% agreement for the CDT scores. Scoring challenges emerged for the CDT in applying the Rouleau method and therefore adaptions/clarifications were agreed; for an error in “rotation of numbers”, six numbers or more rotated numbers were required to be interpreted as rotated. For the scoring of the Papadum tests, discrepancies emerged in relation to what constituted as a triangular shape; it was agreed that a minimum ratio of 2:1 regarding the top of the section of Papadum being at least twice the length of the bottom part was required to consider the Papadum to be “triangular” in shape.

To examine the distributions of the data, histograms and box plots were used to visually analyse whether the data were distributed normally. The Kolmogorov-Smirnov Test to assess the normality in the study data (actual papadum, paper papadum and CDT) was also used. The following results were obtained; D(148)=.126, p < 0.001 for the actual papadum test, D(148)=.203, p < 0.001 for the paper papadum test and D(148)=.206, p<.001 for the CDT. Therefore, data did not follow a normal distribution and nonparametric analyses were used.

Education and performance on the tests in healthy controls

The relationship between education and performance on CDT, Papadum and Paper Papadum within the healthy controls was examined using Spearman Correlation. The correlation co-efficient for actual papadum was rs=-.010, p=.927; for the paper papadum it was rs=.054, p = 0.618; and for CDT it was rs = 0.507, p<.001. Thus, there was a significant relationship between education and CDT performance but there was no association between either paper or actual papadum and education.

Figure 1 presents a scatterplot of education (in years) against CDT score to illustrate the nature of the relationship in the healthy controls. The plot shows that there is a curvilinear relationship such that CDT scores increase with education up to about 8 years, and then plateaus.

Figure 1. Scatterplot of education (in years) against CDT total score in healthy controls.

This was explored in a regression analysis that included CDT as the dependent variable and education in an initial model and then education squared was added to the model. The results confirmed that addition of education squared increased the predictive power of the model (Model 1, F (1,87) = 39.25, p<.001, R²=.311; Model 2, F(2, 86) = 29.16, p < 0.001, R²=.390, R²change=.093, Fchange (1,86) =13.45, p < 0.001).

Age & performance on the tests in healthy controls

The relationship between age and performance on CDT, actual papadum and paper papadum within the healthy controls was examined using Spearman correlation. The correlation co-efficient for actual papadum was rs= .014, p=.898, for paper papadum was rs= −.017, p=.876 and for CDT rs=-.054, p=.612.

Sex & performance on the tests in healthy controls

There was no significant effect of sex for the actual papadum (U = 932, Z= −.357, p = 0.721) or paper papadum (U = 875, Z= −.898, p=.369) amongst the healthy controls. For the CDT, females had lower scores though the difference was not significant, with a small-medium effect size (U = 744, Z= −1.954, p = 0.051, r = 0.21). The female healthy controls had a lower level of education and in a regression model with CDT as the dependent variable and education and sex as predictors, only education was significantly associated with CDT performance (beta=.539, t = 5.941, p < 0.001), whilst sex was not a significant predictor (beta= −.097, t= −1.069, p=.288).

Relationship between tests for total sample, healthy controls and patient group

The Spearman correlation for the total sample (healthy controls and patients) between the actual papadum and CDT was rs=.221, p=.007; the paper papadum and CDT was rs=.304, p<.001 and the correlation between actual papadum and paper papadum was rs=.582, p= <.001.

However, there was no significant relationship between CDT and papadum tests when looking at the healthy and patient groups separately. The Spearman correlation for healthy controls between the actual papadum and the CDT was rs= −.052, p=.626; the paper papadum and the CDT was rs= −.038, p=.724 and the correlation between the actual papadum and the paper papadum was rs =.486, p <.001. A regression analysis with CDT as dependent variable and paper papadum, education, age and sex as predictors found that education was a significant predictor (beta = 0.565, t = 6.083, p < 0.001) but paper papadum was not (beta=-.056, t= −.625, p=.534) and nor was sex (beta= −.116, t = 1.248, p=.215) or age (beta=-.112, t= −1.204, p=.232). A similar result was obtained when using the actual papadum score, education, age and sex as predictors, such that education was a significant predictor (beta=.559, t = 6.051, p < 0.001) but actual papadum was not (beta= −.082, t= −.926, p=.357) and nor was sex (beta= −.122, t= −1.317, p=.191) or age (beta= −.115, t= −1.229, .223).

For the patient sample the Spearman correlation for actual papadum and CDT was .041, p=.760; paper papadum and CDT was .184, p=.164 and the correlation for actual papadum and paper papadum was .379, p=.003. A regression analysis with CDT as dependent variable and paper papadum, age, education and sex as predictors found that sex was a significant predictor (beta=-.344, t = 2.517, p = 0.015) but paper papadum (beta=.102, t=.836, p=.407), age (beta 0.39, t=.319, p = 751) and education (beta=.152, t = 1.120, p=.268) were not. A similar result was obtained when using the actual papadum score, age, sex and education as predictors, such that sex was a significant predictor (beta=-.346, t = 2.517, p = 0.015) but actual papadum (beta=.037, t=.301, p=.765), age (beta = 0.52, t = 4.16, p=.679) and education were not (beta=.160, t = 1.175, p=.245). This was further explored by examining dementia severity data from clinician ratings on the Clinical Dementia Rating Scale (CDR; Hughes et al., 1982). For the patient group the median score was 1, interquartile range 1. Females (Median =2) were found to be more severely impaired than males (Median =1) (U = 276, Z=-2.335, p=.020) suggesting that gender was associated with CDT performance as a result of differences in dementia severity.

As the patients and control groups differed in mean age, length of education and sex, the groups were compared with a binary logistic regression, with group as dependent variable, and age, education and sex as covariates along with actual Papadum, paper Papadum or CDT scores in three separate models. Results showed that each of the tests significantly distinguished the groups over and above differences in age/education/sex. The logistic regression model for CDT was statistically significant (χ² (3) =111.47, p < 0.001, Nagelkerke R²=.716), with CDT score being a significant predictor of group membership (p < 0.001), correctly classifying 87.8% of participants. Similarly, the model for the actual Papadum was significant (χ² (3) =72.07, p < 0.001, Nagelkerke R²=.521), with actual Papadum score being a significant predictor of group membership (p < 0.001), correctly classifying 83.8% of participants, and finally the model for the paper Papadum was also significant (χ² (3) =76.76, p < 0.001, Nagelkerke R²=.547) with paper Papadum score being a significant predictor of group membership (p < 0.001), correctly classifying 80.4% of participants.

Correlation between ACE III and papadum tests

In the whole sample there was a significant but modest correlation between ACE III total scores and both the actual Papadum (rho = 0.303, p < 0.001) and the paper Papadum (rho = 0.391, p < 0.001). However, within the separate healthy and patient groups correlations were not significant.

The correlations between the ACE III sub-scale scores and the Papadum tests were also examined in the whole sample. Subscale scores from three patients were not available and so for this analysis there were 89 controls and 56 patients. For the actual Papadum they were: Attention (rho=.292, p<.001), Memory (rho=.334, p<.001), Fluency (rho=.200, p = 0.016), Language (rho=.196, p = 0.018) and Visuospatial (rho=.257, p = 0.002). For the paper Papadum they were Attention (rho=.344, p<.001), Memory (rho=.392, p<.001), Fluency (rho=.304, p<.001), Language (rho=.267, p = 0.001) and Visuospatial (rho=.381, p < 0.001). Within the separate healthy and patient groups none of the correlations were significant.

Association between CDT and papadum tests and MMSE/CDR in patients

The regression model for MMSE with CDT and education as predictors was significant (R² = .409, F(2,56) =19.386; p<.001) with both education (beta = .245, t = 2.259, p = 0.028) and CDT (beta = .517, t = 4.764, p < 0.001) significant independent predictors. The models for MMSE with paper papadum/actual papadum and education as predictors were significant but, in each case, it was only education that was a significant independent predictor.

The regression model for CDR with CDT and education as predictors was significant (R² = .248, F(2,56) =9.236; p<.001) with the only significant independent predictor being CDT (beta = −.471, t=-3.849, p < 0.001). The regression model for CDR with paper papadum and education as predictors was also significant (R² = .155, F(2,56) =5.121; p = 0.009) with the only independent predictor being paper papadum (beta = −.326, t=-2.644, p=.011). In the regression model for CDR with actual papadum and education as significant predictors, neither actual papadum nor education were significant independent predictors.

Comparison of AD and VaD patient performance

There was no significant difference in performance between the AD and VaD groups on the CDT (AD Mean 4.73 (SD 2.48), Median = 4 (IQR = 3); VaD Mean 5.45 (SD 2.56), Median =5 (IQR 5); Mann Whitney U = 353, Z=-1.255, p = 0.209); Paper Papadum (AD Mean 9.97 (SD 4.52), Median = 10 (IQR = 4); VaD Mean 9.76 (SD 5.48), Median =10 (IQR 6); Mann Whitney U = 426, Z= −0.140, p = 0.889); or Actual Papadum (AD Mean 9.50 (SD 4.15), Median = 10 (IQR = 3); VaD Mean 10.59 (SD 3.51), Median =11 (IQR 4); Mann Whitney U = 316.5, Z=-1.816, p = 0.069).

Diagnostic accuracy

Receiver Operating Characteristic (ROC) curves for the actual papadum (Figure 2), paper papadum (Figure 3) and CDT scores (Figure 4) differentiating participants with dementia and healthy controls were constructed. The area under the curve (AUC) for the CDT was .794, the optimal cut off score for the CDT in this study was found to be 6.5 which displayed a sensitivity of .712 and specificity of .82 with Youden’s index as 0.532 and LR + 2.847 and LR- 0.253. Positive and negative predictive values were PPV (0.811) and NPV (0.724). For scoring of the CDT, Rouleau et al. (1992) stated ≤7 as cut off and Duro et al. (2018) found ≤7 also.

Figure 2. ROC curve for actual Papadum.

Figure 3. ROC curve for paper Papadum.

Figure 4. ROC curve for CDT.

The AUC for the actual Papadum was .778, the optimal cut off score for the Papadum in this study was found to be 13.5 with a sensitivity of .847 and specificity of .596 with Youden’s index as 0.443 and LR + 3.895 and LR- 0.477. Positive and negative predictive values were PPV (0.855) and NPV (0.581).

An alternative approach to selecting cut-offs is to focus on a definition of impairment based on percentiles. For the actual papadum in the healthy controls, only 2.2% of the sample scored 8 or less and this therefore may be a useful means of defining impaired performance. With this cut off, 30.5% of the patient sample scored 8 or less.

The AUC for the Paper papadum was .782, the optimal cut off score for the Paper papadum version in this study was found to be 15.5 with a sensitivity of .881 and specificity of .562 with Youden’s index as 0.443 and LR + 4.723 and LR- 0.497. Positive and negative predictive values were PPV (0.877) and NPV (0.571). Manoj et al. (2015)’s Pizza study reported a cut-off of 13/18 and was previously found to show a maximum Youden index of 0.50 with a sensitivity of 0.643 with a specificity of 0.857, AUC 0.802.

Alternatively, selecting a cut-off based on percentiles for the paper papadum in the healthy controls, only 3.4% of the sample scored 9 or less and this therefore may be a useful means of defining impaired performance. With this cut off, 31% of the patient sample scored 9 or less.

Discussion

This study shows that CDT performance is impacted by level of education. The association with education was strong in the lower end of educational range (up to about 8 years of education) and then reduces as educational levels increase, suggesting that the test may be useful for those with higher levels of education but is problematic in relation to use with people with less than about eight years of education. This finding is consistent with several other studies in other parts of the world that have demonstrated the effect of education on CDT performance (Ainslie & Murden, 1993; Crombie & Evans, 2021; Kim & Chey, 2010) meaning that caution must be used if this test is used as a screening tool for cognitive impairment or dementia in less educated or illiterate populations.

The aim of the development of the Papadum/Pizza test was to capture cognitive abilities such as attention, planning, problem solving, praxis skills, visuo spatial and visuo constructive abilities, similar to those assumed to be required for effective performance of the CDT, but without the requirement for a certain level of literacy or experience in the use of writing implements. The Papadum/Pizza test is quickly administered, taking on average two to three minutes. It does not involve extensive training to administer or score.

The study highlights the challenge of taking a test scoring system designed in a Western, English-speaking context and using it in a different cultural and linguistic context. The Rouleau CDT scoring system was adapted to address the common inclusion by healthy controls of numbers on the clock in different scripts (i.e. Bengali and English). It also highlighted the importance of having a robust reliability analysis process to ensure consistency of scoring between raters. Despite using a well-established scoring system there were examples of where well-trained raters gave different scores, requiring adjudication from additional senior researchers. For this study this process ensured consistency of scoring between the dementia and healthy control groups. But it raises questions regarding the reliability of scoring of tasks such as the CDT in clinical practice, particularly in a cross-cultural context where issues may arise that would not have occurred in the well-educated, mono-lingual context in which the test scoring system was originally developed. No association was found between education and the paper or actual Papadum tests. This was what was expected and supports the idea that the Papadum/Pizza test may be useful for those with low levels of education.

There were significant correlations between both versions of the Papadum (actual and paper) test and the CDT when analysing the sample of healthy controls and patients combined. However, within just the healthy controls (and within the patient group) correlations were very low. A stronger relationship between CDT and the Pizza test was reported in Manoj et al. (2015) study. In the present study, there was a much greater range of education which may have been impacting on the CDT performance to a much greater extent than in Manoj et al.’s study. Another possibility is that the cognitive skills required for the CDT are different to those needed to complete the Papadum test. It is possible that there is overlap in the cognitive demands of each test, but the relative contribution of various cognitive skills varies between the tasks, reducing the precision of any association between test performance. Although the diagnostic accuracy of the Papadum test was similar to that of the CDT, given the results from this study it is not yet clear which specific constructs the Papadum test is measuring. Another relevant factor is that screening tools such as the CDT typically have a highly skewed distribution of scores as they are relatively “easy” tasks that are completed perfectly by a large proportion of participants. In our healthy control sample, although there was a wide range of scores, 73% of participants scored within the 8-10 point range (max score is 10). Similarly, for the papadum tests there was also a full range of scores obtained but for the actual papadum test, 28.1% scored the maximum score of 18, and for the paper papadum 52.8% scored the maximum score.

We did not find a significant relationship between age and CDT or Papadum Test performance. This contrasts with some of the previous studies of the CDT that have indicated significant differences across age groups or significant correlations with age on neuropsychological screening tools (Liu et al., 2011).

We also did not find a relationship between sex and Papadum test performance. Regarding CDT performance for the patient sample, there was an effect of sex on performance. Women were shown to be more significantly impaired in their dementia ratings therefore this may have influenced results. On the CDT (healthy controls), the effect of sex was just above the cut off for significance, with a small-medium effect size. As the females had a slightly lower level of education it was possible that education differences could account for the modest difference in CDT performance between males and females, and it was noted that in a regression model with education and sex as predictors, only education significantly predicted CDT performance. Previous studies have not found an influence of sex on CDT performance (Kim & Chey, 2010; Shanhu et al., 2019).

Across the whole sample there was a significant, medium correlation between ACE III scores and both versions of the papadum test. However, in the separate groups the association was not significant. As the ACE III assess a wide range of other cognitive functions that may vary considerably in patients with dementia, it is perhaps not too surprising that correlations would be modest. The lack of association between ACE III and papadum tests in the healthy controls may also be because variance in ACE III total scores is driven by variability in a wide range of cognitive functions that are not required for papadum performance. At the level of subscale scores, across the whole sample there were again significant but modest correlations between both versions of the Papadum test and the ACE III sub-scales. The paper Papadum showed slightly higher correlations, with the highest being for the association with the visuospatial and memory domains, and the lowest being with fluency and language domains. Nevertheless, correlations were modest and in the separate patient/healthy samples there were no significant correlations with any of the domains. This suggests again that it is not clear what constructs are being captured by the Papadum tests.

In the patient group CDT scores were associated with MMSE scores (controlling for education), which is consistent with previous findings (Shulman, 2000) but this was not the case for the Papadum tests. This suggests that the CDT was sensitive to processes also reflected in the MMSE tasks. The papadum tests were not associated with MMSE performance, again perhaps indicating that different cognitive or psychological factors were influencing performance. The paper papadum was however associated with CDR scores suggesting it is sensitive to dementia severity.

In relation to diagnostic accuracy, the area under the curve (AUC) measures (which were all in the “fair” range), sensitivity, and specificity for CDT and both forms of the Papadum test were similar, albeit modest. The fact that a test aimed at testing cognitive skills such as planning, praxis, and visuo-spatial functions was completed relatively successfully by some patients with dementia is not surprising given that deficits in these areas are not required for a diagnosis of dementia, particularly Alzheimer’s disease, even though they may be present in some patients.

The Papadum test could provide a viable alternative as a screening tool to the CDT for use with people who are illiterate or have low levels of education, as results were not influenced by level of education. If a participant performs poorly on the Papadum, it could indicate difficulties with visuo-spatial function and/or with planning and would indicate further investigation is required. Other ways of testing the specific constructs of interest may be required. Thus, as with the CDT, the Papadum test could be used as a screening tool, as a means of detecting problems that can be investigated in further assessment (i.e. favouring sensitivity over specificity). It is most realistic that the paper version of the Papadum would be utilised in clinic settings.

Future research

Future studies should assess those who are illiterate to assess diagnostic accuracy of the Papadum test in relation to the diagnosis of dementia. In relation to the CDT, in this study, the Rouleau scoring system was used but was adapted slightly to include information such as including two scripts (e.g. Bengali and English) in the clock face, which was believed to be important to capture the cultural differences. Future research could also include using various scoring versions of the CDT to compare with Papadum/Pizza test performance. The variability of scoring methods can mean that the results of various studies are difficult to compare (Shulman, 2000). Future research could also consider the inclusion of qualitative analysis.

Regarding future validation studies, it will be important to assess the influence that any motor impairment or visual difficulties have on the Papadum test performance.

There was a wide range of education amongst the participants in this study, though the average length of education for both patients and healthy controls was well above the average for the population in India. This in part reflects the fact that only literate participants were included to ensure that participants could do both the CDT and Papadum tasks. It also reflects the population who attended the hospitals from where participants were recruited. It is important that future studies aim to match participant recruitment closer to population averages to increase confidence that results would generalise to the wider population. Recruitment of larger samples would also allow for more precise matching between patients and healthy controls. The use of regression models allowed age, education and sex to be included as covariates when examining the relationship between CDT and Papadum performance, but it would be good for future studies to match samples on these demographic factors.

There was no difference in performance between the AD and VaD groups on either the CDT or Papadum tests. The study was not powered apriori to make this specific comparison, but it is noteworthy that effect sizes were very small. The result is, however, consistent with findings from the reivew of Tan et al. (2015) who reported that most studies of the CDT have found no difference in CDT scores between AD, VaD, Dementia with Lewy Body or Parkinson’s disease dementia, though patients with fronto-temporal dementia consistently score higher on the CDT compared to AD patients. It would be interesting to see if there are similar results for the Papadum tests, when AD patients are compared with patients with forms of dementia that more strongly affect visuo-spatial or executive planning functions.

One limitation of the study was that the data were collected in India and some of the scoring to examine reliability was undertaken using photographs of the stimuli. Although it is believed that this did not compromise the scoring, it would be better in future (where possible) to score original materials (CDT or Papadum) as this may improve reliability further. Other limitations include that no measure of performance validity was included and the sample was derived from one location (albeit two centres within this location) indicating that the results may not be representative of the wider Indian population.

Conclusion

Education had a significant association with the CDT but not with the Papadum test. Across the whole sample there was a significant correlation between CDT and Papadum but not within separate groups of healthy controls and patients. Diagnostic accuracy for CDT was similar to that for Papadum.

This study further provides evidence to support the findings from the Pizza test study (Manoj et al., 2015) that the pizza/papadum test may be useful as an alternative to the CDT that is not impacted by education. However further validation studies are required before we can conclude that this task will be useful as an alternative to the CDT with individuals who have a lack of education or literacy as a means of assessing planning, visuospatial and praxis skills that may be impaired as part of a neurodegenerative process.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

The author(s) reported there is no funding associated with the work featured in this article.