56
Views
0
CrossRef citations to date
0
Altmetric
Original Research

Maximally efficient two-stage screening: Determining intellectual disability in Taiwanese military conscripts

, &
Pages 39-44 | Published online: 27 Jan 2009

Abstract

Objective:

The purpose of this study was to apply a two-stage screening method for the large-scale intelligence screening of military conscripts.

Methods:

We collected 99 conscripted soldiers whose educational levels were senior high school level or lower to be the participants. Every participant was required to take the Wisconsin Card Sorting Test (WCST) and the Wechsler Adult Intelligence Scale-Revised (WAIS-R) assessments.

Results:

Logistic regression analysis showed the conceptual level responses (CLR) index of the WCST was the most significant index for determining intellectual disability (ID; FIQ ≤ 84). We used the receiver operating characteristic curve to determine the optimum cut-off point of CLR. The optimum one cut-off point of CLR was 66; the two cut-off points were 49 and 66. Comparing the two-stage window screening with the two-stage positive screening, the area under the curve and the positive predictive value increased. Moreover, the cost of the two-stage window screening decreased by 59%.

Conclusion:

The two-stage window screening is more accurate and economical than the two-stage positive screening. Our results provide an example for the use of two-stage screening and the possibility of the WCST to replace WAIS-R in large-scale screenings for ID in the future.

Introduction

The prevalence of intellectual disability (ID) is increasing globally, especially in less developed countries.Citation1 Most of the maladaptive behaviors of ID appear before the person reaches 18 years of age.Citation2 The comorbidities of ID are associated with many kinds of diseasesCitation3Citation5 and a varying number of people with ID present at police stations.Citation6,Citation7 Military conscription is mandated in Taiwan, thus an economical and accurate means of screening men who are not competent for the military is vital. In fact, among various factors related to learning and adjustment, the weighting of intelligence is most considerable. People with ID may have poor communication skills, and also lack or be deficient in self-care, social skills, home living, community use, self-direction, health and safety, functional academics, leisure, and work.Citation8 Intelligence as a concept has many different definitions and measuring tools.Citation9 According to Wechsler’s definition,Citation10 intelligence is interpreted as “the aggregate or global capacity of the individual to act purposefully, to think rationally and to deal effectively with his environment”. The Wechsler Adult Intelligence Scale-Revised (WAIS-R) is a widely used intelligence test used to measure the intelligence of those between 16 and 74 years of age. Although the reliability and validity of the WAIS-R are quite satisfactory,Citation9 the test is not economical for mass screening in military settings to identify intellectually impaired recruits.

On the other hand, the Wisconsin Card Sorting Test (WCST) is a neuropsychological test that is used generally as a measure of abstract reasoning ability and of the ability to utilize environmental feedback to shift cognitive strategies.Citation11 The computerized WCST has been developed and is administered automatically, including stimuli-presenting and result-scoring, and the examiner only needs to monitor the assessment process. Generally, the computerized WCST is less expensive and requires less time than the WAIS-R and is much easier to administer and interpret.

A two-stage screening process entails using a more inexpensive, less invasive test at the first stage (eg, computerized WCST), and a more expensive or more invasive, but more sensitive and specific test at the second stage (eg, WAIS-R).Citation12 The two-stage screening can be subtyped into two-stage positive and two-stage negative screening. Two-stage positive screening implies that only those who screened positive on the first test will be scheduled for a later assessment, which is more expensive or more invasive. Participants who screen positive in both stages will be referred for further intervention programs (). Two-stage negative screening picks out more participants and is used for more life-threatening disorders (eg, acquired immunodeficiency syndrome). Participants who screen positive in the first stage will be directly referred for treatment. Additionally, participants who screen negative in the first stage need to be screened in the second stage. Those who screen positive in the second stage will also be referred for further treatment or exempted from military service in this study ().Citation13

Figure 1A Two-stage positive screening.

Note: Black box indicates the group of participants referred for further intervention.
Figure 1A Two-stage positive screening.

Figure 1B Two-stage negative screening.

Note: Black boxes indicate groups of participants referred for further intervention.
Figure 1B Two-stage negative screening.

We combined the concepts of two-stage positive and two-stage negative screening in this study to create a new screening approach called the “two-stage window screening” (). In this approach, we selected two cutoff points from a receiver operating characteristic (ROC) curve with maximized area under the curve (AUC), one in the higher and one in the lower area. We assumed those scoring higher than the higher cutoff point as having normal intelligence, and those scoring lower than the lower cutoff point as intellectually disabled; the only group that needs to be further tested is the group between the two cutoff points. This group has a higher probability of being misclassified. Using this method of screening would decrease the number of people who need to test in the second stage. Previous investigations have shown that when the prevalence of a disease was enhanced, the positive predictive value (PPV; the proportion of true positives among individuals who tested positive) would be increased. When a disease has a higher prevalence, the screening instrument will show a better accuracy in predicting the disease.Citation12 Other than increasing the accuracy of prediction, decreasing the amount of people needed to test and will further decrease the cost of testing.

Figure 2 Two-stage window screening.

Note: Participants whose scores were in the thickened area have to be tested in the second stage.
Figure 2 Two-stage window screening.

In Taiwan, approximately 150,000 young men are conscripted into the military every year, for a two-year enlistment. In the past, the Symptom Checklist (SCL-90-R)Citation14,Citation15 was used as a screening instrument for all military conscripts, but it was costly and inefficient. At present, only those who have shown maladjusted behavior are brought to the psychiatrist, and further assessed by psychologists using the WAIS-R. The purpose of this study was to investigate whether the application of two-stage window screening in large-scale screening of military conscripts for ID, using the WCST in the first stage and the WAIS-R in the second stage, would be more accurate and cost-efficient than the two-stage positive screening method.

Designs and methods

Participants

The protocol of this study was approved by the committee of research ethics (the Institutional Review Board) at Kaohsiung Armed Forces General Hospital in southern Taiwan. We collected 99 conscripted soldiers from the largest branch of the military in southern Taiwan, whose educational levels were senior high school level or lower, to be the participants. In Taiwan, only males were conscripted, thus all participants were male between the ages of 20 and 24 (M = 20.24, SD = 0.624). After detailed explanation regarding the purpose of the study, the participants gave their consent. Subsequently, all participants completed the WAIS-R and the computerized WCST assessment. No participant had any clinical evidence of psychiatric disorders according to the Chinese versions of the Structured Clinical Interview for DSM-III-R (SCID).Citation16

Instruments

Wechsler Adult Intelligence Scale-Revised

WAIS-R has been widely used in Taiwan in recent years to measure the intelligence of those from 16 to 74 years of age. It comprises 11 subtests: six subtests on verbal intelligence and five subtests on performance intelligence.Citation17 This instrument was used as the gold standard in this study. The WAIS-R generally takes 60 to 90 minutes to administer.Citation17

Wisconsin Card Sorting Test

The computerized WCST was adopted for this study, this computerized version was developed based on the standardized criteria of Heaton and colleaguesCitation11 by Tien and colleagues in 1996 using Turbo Basic 1.1.Citation18 The assessment illustrations comprise trends in speech sound and written Chinese characters. The computerized system can be adjusted by the test administrator based on the conditions of the testee. The participants were given a pack of 128 response cards and told to match each response card to one of the stimulus cards. After matching 10 cards using the first sorting category (color), the participants shifted to the second category (form), and then to the third (number). The procedure was repeated twice, until the participants completed six categories, or until all 128 response cards were used. The WCST index consists of the following: percent preservative errors (PPE), trials to complete the first category, failure to maintain the set (FMS), learning to learn, categories achieved, and conceptual level responses (CLR). Higher scores in the first three indices indicate a worse performance. In contrast, higher scores in last three indices indicate a better performance. Generally, the manual WCST takes 20 to 30 minutes to administer,Citation11 administered by those with psychological assessment training, the computerized version provides a more reliable and administration, data acquisition accuracy and scoring than the manual version.Citation18

Statistical analysis

Data were analyzed with SPSS for Windows (version 15.0; SPSS Inc., Chicago, IL) and a computer program for accuracy analysis.Citation19 According to the DSM-IV definition, if the FIQ of the individual is equal to or lower than 70, the individual is defined as having ID; if the FIQ of the individual is between 71 and 84, then the individual is defined as having borderline intelligence.Citation20 Hence, this study used the results of the WAIS-R as the “gold standard” to estimate whether the testee had ID. The individual would be placed in the ID group if the FIQ was equal to or lower than 84, and in the normal group if the FIQ was greater than 84.

After using multiple logistic regression analysis to verify the indexes of the WCST that could be used to predict ID, we used ROC analysis to determine the optimum cutoff point of the index. ROC analysis is a nonparametric approach that was first developed in psychophysics.Citation21 This method of analysis described a function that summarizes all possible performances of a case faced with the task of detecting a signal from noise under a curve.Citation22 The principle of ROC analysis has been applied to screening psychiatric illness.Citation23Citation25 The area under the ROC curve (AUC) could be regarded as the probability of correct prediction.Citation26,Citation27 We first selected the largest AUC value to be the optimum cutoff point of the predictive index, and the point was set as the basic (or fixed) point. We tried to compute the AUC of the basic and other points of the predictive index. The largest AUC value would be the optimum cutoff point of the predictive index.

Other than that, we also compared the sensitivity (the ability of a test to correctly identify those who have the characteristic of interest), specificity (the ability of a test to correctly identify those who do not have the characteristic of interest), PPV (the proportion of true positives among those who test positive), negative predictive value (NPV; proportion of true negatives among those who test negative),Citation28 AUC (probability of correct prediction),Citation26,Citation27 and assessment expense of the two-stage positive screening and two-stage window screening (each WAIS-R assessment costs US$21.50).

Results

The distribution of the WAIS-R full IQ and WCST scores and the correlation among each sub-test is presented in . As assessed with the WAIS-R, 38 (38.4%) out of 99 participants had a FIQ lower than or equal to 84, which means they were intellectually disabled. Further, multiple logistic regression analysis results showed that the CLR index in the WCST is a valid predictor for ID (β = −0.047, p = 0.000) (), indicating that a lower CLR score would result in a higher possibility of ID.

Table 1 The distribution of the WAIS-R and WCST scores, and the correlation among each subtest

Table 2 Results of multiple logistic regression analysis of the indexes of the WCST and ID

The optimum cutoff point of the CLR index obtained from the ROC curve was 66. When a CLR index of 66 as the cutoff point was used in two-stage positive screening, 61 were determined to be in the intellectual disability group and 38 were in the normal intelligence group (). This group of 61 persons with positive result for ID will be screened in the second stage, and further tested with the WAIS-R. When this method was used, the results indicated that the AUC was 0.71, sensitivity was 0.87, specificity was 0.54, PPV was 0.54, and NPV was 0.87.

Table 3 Results of using one cut-off and two cut-off points in each group showing intellectual status (Unit: person)

The two optimum cutoff points of the CLR index obtained from the ROC curve were 66 and 49 (AUC = 0.74). When the two-stage window screening method was used, with the CLR index and two cutoff points of 66 and 49, it was found that 25 participants had CLR scores between the two cutoff points (). Eleven of the 25 participants, whose FIQ with the WAIS-R was equal to or lower than 84, could have been true-positives in the two-stage window screening, and the others, whose FIQ WAIS-R results were higher than 84, could have been true-negatives. After re-setting the classification, the AUC was 0.82, sensitivity was 0.87, specificity was 0.77, PPV was 0.70, and NPV was 0.90.

When the two-stage positive screening was compared with the novel screening method, AUC, sensitivity, specificity, PPV, and NPV were found to be increased in the novel screening method. Moreover, the cost of the second-stage screening using the WAIS-R assessment decreased from US$1311.50 to US$537.50 ().

Table 4 Comparison of results using two-stage positive screening and two-stage window screening

Discussion

Generally, those conscripts who have moderate, severe, and profound ID can be determined before they enlist in the military. Mild and borderline intellectual disability is not easily perceived during the health examination; so those conscripts can usually enter the military without a hitch. Previous investigations have shown that when the prevalence of a disease is enhanced, the PPV value is increased. When a disease has a higher prevalence, the screening instrument will show a better accuracy in predicting the disease.Citation12 Hence, the present study selected those participants whose educational levels were on par or lower than the senior high school level, to increase the prevalence of ID. In this study, we found that the AUC and PPV of two-stage positive screening were larger than those of two-stage window screening. As a result, if we could increase the cutoff points of the CLR, it would contribute to a better accuracy in classification or a higher probability of prediction. Moreover, two-stage window screening would be more accurate and economical than two-stage positive screening for predicting ID. Two-stage window screening can save approximately 59% money than the two-stage positive screening method (1311.5/537.5 US$ = 0.41). Since the participants were sampled from the largest branch of the military in southern Taiwan, the results of this study can be generalized to military sites throughout Taiwan.

One limitation of the study is that the screening procedure ignores the cultural, ethnic, socioeconomic, and adjustment factors of these conscripts,Citation8,Citation29 which have been proposed by the American Association on Mental Retardation (AAMR) as important factors in the definition of ID. This proposal was made because ID can disappear as the person enters society, if the initial diagnosis did not adequately consider adaptive function. Another limitation is the small sample size of the study, since only 99 conscripts were recruited, the ROC-curve analysis can only describe a tendency, thus a larger scale research is needed to verify whether WCST can be used to screen the intelligence of military conscripts using two-stage window screening in the future.

The algorithm that we propose for further studies is described as follows: using nomination techniques, cadres can determine problem individuals through initial observation (ie, copying skills and adjustment abilities), and these individuals can be sent to the psychiatrist afterward. During the interview process, they will be required to take the WCST. The prevalence of ID will be higher than the random sampling in the initial observation of the military cadres; moreover, the PPV value of the WCST to predict ID will also increase. If the WCST is verified to be accurate and valid to predict ID, it can be used as a first screening in the testing process for ID. The WCST is easier to operate and time saving in estimating ID, and can be used for large-scale screening for ID in the military in future.

Conclusion

Using the two-stage window screening, the AUC and the PPV increased. Moreover, two-stage window screening is more accurate and economical than two-stage positive screening. Our results provide a foundation for the use of two-stage screening and the WCST to replace the WAIS-R in large-scale screenings for intellectual disability.

Acknowledgements

This study was supported by a grant (numbers 9312) from the Medical Affairs Bureau, Ministry of National Defense, Taiwan. The authors would also like to acknowledge all researchers who assisted in this study.

References

  • World Health OrganizationHealthy aging: adults with intellectual disabilityCanberraAustralian Institute of Health and Welfare2000
  • WenXThe definition and prevalence of intellectual disabilityCanberraAustralian Institute of Health and Welfar1997
  • HoggJNorthfieldJTurnbullJCancer and people with learning disabilities: The evidence from published studies and experiences from cancer servicesKidderminsterBild Publications2001
  • SplunderJStilmaJSBernsenRMDEvenhuisHMPrevalence of visual impairment in adults with intellectual disabilities in the Netherlands: cross-sectional studyEye2006201004101016151486
  • SullivanSGHussainRThrelfallTBittlesAHThe incidence of cancer in people with intellectual disabilitiesCancer Causes Control20041512011205
  • GudjonssonGHClareICRutterSPearseJPersons at risk during interviews in police custody: The identification of vulnerabilities. The Royal Commission on Criminal Justice. Research Study 12LondonHMSO1993
  • MurphyGHarnettHHollandAJA survey of intellectual disabilities among men on remand in prisonMent Handicap Res199588191
  • LuckassonRCoulterDLPollowayEAReissSSchalockRLSnellMESpitalnickDMStarkJAMental retardation: Definition, classification, and systems of supports9th edWashington, DCAmerican Association on Mental Retardation1992
  • GregoryRJPsychological Testing: History, Principles, and Applications2nd edBoston, MAAllyn and Bacon1992
  • WechslerDThe measurement of adult intelligenceBaltimore, MDWilliam and Wikins1939
  • HeatonRKCheluneGJTalleyJLKayGGCurtissGAdministration and scoringHeatonRKWisconsin Card Sorting Test Manual: revised and expandedOdessaPsychological Assessment Resources Inc1993520
  • GordisLEpidemiologyPhiladelphia, PAW. B. Saunders1996
  • GlascoeFPFosterEMWolraichMLAn economic analysis of developmental detection methodsPediatrics1997998308379164778
  • DerogatisLRLipmanRSCoviLSCL-90: an outpatient psychiatric rating scale-preliminary reportPsychopharmacol Bull1973913284682398
  • DerogatisLRMelisaratosNThe Brief Symptom Inventory: an introductory reportPsychol Med1983135956056622612
  • ChenYCKuCSInstruction Manual for the Structured Clinical Interview for DSM-III-R (SCID)Kaohsiung, TaiwanKaohsiung Medical College1993
  • WechslerDWechsler Adult Intelligence Scale-RevisedSan Antonio, TaxesThe Psychological Corporation1981
  • TienAYSpevackTVJonesDWPearlsonGDSchlaepferTEStraussMEComputerized Wisconsin card sorting test: comparison with manual administrationKaohsiung J Med Sci1996124794858774117
  • SwetsJAPickettRMEvaluation of Diagnostic Systems: Method from Signal Detection TheoryNew YorkAcademic Press1982
  • American Psychiatric AssociationDiagnostic and Statistical Manual of Mental Disorder4th edWashington, DCAmerican Psychiatric Association1994
  • SwetsJASignal Detection and Recognition by Human ObserversNew York, NYJohn Wiley and Sons1964
  • EganJPSignal Detection Theory and ROC AnalysisNew York, NYAcademic Press1975
  • MariJJWilliamsPA comparison of the validity of two psychiatric screening questionnaires (CHQ-12 and SRQ-12) in Brazil, using relative operating characteristics (ROC) analysisPsychol Med1985156516594048323
  • MurphyJMBerwickDMWeinsteinMCBorusJFBudmanSHKlermanGPerformance of screening and diagnostic testsArch Gen Psychiatry1987445505553579501
  • LirioRBDonderizICPerez AbaloMCMaximum likelihood estimation of signal detection model parameters for the assessment of two-stage diagnostic strategiesInt J Biomed Comput1992311171261516999
  • GrunkemeierGLJinRReceiver operating characteristic curve analysis of clinical risk modelsAnn Thorac Surg20017232332611515859
  • SwetsJAROC analysis applied to the evaluation of medical imaging techniquesInvest Radiol197914365377
  • SzkloMNietoFJEpidemiology beyond the basicsSudbury, MAJones and Bartlett Publishers2004
  • MasonJMurphyGPeople with an intellectual disability in the criminal justice system: Developing an assessment tool for measuring prevalenceBr J Clin Psychol20024131532012396258