Psychometric properties of the Psychopathology in Autism Checklist (PAC)

Abstract

Psychiatric disorders in individuals with co-occurring autism spectrum disorders (ASD) and intellectual disability (ID) are common, but diagnosis presents many challenges. The Psychopathology in Autism checklist (PAC) is among the very few instruments specifically developed for this group of individuals. The psychometric properties of the PAC (i.e. criterion validity, specificity, sensitivity and predictive values) were explored by comparing scores with assessments on the Aberrant Behavior Checklist (ABC) and examining how well assessment by the PAC at referral predicts final clinical diagnoses. Results indicated a significant correlation with the ABC, further supporting the validity of the PAC. Sensitivity and Specificity for specific diagnoses were variable, although positive predictive value for “any diagnosis’ was relatively high. The study confirms the potential value of the PAC as a screening checklist but highlights the need for clinical diagnosis to be based on a multimodal, multidisciplinary assessment.

Keywords:

• intellectual disability
• autism spectrum disorders
• psychiatric disorders
• assessment
• psychometric properties

Introduction

Individuals with co-occurring autism spectrum disorders (ASD) and intellectual disability (ID) are reported to have higher prevalence of psychiatric disorder, both when compared to non-autistic individuals with ID and to the general population (Bakken et al. 2010, 2016, Dunn et al. 2020, Helverschou et al. 2011a, Hove and Havik 2010, Hollocks et al. 2019, Lai et al. 2019, Rosen et al. 2018). It is therefore recommended that mental health assessment should constitute an integral aspect of clinical care for these individuals, with regular screening, evaluation, and treatment for co-occurring psychiatric disorders being part of standardized, ongoing support (Havdahl and Bishop 2019). However, identification of psychiatric disorders in individuals with ASD and ID is challenging. Thus, superficial similarities and apparent symptom overlap between ASD and symptoms of psychiatric disorder can lead to difficulties in distinguishing between ASD and psychiatric disorder, as well as in differentiating between specific psychiatric conditions in individuals with ASD (Bakken et al. 2016, Helverschou et al. 2008, 2011a, Long et al. 2000, McDougle et al. 2000, Reaven and Hepburn 2003, Rosen et al. 2018, Volkmar and Cohen 1991, Wing 1996).

Assessments of psychiatric disorder typically rely on descriptions of the subjective experiences of the individual being assessed (e.g. DSM-5, American Psychiatric Association 2013, ICD-10 and ICD-11; World Health Organization 1992, 1993, 2018), and require comprehensive information about symptoms and difficulties, and the duration and frequency of problems. This information is typically obtained via interview and/or self-rating checklists (Othmer et al. 2005). However, many individuals with ASD, including those with good verbal abilities, have difficulties describing emotional experiences and inner states (Helverschou et al. 2011a, Underwood et al. 2015), and hence in reporting information that is needed to identify psychiatric disorders. In autistic individuals with co-occurring ID, the diagnostic process is further complicated by their difficulties in comprehension and expression, and in those with very limited verbal skills, identification of psychiatric disorder frequently relies on clinicians’ own interpretations of behavioural expressions of symptoms. (cf. Royal College of Psychiatrists 2001, Fletcher et al. 2016).

The complexity of diagnosing psychiatric disorders in individuals with ASD is illustrated by the large variation in reported prevalence rates, ranging from 16% to 73% in various studies (Lai et al. 2014). Inconsistencies in prevalence estimates arise from a number of underlying factors, including differences in diagnostic criteria and measures. Thus, in defining the presence of co-occurring psychiatric disorder in individuals with autism, Hutton et al. (2008) explicitly required the onset of new symptoms that represented a qualitative change in individuals’ behaviour and/or level of functioning, not just a worsening of already existing ASD features. These criteria contrast with those of other researchers who include an increase in typical ASD symptoms (i.e. more intense ruminations and repetitive and ritualistic behaviour) as possible indicators of psychiatric disorder (Ghaziuddin 2005, Helverschou et al. 2011a).

Studies reporting higher rates of psychiatric disorders (≥ 70%) have tended to use instruments developed for the general population (Simonoff et al. 2008), or modified versions of these (e.g. Autism Co-Morbidity Interview-Present and Lifetime Version [ACI-PL]; Leyfer et al. 2006). However, the use of standard measures increases the risk that signs and behaviours associated with ASD may be misinterpreted as psychiatric symptoms (Bakken et al. 2016, Bakken and Høidal 2014). Similarly, although there now exists a number of instruments developed for the identification of mental health problems in individuals with ID (see Flynn et al. 2017) there are limited data on the reliability and validity of these measures for individuals who also have ASD (Helverschou et al. 2020b, Leyfer et al. 2006, Underwood et al. 2011). For example, two frequently used psychiatric checklists for people with ID, the DASH II (Matson et al. 1991) and the PAS-ADD (Moss et al. 1998), include autism as one of the disorders to be identified; it is also unclear whether they adequately capture the sometimes atypical expression of psychiatric symptoms observed in ASD. The Aberrant Behavior Checklist (ABC; Aman and Singh 2017, Kaat et al. 2014, Brinkley et al. 2007), although not specifically developed for individuals with ID and ASD is one of the few instruments that does appear to show good psychometric properties when used with this group. The validity and reliability of other potentially useful instruments (e.g. the Developmental Behaviour Checklist [DBC], Gray et al. 2018) still have to be determined, and there is a clear need for measures expressly designed to identify mental health problems in people with ID and ASD.

The Psychopathology in Autism Checklist

The Psychopathology in Autism Checklist – PAC (Helverschou et al. 2008, 2009) was specifically developed to distinguish between the core symptoms of ASD and symptoms of psychiatric disorder as a prerequisite for developing more valid psychiatric diagnoses in people with ASD. The PAC includes items representing symptom descriptions for four groups of psychiatric disorders (psychosis, depression, anxiety, obsessive-compulsive disorder; OCD) that have been demonstrated not to overlap with the core characteristics of ASD (Helverschou et al. 2008). In addition, the PAC includes a general adjustment problem (GAP) subscale, with non-specific items regarded as general indicators of impaired functioning or mental health problems. Items in the GAP subscale, such as sleeping problems, general passivity, challenging behaviour and general distress, are typically associated with psychiatric disorder or reactions to difficult circumstances in individuals with ASD and ID (Ghaziuddin 2005, Lainhart 1999, Reiss 1988, Stavrakiki 1999) and are included in the PAC to increase the probability of identifying all individuals in need of further psychiatric assessment.

The PAC is intended as a screening checklist, designed to identify adults with ASD and ID in need of psychiatric services. It is completed by parents/carers and comprises 42 items distributed across 5 subscales; psychosis (10 items), depression, (7 items), anxiety disorders (6 items), obsessive-compulsive disorder (OCD) (7 items) and general adjustment problems (12 items). Each item is rated on a four-point scale (1 = no problem; 2  = minor problem; 3=moderate problem; 4 = severe problem). Scoring involves a two-step procedure: First, individuals with severe general adjustment problems are identified (i.e. average GAP score above cut-off). Thereafter, those individuals who obtain an average score above cut-off for any of the psychiatric subscales are classified as having a possible psychiatric disorder, requiring more comprehensive psychiatric examination.

The initial validation study (Helverschou et al. 2009) compared participants with ASD and ID who had previously been diagnosed with a co-occurring psychiatric disorder with those who had no psychiatric diagnosis. The PAC was found to adequately discriminate between these groups. There was also some evidence that it could identify individuals with different types of psychiatric disorders, in particular psychosis and OCD. The psychometric properties, i.e. internal consistency and inter-rater agreement were found to be acceptable. Cut-off values for subscales were established on the basis of this validation study (Helverschou et al. 2009). However, comparison between the PAC anxiety scale and more comprehensive clinical assessment indicated that the anxiety items in the PAC were insufficient to correctly identify all individuals with anxiety problems and that identification of anxiety was improved if general adjustment problems were included (i.e. a GAP score above cut-off; Helverschou and Martinsen 2011b).

In a previous study investigating the PAC, using a population-based sample of individuals with autism and ID, psychiatric disorders and severe adjustments problems were identified in more than 50% of the ASD and ID group, compared to approximately 20% in a representative sample of ID “only” (Bakken et al. 2010). Thus, the PAC appears to identify higher rates of psychiatric disorders than studies using stricter criteria (i.e. Hutton et al. 2008), but lower rates than studies using instruments not specifically developed for ASD (Simonoff et al. 2008). For more details, see Helverschou et al. (2020b).

In the present paper, the psychometric properties i.e. criterion validity, specificity, sensitivity, and predictive values of the PAC were explored. To examine criterion validity, PAC scores were compared with a well validated checklist for behaviour problems, the Aberrant Behavior Checklist (ABC; Aman and Singh 1986). The ABC was chosen because it is the most thoroughly researched checklist for behaviours that challenge, with several hundred studies investigating its psychometric properties (Aman 2012). It is currently available in more than 25 languages and is easy to administer and score. Psychometric properties vary from satisfactory to excellent (Aman et al. 1985a, 1985b, Ono 1996, Flynn et al. 2017) and its factor structure has been confirmed in numerous studies (Aman et al. 1985a, Ono 1996, Newton and Sturmey 1988). Good psychometric properties have been demonstrated across varying levels of ID (Flynn et al. 2017, Aman 2012), in children and adolescents with ASD (Kaat et al. 2014, Brinkley et al. 2007), and for the Norwegian version (Halvorsen et al. 2019).

To determine how well assessment by the PAC at referral predicts subsequent clinical diagnosis based on detailed multi-disciplinary assessment, specificity, sensitivity, and predictive values (positive and negative) and Area under the Curve (AUC) were explored. Data were collected as part of a national multi-centre clinical study (Helverschou et al. 2020a).

Method

The AUP (Autism, Intellectual disability and Psychiatric disorder) study is an on-going clinical treatment study that began in 2010 (see Helverschou et al. 2020a). It involves eight clinical centres responsible for providing specialist, hospital-level mental health services to individuals with ASD and ID across all four health regions in Norway. The centres provide in-patient and out-patient services, but most services are delivered on a peripatetic outpatient basis in community facilities. A standardised assessment protocol was designed as part of regular service delivery. Assessments include demographic data and information on medical status, behaviour problems, psychiatric symptoms and diagnoses, environmental factors, interventions provided, and evaluations by care staff and family members. Assessments are conducted at three time points: at referral to the service (T1), after 12 months (T2), and after 24 − 27 months (T3). The PAC and the ABC are completed at all three time points by local caregivers with good personal knowledge of the individuals involved.

Clinical diagnoses

Diagnosing mental health problems in individuals with autism and ID is challenging. There is no consensus on best diagnostic practice or the criteria for diagnosing mental illness in this group. There is, however, a general agreement that an accurate diagnostic assessment depends on distinguishing between symptoms that are associated with autism; those that arise from the intellectual disability, and those that indicate additional psychiatric disorder. Thus, diagnosis should be based on identification of qualitative “changes” in the individual’s long-standing features of autism, the presence of conventional psychiatric symptoms, and recognition of idiosyncratic or atypical symptoms (Helverschou et al. 2011a, Kerns et al. 2016, 2020, Underwood et al. 2015).

The eight participating centres are interdisciplinary, involving psychologists, psychiatrists, medical doctors, special educators, learning disability nurses and psychiatric nurses. To improve the quality of and accesses to mental health services for individuals with ASD and ID, the centres have, for over a decade, participated in a national professional network addressing mental health in these patients. The importance of individualized, multimodal and multi-informant approaches to the assessment of mental health problems has been emphasized throughout (Helverschou et al. 2020b, MacNeil et al. 2009). Thus, all professionals at the centres are expected to have high levels of competence and specialist expertise in diagnosing and treating this patient group. Final clinical diagnoses are usually made in interdisciplinary teams, following the completion of the standardised assessment protocol and additional assessments, such as instruments adapted to ID or instruments assessing one particular disorder, as necessary. Thus, the professionals concluding on the final psychiatric diagnosis were aware of PAC scores from all time points as well as results from additional assessments.

Participants

Patients with ASD and ID referred to one of the eight centres for psychiatric assessment or because of behaviour problems were recruited to the study. Potential participants included all patients with co-occurring ASD and ID aged 14 years or older; both new and re-referrals were accepted. There were no exclusion criteria on the basis of severity of ID or presence of other disorders such as ADHD, Tourette syndrome, epilepsy, genetic syndromes etc. Thus, participants included patients who are frequently excluded from research involving co-occurring mental health problems in ASD/ID.

The present paper includes 123 participants assessed at referral (T1), after one year (T 2) and after two years (T3); participants comprised all patients referred to the AUP study with a final clinical diagnosis at T3. However, seven participants had missing ratings on the ABC, resulting in a total of 116 participants for that analysis. Demographic information is presented in Table 1. All participants have received a clinical diagnosis of ID based on ICD-10 (WHO 1992) prior to inclusion in the current study, by psychologists or psychiatrists in hospital-level services. For the current study, ID was rated as either mild/moderate or severe/profound, based on scores on the Vineland Adaptive Behavior Scales, second edition, expanded Interview Form (Sparrow et al. 2008) and clinical background information from hospital records. The Vineland was completed by professionals from the participating centres. More detailed cognitive data were not available due to the challenges of obtaining reliable IQ results from patients with severe cognitive, behavioural or psychiatric difficulties.

Table 1. Participant demographics.

Age	16 − 58 years	(M = 28.1, SD = 9.8)
Gender
Females	39	(31.7%)
Males	84	(68.3%)
Level of ID
Mild/moderate	82	(66.7%)
Severe/profound	41	(33.3%)

All participants have received a clinical diagnosis of ASD based on ICD-10 (WHO 1992) by psychologists or psychiatrists in hospital-level services prior to inclusion in the current study. Level of ASD symptoms is measured by the Social Communication Questionnaire (SCQ) (Rutter et al. 2003). For some participants, the ADI-R (Rutter et al. 2003) and ADOS (Lord et al. 2000) had been completed as part of previous assessments. However, routine use of ADI-R/ADOS for all patients is not feasible because of the increased demands on staff time and training; the clinical validity of these instruments may be compromised by co-occurring psychiatric disorders. Most participants had Social Communication Questionnaire scores above 15 (Median SCQ = 21.7; range 6 − 35) (i.e. at/above the suggested cut-off for ASD; Rutter et al. 2003).

Procedure

The association between symptoms of mental disorder assessed by the PAC and level of problematic (“challenging”) behaviours, assessed by the ABC, was analysed at each time point. The specific mental health problems identified by the PAC at T1 were compared with the final clinical mental health diagnoses at T3. The numbers of individuals scoring positively for the presence of any psychiatric disorder on the PAC at referral were also compared with those given any clinical psychiatric diagnosis at T3. However, in the final clinical diagnoses, two additional diagnostic categories (post-traumatic stress disorder (PTSD) and bipolar disorder) were identified that are not included in the PAC. In order to account for these cases, all participants with a diagnosis of PTSD were rated as having an anxiety disorder and bipolar disorder was rated as depression; in addition, participants who received co-occurring anxiety and depression diagnoses were rated as having both disorders.

Ethical issues

Informed consent was obtained from all the patients and/or their guardians. Data were anonymised and processed without any directly recognisable information. The project was approved by the Privacy Data Protection Supervisor (Local IRB, Institutional Review Board) at Oslo University Hospital, Oslo, Norway. Approval # 2010/19579.

Statistical analysis

Data were analysed using the Statistical Package for the Social Sciences (SPSS, version 25). Spearman’s rho was used to compare correlations between total scores on the ABC and the PAC. Because of the number of comparisons conducted, alpha level was set at p<.01.

For the computations of sensitivity, specificity and predictive values, true and false positive were identified as well as true and false negatives. To identify the discriminative properties of PAC at T1, we employed ROC analysis, reporting AUC-values. An AUC of 0.5 was considered unacceptable; 0.6 as poor; 0.7 to 0.8 as acceptable; >0.8 to 0.9 excellent (Hosmer and Lemeshow 2000). All analyses were performed for the total sample and repeated for the sample separated by level of ID: mild/moderate and severe/profound.

Results

Correlations between PAC and ABC total scores are presented in Table 2. (See supplementary material for PAC and ABC scores at each time point.) Correlations between the two measures at each time point (T1, T2, and T3) were all significant and medium to large (r = .47- to .59; p < .01). Correlations between the PAC total score and the ABC total score at different times were weak to moderate, although generally significant (p < .01).

Table 2. Spearman’s rho correlations between total scores for the aberrant behavior checklist and the Psychopathology in Autism Checklist (n = 116).

	ABC T1	ABC T2	ABC T3
PAC T1	.491 (p < .001)**	.282 (p = .002)**	.283 (p = .002)**
PAC T2	.200 (p = .031)*	.465 (p = < .001)**	.304 (p = .001)**
PAC T3	.319 (p < .001)**	.255 (p = .006)**	.589 (p < .001)**

Notes.

** p < .01 (two-tailed). A Benjamini-Hochberg correction (Benjamini & Hochberg, 1995) using a false discovery rate of .05 was applied to control for multiple comparisons. All p-values were below the Benjamini-Hochberg critical values.

Sensitivity and specificity data for the total sample are presented in Table 3. Sensitivity (proportion of cases where the PAC assessment predicted final clinical diagnosis) varied between .79 − .38. Sensitivity was highest for anxiety followed by psychosis and depression; sensitivity for OCD was low. Specificity (cases correctly identified by PAC as not having the specific disorder) ranged from .85 to.51. Sensitivity for any psychiatric disorder (i.e. scoring above cut-off for any disorder on PAC) was .72; specificity for any psychiatric disorder was .47. The frequency of disorders ranged from n = 8 (OCD) to n = 26 (anxiety), and n = 87 for any mental disorder.

Table 3. Discriminative properties of the Psychopathology in Autism Checklist (PAC) in the total sample (n = 123).

	Sensitivity	Specificity	PPV	NPV	AUC
Psychoses	.65	.71	.26	.93	.716
Depression	.65	.51	.26	.84	.647
Anxiety	.79	.62	.38	.91	.726
OCD	.38	.85	.15	.95	.726
Any Psychiatric disorder	.72	.47	.77	.41	.617

OCD = Obsessive compulsive Disorder.

PPV = positive predictive value, NPV = negative predictive value, AUC = area under curve.

Positive predictive values (PPV) reflect the probability of scoring above cut-off on a PAC sub-scale and being clinically assessed to have the respective disorder. PPV was high (.77) for “any psychiatric disorder”, but lower for specific conditions. Negative predictive value (NPV) reflects the probability of scoring below the cut-off on a PAC sub-scale while being clinically assessed as not having the disorder in question. NPVs were high (>80) for specific conditions but low for any disorder. AUC values were acceptable (>.70) for psychosis, anxiety and OCD but poor for depression and any disorder.

The sample was then divided into two groups based on ID level. Among participants with mild/moderate ID (see Table 4), sensitivity varied between .69 and .20, with the lowest sensitivity for OCD. Specificity varied between .86 and .48, with the highest specificity for OCD, followed by psychosis, anxiety, depression, and any disorder. PPV varied between .79 and .08, and the NPV between .94 and .34. AUC values were acceptable for psychosis (>.70) but poor for the other conditions.

Table 4. Discriminative properties of the PAC among participants with mild/moderate level of ID (n = 82).

	Sensitivity	Specificity	PPV	NPV	AUC
Psychoses	.60	.76	.36	.89	.726
Depression	.68	.51	.295	.84	.698
Anxiety	.63	.59	.27	.87	.668
OCD	.20	.86	.08	.94	.639
Any psychiatric disorder	.69	.48	.79	.34	.623

N = 82, OCD = Obsessive compulsive Disorder.

PPV = positive predictive value, NPV = negative predictive value, AUC = area under curve.

Sensitivity for participants with severe/profound ID varied between .79 and .38 (see Table 5), again with the lowest sensitivity for OCD. Specificity varied between .85 and .47 and was high for OCD but lowest for depression and any disorder. PPV varied between .72 and .12, and NPV between 1.00 and .58. AUC values were excellent (>.80) for anxiety, psychosis and OCD but poor for any disorder and unacceptable for depression.

Table 5. Discriminative properties of the PAC among participants with profound/severe level of ID (n = 41).

	Sensitivity	Specificity	PPV	NPV	AUC
Psychoses	.65	.62	.12	1.0	.865
Depression	.65	.5	.19	.85	.523
Anxiety	.79	.69	.57	1.0	.816
OCD	.38	.84	.25	.97	.877
Any Psychiatric disorder	.72	.47	.72	.58	.635

N = 41, OCD = obsessive compulsive disorder.

PPV = positive predictive value, NPV = negative predictive value, AUC = area under curve.

Discussion

The psychometric properties of the PAC were explored in a clinical sample of individuals with ASD and ID referred for psychiatric assessment or behaviour problems. The present study represents the first comparison of the PAC with another checklist and the significant correlations with the ABC add to previous data on the validity of the PAC (Helverschou et al. 2008, 2009) as well as confirming the relationship between behaviour problems and psychiatric disorders in individuals with ASD and ID identified in previous research (e.g. Emerson et al. 1999, Moss et al. 2000, Minshew 2006, Myers and Winters 2002, Myrbakk and von Tetzchner 2008, Painter et al. 2018). The mechanisms involved in these associations remain unclear, but it has been suggested that difficulties in emotion regulation may be an important moderator in the relationship between behavioural disturbance and mental health. (Cooper 2016).

Sensitivity and specificity of the PAC were calculated by comparing screening by the PAC at referral with final clinical diagnoses after two years of treatment. For most participants, diagnostic ascertainment was based on additional assessments, clinical experience and multi-professional collaboration. Sensitivity and specificity levels varied between acceptable and low. AUC values were acceptable (i.e. >.70) for anxiety, psychoses and OCD but were poorer for depression and any psychiatric disorder. In addition, analyses according to level of ID, indicated somewhat better psychometric properties for the PAC in individuals with severe or profound levels of ID compared to those with mild/moderate ID. Thus AUC values in the former group were excellent for anxiety, psychosis and OCD albeit poor- unacceptable for any disorder and depression. Since assessment of psychiatric disorders is particularly challenging among individuals with intellectual disabilities (Painter et al. 2018) the results of the present study are encouraging. The high positive predictive value for any diagnosis on the PAC also suggests that scoring above cut-off for any disorder on the PAC indicates the need for a much more detailed, clinical diagnostic assessment.

The low sensitivity rates for OCD in the present study may be related to the low prevalence of OCD (n = 7) identified in the present sample. This finding contrasts with the original PAC validation study (Helverschou et al. 2009) where OCD was most clearly differentiated from other psychiatric disorder. However, that study was based on a small sample of patients with previously diagnosed psychiatric disorders based on standard diagnostic criteria. The present data highlight the particular challenges in diagnosing OCD in individuals with ASD and ID in regular clinical services (Helverschou et al. 2011a, Bedford et al. 2020, Santore et al. 2020). Thus, the ritualistic and repetitive behaviours that are core characteristics of ASD may be easily misinterpreted as symptoms of OCD and vice versa (Ghaziuddin 2005, Scahill et al. 2006). However, ASD-related repetitive behaviours do not seem to occur against the person’s will, in contrast with the uncontrollable and unpleasant compulsions typically associated with OCD (Scahill et al. 2006) and the two conditions have been found to be distinct disorders (Bedford et al. 2020). It is also important to be aware that proxy and self-report descriptions of OCD may be very different in individuals with ASD (Santore et al. 2020). In the current study, only observer data were collected, and, as Santore et al. (2020) suggest, it may be necessary to incorporate both the subjective experiences of individuals as well as the perspective of outside observers in order correctly to interpret and differentiate these symptoms.

Clinical implications

Ideally, checklists should correctly identify all individuals with and without the disorder (s) in question. However, most checklists fall well short of this ideal (Lalkhen and McCluskey 2008) and although it is desirable to have a test that is both highly sensitive and specific this is often not possible. High sensitivity checklists, by definition, are good at identifying actual cases of the disorder but they frequently come with a fairly high rate of false positives (Lalkhen and McCluskey 2008). The PAC, like many other mental health checklists, is not dichotomous, but uses a continuous scale, with a specified threshold or cut-off scores indicating a positive finding. Thus, the trade-off between sensitivity and specificity is dependent on decisions about the cut-off score to be employed (McDonald and Calhoun 2010). The cut-off values of the PAC were chosen to reduce rates of false positives and were derived from the validity study of Helverschou et al. (2009). That study was based on a cohort of individuals with ASD and ID and previously identified psychiatric illness who were selected on the basis that they met strict criteria for the psychiatric disorders diagnosed. For participants such as those in the present study, who comprised a very heterogenous group of individuals with ASD and ID referred for psychiatric assessment, cut-off values may need to be lowered in order better to identify individuals requiring a more focussed psychiatric examination (Helverschou et al. 2020a).

Nevertheless, in a diagnostically challenging population, such as this, where care providers may have a low threshold for referring people to mental health assessment, the relatively high positive predictive value of the PAC for “any psychiatric disorder” is a clear advantage. Although a positive PAC screen may lack precision concerning specific diagnoses, our results indicate that the use of this short and easy-to-use checklist can help to ensure that individuals at risk of psychiatric disorders are referred to appropriate services for full diagnostic assessment. This may be particularly helpful in community settings, among general practitioners, and in ASD/ID-specific services when considering referral to specialist mental health services. Furthermore, the PAC has demonstrated its value both for assessment and treatment monitoring in several case studies involving individuals with ASD and ID, including cases with a high degree of complexity (Bakken et al. 2014, Kildahl et al. 2017, 2019a, 2019b, 2020a, 2020b, Rysstad et al. 2020).

At present, there is no consensus on best diagnostic practice or the criteria for diagnosing mental illness in persons with ASD and ID. Confounds between ASD and mental illness may cause diagnostic overshadowing in two ways: mental illness may be overlooked when symptoms are attributed to impairments associated with ASD; alternatively ASD may be overlooked if psychiatric symptoms overshadow the characteristics of ASD (Cholemkery et al. 2014, Geurts and Jansen 2012, Helverschou et al. 2011a). The results of the present study highlight the need to distinguish between the symptoms associated with autism and those that indicate additional psychiatric conditions. Thus, diagnostic assessment should focus on qualitative changes in the individual’s long-standing features of autism, and, as well as identifying “typical” psychiatric symptoms, should be alert to an unusual or idiosyncratic presentation of these symptoms. Our findings also concur with previous studies indicating that a reliable diagnosis of psychiatric disorder in individuals with ASD and ID should not rely on questionnaires or checklists alone but requires a multimodal diagnostic process, combining clinical expertise and a range of different assessment methods (Helverschou et al. 2020b, MacNeil et al. 2009). It is important, too, to work collaboratively with the family and others who know the person well, in order to obtain information on mental health and well-being, changes to typical patterns of behaviour and mood, and to provide information on idiosyncratic or atypical symptoms (Bakken and Høidal 2014, Helverschou et al. 2011a, MacNeil et al. 2009, Kildahl et al. 2017, 2020a, 2020b, Rysstad et al. 2020). Thus, for diagnosis in clinical services, more detailed and in depth clinical interviews and observations are recommended (Hollocks et al. 2018, Oliver et al. 2020).

Limitations

A limitation of the present study is that, because assessments were performed in regular clinical services, final psychiatric diagnoses were made by physicians and clinical psychologists who were not blind to the results of the PAC assessment at the start of treatment. They were aware of PAC scores from all time points as well as results from additional assessments when concluding on the final psychiatric diagnosis. The validity of the final clinical diagnoses is of vital importance, and in future research it would be preferable to have independent raters, but this was not possible in the present study. The risk of such bias may be particularly high in participants with severe/profound ID because so few other instruments are available for this group. Nevertheless, undertaking the study in “real-life” conditions can also be considered a strength, and brings the PAC closer to real world use.

Further research

There is a general agreement that it is necessary to use instruments adapted to ASD in order to identify psychiatric disorders among individuals with ASD and ID, but few such instruments are available (Underwood et al. 2011). In view of the positive results of the current study, there is a need to explore the factor structure of the PAC; to compare assessments with the PAC with other instruments for individuals with ID, and ideally also with instruments adapted for ASD and ID, such as the Developmental Behaviour Checklist (DBC; Gray et al. 2018). It will also be important to explore the psychometric properties of the PAC in other ASD populations, such as children and individuals without ID.

Conclusion

The significant relationship between the PAC and the ABC, and the positive predictive value found for “any psychiatric disorder” on the PAC provide further support for the clinical validity of this instrument. Sensitivity and specificity levels were, however, variable and highlight the fact that the PAC is only to be used as a screening checklist, and that a final diagnostic conclusion must be based on a multimodal assessment approach. Overall, the PAC seems to perform better in identifying anxiety, depression, psychoses, or any psychiatric disorder, but is poorer in identifying OCD. Separate analysis according to level of ID indicates that the PAC may work better for participants with severe and profound level of ID than those with moderate or mild ID. More research is needed in order to improve the assessment of psychiatric disorders among individuals with ASD and ID. In the meantime, the PAC is one of few checklists specifically developed for this population, and despite its limitations, the present data indicate its potential for identifying individuals in need of further psychiatric assessment.

Acknowledgements

The authors thank all participants in the AUP multicentre study, their care staff and family members, as well as centres and clinicians collecting data for the study. We also thank Patricia Howlin for her helpful and supportive input during the writing and revision of this manuscript.

Disclosure statement

No potential conflict of interest was reported by the authors.