Standardization and cross-cultural comparisons of the Swedish Conners 3^® rating scales

Abstract

Purpose: The Conners Rating Scales are widely used in research and clinical practice for measuring attention deficit/hyperactivity disorder (ADHD) and associated problem behaviors, but country-specific norms are seldom collected. The current study presents the standardization of the Swedish Conners 3^® Rating Scales. In addition, we compared the Swedish norms to those collected in the U.S. and Germany.

Material and methods: The study included altogether 3496 ratings of children and adolescents aged 6–18 years from population-based samples.

Results: The scores obtained for the Swedish Conners 3^® showed satisfactory to excellent internal consistency for most subscales and excellent test–retest reliability. Across-informant correlations were modest. Cross-country comparisons revealed that aggression symptoms rated by teachers and ADHD symptoms rated by parents differed between Sweden, Germany and the U.S. Executive functioning deficits also varied as a function of rater and country, with German and Swedish teachers reporting increasing behavior problems with age, whereas a decrease was observed in the U.S. For some subscales, the observed cross-cultural differences were large enough for a child to be classified as being within the normal range (t-score <60) in one country and within the clinical range (t-score > 70) in another country.

Conclusion: The present study shows that the Swedish adaptation of the Conners 3^® provides consistent and reproducible scores. However, across-informant ratings were only modest and significant cross-cultural differences in scoring were observed. This emphasizes the need for multi-informant assessment as well as for national norms for rating instruments commonly used within child and adolescent psychiatry research and clinical settings.

Keywords:

• Rating scale
• standardization
• reliability
• ADHD
• cross-cultural

Introduction

An integral part of the assessment of childhood mental disorders is the collection of multi-informant data, such as ratings made by parents, teachers, and self-ratings. In the fifth version of the Diagnostic and Statistical Manual of Mental Disorders [DSM-5; 1] information from multiple raters and across contexts is mandatory for a diagnosis of attention deficit/hyperactivity disorder (ADHD). In addition, as psychiatric comorbidity is the rule rather than the exception in ADHD [2], assessment of co-occurring symptoms – such as aggression, defiance, conduct problems, executive functioning deficits, peer problems, and learning problems – is recommended by most clinical guidelines for the clinical management of ADHD (e.g. NICE guidelines [NG87]; https://www.nice.org.uk/guidance/ng87). To facilitate these requirements for quality in care and research, psychometrically sound questionnaires are desirable.

The Conners Rating Scales are among the most widely used questionnaires in international research and clinical practice for rating symptoms of ADHD and comorbid externalizing disorders such as conduct disorder (CD) and oppositional defiant disorder (ODD). The third version of the Conners Rating Scales [3] was originally published in 2008 and has subsequently been adapted to several cultures and languages. It integrates the same key components as its predecessors, but also operationalizes novel features, such as a closer link to the DSM-5 [1], including direct item linkage with the symptom criteria for ADHD, CD, and ODD, as well as impairment items to aid evaluation of special education needs. However, for clinical purposes, individual scores on rating instruments such as the Conners 3^® are of limited value without the availability of national norms. Still, despite the cross-cultural differences evident in ADHD ratings [4,5], proper national standardizations are rare, and many countries – particularly those with a small population size or seldom-spoken languages – use U.S. norms.

Given the importance of establishing country-specific psychometric properties, including national norms, the aim of the present study was to conduct a standardization of the Swedish adaptation of the Conners 3^® [6]. More specifically, the study investigated the scores obtained for the Swedish Conners 3^® with regard to internal consistency, retest and inter-rater reliability, population means and standard deviations, as well as age and gender differences. Finally, we conducted culturally informative comparisons of the Swedish, German and U.S. standardizations of the Conners 3^®.

Materials and methods

Participants and procedure

The Swedish standardization of the Conners 3^® included altogether 3496 ratings (1471 parent ratings, 1146 teacher ratings, and 879 self-reports) collected from population-based samples. More detailed information on the number of ratings by age, gender, and informant type is presented in Table 1. Participants were recruited from three different sources. First, we recruited participants through the national Swedish survey panel SPAR, which contains a representative sample of Swedish households across the country, by sending regular mail to a stratified (i.e. child’s age and geographic location) random sample of families with children aged 6–18 years. Second, contacts were made with primary and secondary schools, representing both urban and rural areas throughout Sweden. Third, PFM Research in Sweden Ltd, a company specializing in market surveys, recruited parents, teachers, and children through their web panel containing over 70,000 households. Most of the data from all three sources (84% self-reports, 86% parent, 89% teacher reports) were collected via an online platform, the remainder completed paper forms. Written informed consent was obtained from all participants. The local ethics committee approved the study design and the study has been performed in accordance with the ethical standards of the Declaration of Helsinki.

Table 1. Number of Conners ratings by rater, age and gender for the Swedish standardization.

		Parent reports		Teacher reports		Self-reports
Age	Total	Girls	Boys	Girls	Boys	Girls	Boys
6	167	44	42	40	41	–	–
7	173	43	50	40	40	–	–
8	170	45	43	41	41	–	–
9	331	65	56	41	46	65	58
10	360	76	82	40	52	58	52
11	363	89	64	49	55	55	51
12	337	68	65	43	51	57	53
13	275	58	61	41	40	37	38
14	296	57	62	40	47	41	49
15	302	67	51	50	46	52	36
16	278	64	50	43	45	39	37
17/18	444	86	83	85	89	57	44
Total	3496	762	709	553	593	461	418

Measures

The Conners 3^® [3] assesses ADHD symptoms and behavior problems known to co-occur with ADHD. The first edition of the Conners Ratings Scale was introduced almost 30 years ago [7]. Since then, it has been one of the most commonly used parent and teacher rating scales for assessing childhood behavior problems in clinical and research settings [8]. The scores obtained for the original U.S. version of the Conners 3^® have proven to be reliable and valid [3]. Good psychometric properties have also been reported for the scores obtained for the German version [9,10]. The Conners 3^® assesses not only dimensional levels of ADHD, CD, and ODD traits, but also DSM-oriented symptom scales and diagnostic categories according to the DSM-5 [1]. In the present study, we used the long version of the Conners 3^®, which includes 99–115 items depending on the informant version. The Conners 3^® includes seven content scales (i.e. inattention, hyperactivity/impulsivity, executive functions, learning problems, aggression, peer relations, and family relations) and four DSM-oriented scales (i.e. ADHD inattentive, ADHD hyperactive/impulsive, ODD, and CD). The peer relations subscale is only available for parents and teachers, whereas the family relations subscale is only available for self-ratings. For all three informant groups, symptoms are rated on a 4-point Likert scale with severity ratings from 0 (not at all/never) to 3 (very much/very frequently).

Statistical analyses

The reliability of the Conners 3^® scores was determined by investigating internal consistency, test–retest reliability and inter-rater reliability. Internal consistency was calculated using Cronbach’s alpha for the scores obtained for each one of the instrument’s subscales (see description of the subscales above under ‘measures’). In line with recommendations [11], an alpha-value of .60 to .69 is considered questionable, .70 to .79 fair, .80 to .89 good, and ≥.90 excellent. Test–retest reliability was assessed using Pearson’ correlations for teacher ratings (n = 22), using ratings collected at an interval of 2–4 weeks. Finally, inter-rater agreement was assessed using Pearson’ correlations by comparing the scores provided by parents, teachers, and self-ratings.

Analyses of variances (ANOVAs) were used to investigate effects of age and gender. With regard to effects of age, we followed the procedure used in the original U.S. standardization and divided the sample into three age categories: 6–9 years, 10–13 years, and 14–18 years. For self-reports, the youngest age group only included children from age 9 years onwards, as this is the age limit for the Conners 3^®. All group effects were complemented with measures of effect sizes. We used partial eta-squared (η²) and interpreted the results in accordance with Cohen [11], who states that η² of .01 is a small effect size, η² of .06 a medium effect size, and η² of .14 a large effect size.

Results

Internal consistency and test–retest reliability

Cronbach’s alpha values (r_α) for internal consistency and correlation coefficients (r_tt) for test–retest reliability of the Swedish Conners 3^® scores are presented in Table 2. The results show that the internal consistency was good or excellent for most subscales for parent ratings (r_α = .85 to .91), teacher ratings (r_α = .82 to .97), and self-reports (r_α = .81 to .91). The exceptions were the CD subscale for parents (r_α = .66) and the executive function, CD, and ODD subscales for self-reports (r_α = .62 to .73), which showed a somewhat lower internal consistency. Test–retest reliability was found to be very high over a 2- to 4-week period, as measured by teacher ratings (r_tt = .96 to .99).

Table 2. Internal consistency and test-retest reliability.

	Internal consistency			Test–retest reliability
	Parent report (n = 1471)	Teacher report (n = 1146)	Self-report (n = 879)	Teacher report (n = 22)
	α	α	α	r
Content scales
Inattention	.90	.95	.91	.98
Hyperactivity/impulsivity	.91	.97	.85	.99
Learning problems	.85	.91	.82	.98
Executive functions	.88	.91	.72	.96
Aggression	.81	.95	.81	.99
Peer relations	.82	.90	n.a.	.98
DSM-oriented symptom scales
ADHD inattentive	.89	.95	.86	.96
ADHD hyperactive-impulsive	.88	.94	.82	.99
Conduct disorder	.66	.82	.62	.99
Oppositional defiant disorder	.85	.94	.73	.99

n.a.: not available.

Inter-rater agreement

Across-informant correlations (r_ii) for the different subscales are presented in Table 3. With regard to the relation between mothers and fathers (i.e. informants rating the child within the same setting), the results showed that associations were strong (r_ii = .70 to .91), except for hyperactivity/impulsivity, aggression, ADHD hyperactive/impulsive, and oppositional deficient disorder, for which moderate relations (r_ii = .44 to .58) were found. When studying associations between raters across settings, the results showed that relations were in the moderate range (r_ii = .39 to .60), and that they were fairly similar when studying associations between (1) parent and teacher ratings, (2) parent ratings and self-reports, or (3) teacher ratings and self-reports. The exceptions were the subscales for aggression, CD, and ODD, which showed weak or very weak associations for both parent versus teacher ratings (r_tt = .15 to .28) and for teacher ratings versus self-reports (r_tt = .08 to .23). Associations were also weak, although significant, for the aggression and conduct problem subscales when studying agreement between parent ratings and self-reports (r_tt = .34 to .38).

Table 3. Across-informant correlations.

Scale	Mothers to fathers (n = 26)	Parents to teachers (n = 140)	Parents to youths (n = 139)	Teachers to youths (n = 85)
Inattention	.81***	.51***	.56***	.42***
Hyperactivity/impulsivity	.58***	.43***	.59***	.48***
Learning problems	.91***	.48***	.65***	.54***
Executive functions	.70***	.47***	–	–
Aggression	.44***	.15	.38***	.15
Peer relations	.45***	.39***	–	–
ADHD Inattentive	.76***	.47***	.59***	.50***
ADHD Hyperactive-Impulsive	.58***	.43***	.60***	.40***
Conduct disorder	.71***	.16	.34***	.08
Oppositional defiant disorder	.50***	.28***	.49***	.23*

*** p < .001;

* p < .05.

Normative data and effects of age and gender

Means and standard deviations for the parent and teacher ratings on all Conners 3^® content scales divided by age and gender are presented in Table 4. For self-ratings, means and standard deviations are presented in the Swedish Conners 3^® manual [6]. Effects of age and gender are summarized in Table 5. The results showed that for both parent and teacher ratings, effects of age were found for the subscales hyperactivity/impulsivity and ADHD hyperactive-impulsive. For parent ratings, children in the youngest age group were rated as having higher levels of problem behaviors than were children in the middle age group, who in turn had higher problem levels than children in the oldest age group. For teacher ratings, the two youngest age groups were rated as having higher levels of hyperactivity/impulsivity compared to the oldest age group. An effect of age was also found for peer problems for parent ratings and for executive functions for teacher ratings. For peer problems, the two youngest age groups did not differ from one another, although children in both these groups showed lower levels of problem behaviors than did children in the oldest age group. For executive functioning, the youngest age group was rated as having lower levels of problem behaviors than the two older age groups. For self-reports, significant effects of age were found for the following subscales: inattention, learning problems, and ODD. For inattention, the youngest age group had the lowest problem levels, which differed significantly from those of the middle age group, and the oldest age group had the highest problem behavior levels. For both learning problems and ODD, the two youngest age groups rated themselves as having significantly lower levels of problem behaviors compared to the self-ratings of the older age group. The effect sizes were all within the small range.

Table 4. Means (M) and standard deviations (SD) for parent and teacher ratings for all content scales divided by age and gender.

		Inattention				Hyperactivity/impulsivity				Learning problems				Executive functions				Aggression				Peer problems
		Parents		Teachers		Parents		Teachers		Parents		Teachers		Parents		Teachers		Parents		Teachers		Parents		Teachers
Age		M	F	M	F	M	F	M	F	M	F	M	F	M	F	M	F	M	F	M	F	M	F	M	F
6	M	6.05	4.59	5.83	2.78	7.5	6.18	9.80	4.53	5.24	3.98	2.93	1.73	5.38	3.91	4.29	2.98	2.36	1.34	5.46	1.23	1.14	0.82	3.10	1.00
	SD	4.55	2.99	5.44	3.86	5.3	5.78	10.57	5.61	4.45	3.17	3.38	2.53	3.92	2.18	2.81	2.13	2.23	1.49	7.78	2.30	1.78	1.40	3.71	1.67
7	M	4.68	4.59	7.98	3.26	6.2	5.90	14.85	6.28	3.64	3.10	3.35	1.82	4.20	4.17	5.65	3.54	2.54	2.12	5.28	2.26	1.80	1.37	3.93	2.67
	SD	3.58	3.48	7.25	4.50	5.6	5.28	14.81	9.10	4.37	2.70	4.33	2.47	3.37	2.76	4.51	2.36	3.68	2.85	6.88	4.12	2.81	1.86	4.46	3.28
8	M	7.21	4.59	7.93	4.20	5.9	5.25	16.61	4.90	3.69	3.39	2.54	3.02	6.31	4.86	6.85	3.32	2.88	1.73	6.98	1.00	2.19	1.16	3.63	1.76
	SD	5.98	3.18	7.00	5.97	6.8	4.98	14.89	7.98	3.75	3.08	2.83	4.08	5.01	3.53	4.44	3.38	3.25	2.37	8.05	2.06	2.84	1.83	4.25	2.62
9	M	5.60	5.35	8.15	3.24	6.0	4.44	14.20	5.83	3.33	2.97	3.00	1.39	5.71	5.06	7.22	3.51	2.51	1.56	6.78	2.98	1.11	1.24	4.09	3.07
	SD	4.10	4.61	7.87	4.03	4.78	5.30	14.07	8.89	3.61	3.50	3.78	1.99	3.70	4.16	5.37	3.30	2.65	1.99	9.03	5.39	1.37	1.69	4.19	3.80
10	M	6.24	4.53	9.40	3.78	6.06	4.48	12.23	5.35	4.06	3.03	3.79	2.65	6.41	4.78	7.38	3.98	1.89	1.53	6.04	2.40	1.32	1.10	4.06	3.35
	SD	4.77	3.72	7.90	4.67	5.76	4.96	12.63	7.75	4.48	3.66	4.36	3.70	4.23	3.50	5.35	3.82	2.56	2.09	8.15	4.00	1.89	2.10	4.49	4.55
11	M	5.65	4.97	8.56	4.92	5.44	4.85	12.67	6.39	2.37	2.91	3.00	2.69	5.46	4.86	7.02	5.01	2.17	2.31	5.71	3.16	1.46	1.40	4.25	2.78
	SD	4.01	5.21	7.44	6.30	5.86	6.70	11.83	9.84	3.04	3.84	3.86	3.73	4.55	5.05	5.06	4.54	2.43	3.25	8.16	5.36	1.83	2.45	4.71	3.87
12	M	6.40	4.02	8.36	3.30	5.00	3.29	13.20	5.60	3.42	2.52	3.63	1.72	6.26	4.10	7.22	3.77	2.79	1.65	4.90	2.58	2.19	1.63	2.98	2.37
	SD	4.70	4.07	7.03	3.92	4.89	3.96	13.23	7.94	4.38	3.78	4.10	2.40	4.08	3.88	5.01	3.50	3.04	2.20	6.24	5.34	3.29	2.44	3.10	3.72
13	M	5.95	4.93	7.93	3.44	4.64	3.74	15.10	4.05	4.24	3.29	3.05	2.10	6.61	4.84	6.30	4.02	2.37	2.27	6.40	1.80	2.31	1.80	3.73	2.37
	SD	4.70	3.83	7.98	4.89	5.23	4.15	14.02	5.56	4.72	3.94	4.18	3.01	4.70	3.84	5.40	3.73	2.71	3.01	8.97	3.81	2.15	2.65	4.03	3.21
14	M	6.08	4.76	6.68	4.05	3.47	3.24	8.89	3.40	3.53	3.91	3.40	2.85	6.49	5.35	6.81	4.08	2.47	2.41	4.57	1.48	2.73	1.72	3.70	1.80
	SD	4.73	4.57	5.98	6.14	4.19	3.81	11.88	5.34	3.64	4.69	4.34	3.22	5.06	4.98	4.34	4.14	2.81	2.83	7.80	3.57	3.48	2.34	3.97	3.03
15	M	5.33	4.10	6.52	4.68	2.84	3.13	7.96	5.44	3.51	2.92	2.57	2.82	6.20	4.95	6.46	4.54	1.96	1.98	3.83	3.18	3.00	1.80	3.24	3.26
	SD	4.14	3.82	6.78	5.67	2.96	4.23	10.66	7.63	4.48	4.18	3.65	3.28	4.80	4.41	4.93	3.64	2.62	3.28	6.87	5.88	3.48	2.81	3.27	3.80
16	M	4.60	4.03	7.00	6.23	2.81	3.85	7.80	7.49	3.27	3.90	2.58	3.05	5.90	4.59	6.76	5.02	1.56	1.97	4.00	2.56	2.00	2.21	3.00	2.33
	SD	5.21	3.77	6.98	7.32	5.90	4.62	10.85	9.84	4.35	4.04	3.19	4.70	4.27	4.05	5.59	4.31	2.24	2.13	5.37	4.60	2.92	2.88	3.72	3.33
17/18	M	5.52	4.30	8.99	4.74	3.38	2.84	10.16	6.06	4.11	2.73	4.12	2.53	6.77	4.84	7.97	5.21	1.73	1.53	3.97	2.64	2.04	1.64	3.06	2.64
	SD	4.60	4.13	7.70	5.23	4.48	3.95	11.34	6.76	4.98	3.46	4.46	3.25	4.84	4.83	5.62	4.55	2.62	2.40	6.68	5.34	2.26	2.44	3.60	3.40

Table 5. Overview of the effects of age and gender.

	Effects of gender			Effects of age
Scale	Parent ratings	Teacher ratings	Self-reports	Parent ratings	Teacher ratings	Self-reports
Content scales
Inattention	boys > girls	boys > girls	NS	NS	NS	3 > 2 > 1
Hyperactivity/impulsivity	boys > girls	boys > girls	boys > girls	1 > 2 > 3	1, 2 > 3	NS
Learning problems	boys > girls	boys > girls	NS	NS	NS	3 > 1, 2
Executive functions	boys > girls	boys > girls	—	NS	1 < 2, 3	NS
Aggression	boys > girls	boys > girls	boys > girls	NS	NS	NS
Peer relations	boys > girls	boys > girls	—	1, 2 < 3	NS	NS
Family relations	—	—	boys > girls	—	—	NS
DSM-oriented symptom scales
ADHD Inattentive	boys > girls	boys > girls	NS	NS	NS	NS
ADHD Hyperactive-Impulsive	boys > girls	boys > girls	boys > girls	1 > 2 > 3	1, 2 > 3	NS
Conduct disorder	NS	boys > girls	boys > girls	NS	NS	3 > 1, 2
Oppositional defiant disorder	NS	boys > girls	NS	NS	NS	NS

NS = Non significant.

Effects of gender were more pronounced than effects of age, and gender effects were found for all subscales for all raters, except for ODD for parent ratings. Effect sizes were all within the small range except for the teacher ratings on the subscales measuring ADHD symptoms (i.e. the subscales inattention, hyperactive-impulsive, ADHD inattentive, ADHD hyperactive-impulsive, and ADHD Index) and executive functioning, which were in the medium range.

Cross-cultural comparisons

With regard to internal consistency, the scores obtained for the Swedish, U.S. [3] and German [10] standardization of the Conners 3^® are similar, with high alpha-values (i.e. r_α > .80) for most subscales. The exception was the score for the CD subscale, which had the lowest internal consistency in all three countries (r_α ranging from .62 to .83). Similarly to what was found in the present study, test–retest reliability estimates were found to be high for the scores obtained for the U.S. (r_tt = .77–.83) and German (r_tt = .76–.83) versions of the Conners 3^®.

With regard to effects of gender, parents and teachers rated boys, compared to girls, as having higher levels of problem behaviors on most subscales, also in the U.S. and Germany. With regard to self-ratings, no significant gender differences were obtained in the U.S. However, compared to girls, boys rated themselves as having higher levels of problem behaviors with regard to the subscales assessing hyperactivity/impulsivity, aggression, conduct problems, and family relations in both Sweden and Germany. It should be noted, however, that a larger number of significant gender differences were found for teacher/parent ratings than for self-ratings also in Sweden and Germany.

We also noted some interesting differences between the Swedish, U.S. and German standardization when investigating the raw scores for the different subscales. First, the mean values for teacher ratings of executive functioning were shown to increase (i.e. higher problem levels) during the age period 6–9 years in Sweden and Germany, whereas a decrease was found in the U.S. Second, compared to U.S. teachers, Swedish teachers rated children, especially boys, as having higher levels of aggression and ADHD symptoms, with the German mean values falling in between. However, for parent ratings, German parents rated children higher on most subscales as compared to both the Swedish and U.S. parents. These differences sometimes result in very different interpretations regarding the severity of the behavior problem depending on the child’s country of origin. For example, a raw score of 9 on the aggression subscale for a 15-year-old boy rated by his teacher would be translated to a T-score of 58 (i.e. within the normal range) in Sweden and Germany, but a t-score of 76 in the U.S. (i.e. clinical range). Norms were, nonetheless, fairly similar for all three countries for other scales, for example learning problems.

Discussion

The present study examined the psychometric properties of the scores obtained for the Swedish adaption of Conners 3^®. The results showed good internal consistency for most subscale, and excellent test–retest reliability for teacher ratings. Moderate associations were found between different informants, except for the subscales assessing aggression, CD, or ODD, for which only weak relations were found when comparing parent ratings, teacher ratings, and self-reports. Boys were generally regarded as having higher levels of problem behaviors, and gender differences were more pronounced for teacher and parent ratings than for self-ratings. Significant age effects were primarily found for hyperactivity/impulsivity and peer problems, and all of these had small effect sizes. Important cross-cultural differences between the Swedish, U.S., and German standardization of the Conners 3^® were found, which emphasizes the need to collect national norms for rating instruments commonly used within child and adolescent psychiatry research and clinical settings.

With regard to internal consistency, values were high (i.e. r_α > .80) for most subscales rated by teachers and parents, except for conduct problems as assessed by parents. For self-ratings, the internal consistency was below .80 for three subscales: executive function, CD, and ODD. The reason why the CD subscale had somewhat lower reliability compared to the other scales in the Swedish, U.S., and German standardization could be that it includes all four symptom domains found in the DSM-5 for CD: (1) aggression toward people and animals, (2) destruction of property, (3) deceitfulness or theft, and (4) serious violations of rules [1]. As these different types of problem behavior do not necessarily occur to the same extent in an individual, it is understandable that the internal consistency has repeatedly been shown to be somewhat lower for this subscale. Because some subscales involve behaviors that do not necessarily always co-occur, it is recommended that when using the Conners 3^® in clinical practice, an item-level analysis should be conducted to determine what items are contributing most to an elevated score on each subscale [3].

Concerning test–retest reliability for the scores obtained for the Swedish adaption of the Conners 3^®, and as shown in a review by Pelham and colleagues [12], estimates are equally high or higher than those obtained for other rating scales assessing ADHD symptom levels. However, it should be noted that the very high test–retest estimates for the scores obtained for the Swedish standardization of the Conners 3^® (i.e. all r_tt > .95) may be an overestimation, as two ratings from the same informant were only obtained from highly experienced teachers and the sample size for this analysis was small (n = 22). It will be important for future studies to replicate these findings using a large sample and to also investigate test–retest reliability using the scores obtained from self- and parent ratings.

With regard to effects of gender, boys were generally shown to have higher levels of problem behaviors compared to girls. With a few exceptions, the same pattern of results was obtained in both the U.S. and Germany. However, it is interesting to note that, especially in the U.S., but even to a lesser extent in Sweden and Germany, gender differences were found in many more areas for teacher/parent ratings compared to self-ratings. The reason for this finding could be that the children themselves use another frame of reference when rating their own behavior. It could be the case, for example, that children self-rate their behaviors in comparison with other same-sex children, whereas parents and teachers make more across-sex comparisons. There is also previous research showing that boys tend to under-report their problem levels more than girls do [13], which could explain why gender differences were less pronounced for self-ratings.

Only modest associations were found between the scores obtained from different sources (i.e. parents, teachers and self-ratings), which is in line with previous research [see review 14]. The crucial question concerns the reason for this disagreement. Hartman and colleagues [15] used twin modeling to examine parent and teacher ratings of ADHD symptoms and they concluded that disagreement is strongly due to raters observing different behaviors, some of which is valid and some of which is due to bias. They further argued that parent ratings are more biased than teacher ratings, which is most likely due to teachers having experience of the behaviors of a larger number of children. In addition, ADHD symptoms may be more easily observed in the school setting.

Meyer and colleagues [14] emphasized that different methods have particular strengths and weaknesses and that each method therefore identifies useful data not available from other sources. Thus, when inconsistencies in ratings between informants are found, this does not necessarily indicate poor validity for one type of rating, but rather that multiple reports are needed to obtain a complete picture of a child’s behavior problems. For instance, self-ratings can provide a more in-depth understanding of the underlying reasons for children’s attention problems, as exemplified in the U.S. Conners 3^® manual [3], where a teenaged girl is found to have attention problems in school primarily because she is depressed. That said, the extent to which children are capable of making valid ratings of their own behavior could also be called into question. For instance, one recent study found that parent ratings of ADHD symptoms in early and late adolescence, using the Child Behavior Checklist [CBCL; 16], significantly predicted a large number of negative outcomes in early adulthood, while self-ratings, using the Youth Self-report [YSR; 17], did not contribute to predicting almost any outcome once parent ratings had been taken into account [18]. However, with regard to the scores obtained for the Swedish adaption of the Conners 3^®, it should be noted that the internal consistency was similar across raters, and across-informant correlations were generally higher between teacher and self-ratings compared to teacher and parent ratings. Thus, for the Conners 3^®, there are no reasons why the scores for the self-ratings should not be considered just as reliable as those obtained for teacher and parent ratings. However, future studies should investigate test–retest reliability for the scores obtained for self- and parent ratings as well, and include a clinical sample, as it has been suggested that children with ADHD tend to have inflated self-perceptions of their own competences (i.e. a higher positive illusory bias compared to controls) [19]. Thus, self-ratings made by children and adolescents with ADHD need to be evaluated with caution, especially if they show low concurrence with both parent and teacher ratings.

With regard to raw scores, deficits in executive functioning were increasing between age 6 and 9 years in both Sweden and Germany, but decreasing in the U.S. Given that studies investigating the development of executive functioning across childhood have consistently found that these skills increase during childhood [20], there is little reason to believe that these age differences indicate that young children are indeed more skilled than older children. Instead, it is likely that teachers’ demands on executive skills differ depending on the child’s age, and such demands may differ across countries. In Sweden, the increase in problems with executive functioning is especially large between age 6 and 7, which could be related to the fact that Swedish children do not start mandatory schooling until August the year they turn 7 years old, and the demands on executive skills among 6-year-olds (who are often still in preschool) are therefore relatively low.

Based on the present results, we cannot tell whether there are true differences between how children behave in different countries or to what extent these differences instead reflect differences in how children from different countries are perceived by parents or teachers, or how children view themselves. In order to clarify this, both ratings and more objective measures (i.e. performance-based tests or observations made by objective observers) are needed. In previous studies including both performance-based tests and ratings of executive deficits, it has been demonstrated that cross-cultural differences were primarily observed for ratings and not tests [5]. In addition, cross-cultural differences were found when mental health professionals were asked to rate hyperactive-disruptive behaviors depicted in standardized videotaped vignettes [21]. In fact, cultural norms and rater biases can influence the result so such a large extent that completely different conclusions are drawn based on ratings compared to more objective measures. For example, Chinese children have been shown to be rated as having more severe executive deficits compared to children from western countries when using ratings [5], whereas the opposite pattern of results has been found using laboratory tests [22,23]. Sex differences is another area that is greatly influenced by cultural norms. One previous study [5] found that Iranian parents rated girls as having more executive deficits compared to boys, whereas the opposite pattern was found in other countries (i.e. Sweden, Spain, and China), and no significant sex differences were found for a laboratory test of working memory. Altogether, the issue of rater bias emphasizes the need for studies like the present one as the psychometric properties of an instrument are assessed for the scores produced in a particular sample rather than for the instrument itself.

Conclusively, although this is an area where more research is clearly needed, there are reasons to believe that cross-cultural differences in children’s behavior problems at least partly reflect the subjective perceptions of the rater rather than actual differences in how the child is behaving. Understanding rater biases is a complex issue, which includes societal expectations of children’s behaviors and this can greatly contribute to cross-cultural differences such as those observed in the present study.

Conclusion

Based on the results of the present study, the scores obtained for the Swedish adaption of the Conners 3^® should be regarded as reliable for assessing ADHD symptoms and related behavior problems in children and adolescents. Considering that research collaborations across different countries are increasing and that higher demands are being placed on clinics to use standardized procedures when diagnosing patients, it is of great value to have international ratings instruments available. Thus, the present study adds valuable new information by examining the intercultural comparability of one of the most commonly used questionnaires within child and adolescent psychiatry. As we found several important differences between the scores obtained for the Swedish, U.S., and German versions of the Conners 3^®, our results clearly emphasize the need of not merely translating rating scales into different languages, but also establishing updated country-specific norms.

Acknowledgements

The authors wish to thank all participating teachers, parents, and children. In addition, the authors want to thank Anna Borg, Ina Ghai, and Hogrefe Psykologiförlaget for their support during the data collection.

Disclosure statement

Dr. Bölte discloses that he has acted as an author, consultant, or lecturer for Shire, Medice, Roche, Eli Lilly, Prima Psychiatry, GLGroup, System Analytic, Kompetento, Expo Medica, and Prophase. He receives royalties for textbooks and diagnostic tools from Huber/Hogrefe, Kohlhammer and UTB. Martin Hammar, Steve Berggren, and Eric Zander have no conflict of interest to report.

Additional information

Funding

This research was supported by grants from The Swedish Order of Freemasons and Sällskapet Barnavård to the last author. Lisa B. Thorell, Hanna Christansen, and Sven Bölte have acted as consultants and have received royalties or research funding from Hogrefe.