Concurrent Validity of the Olweus Bully/Victim Questionnaire

Abstract

This study examined the concurrent validity of the Olweus Bully/Victim Questionnaire (BVQ) in a convenience sample of 202 middle school students in central Virginia. This appears to be the first published study to compare BVQ reports of being bullied and of bullying others with independent criteria not subject to shared method variance. Self-reported bullying on the BVQ was significantly correlated with peer nominations for bullying (r = .12, p < .05) and academic grades (r = −.15, p < .05), but not disciplinary infractions. Self-reported victimization was significantly correlated with peer nominations for victimization (r = .42, p < .01) and academic grades (r = −.12, p < .01). These results provide only modest support for the concurrent validity of the BVQ and raise concern about reliance on student self-report to measure school bullying.

KEYWORDS:

• Olweus Bully/Victim Questionnaire
• bullying
• concurrent validity
• self-report
• peer nominations

Self-report surveys are the most frequently used measures to evaluate the success of anti-bullying programs and estimate the prevalence of bullying in schools (Cornell, Sheras, & Cole, 2006). Self-report measures are a rapid and efficient method of gathering data from large numbers of students; however, there is limited research available on their reliability and validity (Cornell et al., 2006; Griffin & Gross, 2004; Leff, Power, & Goldstein, 2004). Self-report bullying surveys are usually measured on an anonymous basis, so that researchers are unable to corroborate student reports or assess their relationship to any independent criteria. This measurement strategy has a profound effect on the field, because the vast majority of bullying research and bullying prevention efforts rely on anonymous self-report measures of unproven validity (Cornell et al., 2006).

The conventional wisdom is that it is necessary to use an anonymous survey in order for students to be willing to provide honest answers; however, there is reason to question this assumption. Three studies have compared anonymous and nonanonymous versions of the same survey to determine whether students are less likely to admit bullying or other problem behaviors when their identity is not anonymous. In a sample of 704 Dutch adolescents, approximately half of the students received questionnaires labeled with their names, but were assured of confidentiality, while the other half were told that their responses would not be connected to their identity in any way (Van de Looij-Jansen, Goldschmeding, & Jan de Wilde, 2006). They found no significant differences between the two groups in reporting bullying, as well as carrying a weapon, using hard drugs, or being truant from school.

Another study by Chan, Myron, and Crawshaw (2005) investigated differences between anonymous versus confidential self-reports of bullying and victimization on the School Life Survey (SLS; Chan, 2002), a self-report questionnaire designed to measure bullying. The SLS was administered to 562 students (ages 6–13) from 30 classrooms. Students were randomly assigned to write their full names on the survey or to take the survey anonymously. There were no significant group differences in reported rates of bullying behaviors or victimization experiences such as hitting, teasing, and lying about other students.

O'Malley, Johnston, Bachman, and Schulenberg (2000) examined the difference between anonymous and confidential reporting on the Monitoring The Future Survey, which is administered annually to nationally representative samples of adolescents in the United States. The survey has a lengthy series of questions concerning personal use of illegal drugs and alcohol, as well as delinquent behaviors such as stealing and weapon-carrying. In this study of more than 34,000 students, half of the youths were administered the survey with explicit assurance that their answers were anonymous and so could not be linked to them, while the other half were told that their answers would be held in confidence, but were required to report their name and address to the researchers. There was little or no difference in the reporting of sensitive information under the two conditions (O'Malley et al., 2000).

Olweus Bully/Victim Questionnaire

The Olweus Bully/Victim Questionnaire (BVQ; Olweus, 1996, 2002) is the most widely used bullying self-report survey in the world (Nansel et al., 2001). Studies using the BVQ have been conducted in at least 15 countries, including Australia, China, Japan, many European countries, and the United States (Eslea et al., 2003; Kyriakides, Kaloyirou, & Lindsay, 2006; Rigby & Slee, 1991; Smith, Madsen, & Moody, 1999; Smith, Morita et al., 1999; Solberg, Olweus, & Endresen, 2007; Zhang, Gu, & Wang, 2000). Studies using the BVQ have yielded a range of prevalence rates for bullying. Eslea et al. (2003) found the percentage of students classified as bullies spanned from 2.0% in China to 16.9% in Spain. Students classified as victims of bullying ranged from 5.2% in Ireland to 25.6% in Italy. Because the survey is usually completed on an anonymous basis, it is not possible to determine whether this range reflects true prevalence differences or if reports of bullying are influenced by uncontrolled factors such as student attitudes toward completing the survey or student familiarity with the concept of bullying.

There is some evidence supporting the construct validity of the BVQ (Olweus, 1999). In an intervention study, Olweus (1994) found correlations in the .60–.70 range between class-aggregated student ratings of bullies and victims, and class-aggregated estimates of self-report ratings. This provides indirect evidence of validity, but does not address the accuracy of the BVQ because these correlations are essentially measures of agreement or consistency in student perceptions of bullying. Olweus acknowledges the lack of published support for the validity of the BVQ, but claims that such evidence exists:

We have made lots of analyses on the internal consistency (reliability), the test-retest reliability and the validity of the Revised Olweus Bully/Victim Questionnaire on large representative samples (more than 5,000 students). The results are generally quite good…Unfortunately, most of this psychometric information has not yet been published, due to lack of time. (Olweus, 2002, p. 1) ¹

The extraordinary value of Olweus's contributions to bullying research and prevention notwithstanding, it is important to have substantial published evidence supporting the validity of an instrument that is used worldwide and serves as the cornerstone for evaluating the effectiveness of bullying prevention programs. The disappointing outcome results for many schoolwide bullying prevention efforts using the Olweus model (Smith, Schneider, Smith, & Ananiadou, 2004) only heighten the need to examine the accuracy of the BVQ (Cornell & Bandyopadhyay, 2010; Cornell et al., 2006).

Solberg and Olweus (2003) examined appropriate cut-offs for estimating bullying prevalence using the two global questions on the BVQ: “How often have you been bullied at school in the past couple of months?” and “How often have you taken part in bullying another student at school in the past couple of months?” They suggest that reporting a frequency of two or three times a month or more for either question distinguishes bullies from nonbullies on measures of externalizing behaviors and victims from nonvictims on measures of internalizing behaviors. One limitation of this finding is that the internalizing and externalizing indicators were contained on the same anonymous self-report survey and correlations could be inflated to an unknown degree by shared method variance. No independent criteria were used to substantiate whether the students were actually involved in bullying or demonstrated characteristics associated with bullying. We were unable to locate any published study examining the correspondence of the two BVQ global questions with independent criteria.

Peer Nominations

Peer nomination is perhaps the most widely used and well-validated method of assessing peer relationships, aggression, and victimization (Achenbach, McConaughy, & Howell, 1987; Ladd & Kochenderfer-Ladd, 2002; Pellegrini, Bartini, & Brooks, 1999). The most straightforward form of this method asks students to nominate classmates who were victims of bullying or had bullied others. The number of nominations a student receives is used as an indicator of his or her bully status, and a cutoff may be used to classify a student as a bully or victim. Since the data are gathered from multiple sources, the use of peer nominations can reduce measurement error and increase reliability (Achenbach et al., 1987; Cornell et al., 2006). Peer nominations are considered efficient and economical, and are particularly sensitive to detecting relational aggression, which is harder to observe (Pellegrini, 2001).

Cornell and Brockenbrough (2004) compared self-report and peer nomination using the School Climate Bullying Survey (SCBS; Cornell & Sheras, 2003). Both surveys asked about bullying within the past month. In a sample of 416 middle school students, they found a small correlation (r = .17) between self-report and peer nominations in identifying victims.

Pellegrini (2001) required students to write their names on the BVQ, but measured victimization with a 9-item scale that asked about behaviors such as teasing and hitting rather than using the two global questions recommended by Solberg and Olweus (2003). Self-reports of victimization were significantly correlated with peer nominations (.32) and diary entries (.34).

Other criteria

Discipline infraction rates are another indicator used to assess the effectiveness of bullying prevention programs (Leff et al., 2004; Wright & Dusek, 1998). As rates of bullying decline, student referrals and suspensions for misconduct should also diminish. Several studies have found that bullies had higher numbers of discipline infractions than other students (Clarke & Kiselica, 1997; Espelage, Bosworth, & Simon, 2000; O'Moore & Hillery, 1991).

Poor academic performance is also commonly associated with bullying. There are multiple reasons why victims of bullying would demonstrate poor academic performance, such as anxiety, poor concentration, and attendance problems. Bullies also tend to have poor grades, which could be both a cause and effect of their aggressive behavior (Luiselli, Putnam, Handler, & Feinberg, 2005; Pereira, Mendonça, & Neto, 2004). Aggressive students who are disrespectful of others may be less willing to apply themselves in their schoolwork. Conversely, students with poor academic aptitude may resort to bullying out of frustration or in an effort to recoup a sense of self-esteem.

The purpose of this study was to examine the concurrent validity of self-reports of bullying others and being a victim of bullying using the BVQ. Student reports on the BVQ were compared with independent criteria that included peer nomination surveys, discipline records, and student academic grades. We hypothesized that self-reports of bullying others on the BVQ would be associated with more peer nominations for bullying others, more discipline referrals and suspensions, and lower grades. In addition, we expected that self-reports of being a victim on the BVQ would be associated with more peer nominations as a victim and lower grades.

In order to link self-reports to independent criteria, we administered the BVQ on a confidential, but not anonymous, basis. In order to minimize students' sense of exposure, they were not asked to provide their names on the survey; instead, each form was marked with a code number linked to an individual student. Each survey form was placed in a corresponding envelope with the student's name, but when teachers distributed the surveys, they discarded the envelopes so that they could not link completed surveys to students. The completed surveys were collected in a single envelope and students were assured that no one other than the researchers would see their responses. In this way, the procedure came as close as possible to anonymity while permitting researchers to link the surveys to other sources of information.

METHOD

Participants

Researchers obtained IRB approval to conduct the surveys and collect other student information used in this study. Data were gathered from a convenience sample of 202 students attending a suburban middle school in central Virginia during the 2005–2006 school year. Ninety-eight (48%) students were boys and 104 (52%) were girls, with a mean age of 12 (range 10 to 14; SD = .90). Students were in grades 6 (37%), 7 (34%), or 8 (29%). The ethnic breakdown was 66% White, 19% African American, 3% Hispanic, 2% Asian American and 6% other.

Measures

Olweus Bully/Victim Questionnaire (BVQ)

Students completed the Revised BVQ (Olweus, 1996), a 38-item self-report measure that includes questions about bullying involvement, attitudes towards bullying, and school climate. The questionnaire provides a lengthy definition of bullying that includes verbal and physical aggression, as well as social forms of aggression such as excluding someone and spreading false rumors about someone.

As recommended by Solberg and Olweus (2003), the two target questions examined in this study were, “How often have you taken part in bullying another student(s) at school the past couple of months?” and “How often have you been bullied at school in the past couple of months?” All of the responses were based on a 5-point Likert scale: It hasn't happened to me in the past couple months, Only once or twice, Two or three times a month, About once a week, and Several times a week.

There are 18 additional questions on the BVQ—9 relating to bullying others, and 9 relating to being bullied—that ask about specific behaviors that are often associated with bullying. Two sample statements are: “I was called mean names, was made fun of, or teased in a hurtful way,” and “I was threatened or forced to do things I didn't want to do.” These groups of questions were used as an additional way to identify possible bullies and victims.

Several studies using the BVQ apply different criteria to define bullying or classify bullies and victims (Dulmus, Theriot, Sowers, & Blackburn, 2004; Eslea et al., 2003; Kyriakides et al. 2006; Pellegrini, 2001); therefore, this study examined three different methods of identifying bullies and victims from the BVQ. The first method used the nine BVQ questions that asked students whether they had engaged in (or been the object of) a series of behaviors associated with bullying, such as being teased or threatened. Any students who endorsed any one of the nine questions with a frequency of at least “once or twice” in the past couple months was classified as a bully (or victim). This procedure was labeled the 9-item criterion.

The second method used the two target questions, “How often have you taken part in bullying another student(s) at school the past couple of months?” and “How often have you been bullied at school in the past couple of months? Any students who answered “only once or twice” or more for either of the two target questions was identified as a perpetrator or victim of bullying. This cut-off was labeled the “at least once” criterion.

The most stringent method used to classify bullies and victims used the standard cut-off recommended by Solberg and Olweus (2003): at least “two or three times a month” for the two target questions. The Olweus standard cut-off yielded only 2 students who identified themselves as bullies and 13 who identified themselves as victims. Therefore, the other two methods using more inclusive criteria were analyzed. Too few students were identified as both bullies and victims to permit analysis of a separate bully/victim group. Collection of a larger sample might provide sufficient numbers of students who identify themselves as both bullies and victims.

Peer nomination

A peer nomination form (Cornell & Brockenbrough, 2004) used in other studies comparing self-report to peer report (Cole, Cornell, & Sheras, 2006; Cornell & Brokenbrough, 2004) was appended to the BVQ and was completed last by all students. The form presented a list of all students in each grade rather than just students in that classroom, because students changed classes several times a day. This procedure allowed students to identify victims of bullying who might have been observed in other classes, on the playground, on a school bus, or in any other location at school. Participants were instructed to write the names of any students (including themselves) whom they knew to be victims of bullying or to bully others in the past month.

The BVQ questions asked about bullying in the past couple of months while the peer nomination asked about bullying in the past month. This discrepancy in time period might contribute to the low correspondence between the two measures if students were precise in their assessment of when the bullying occurred and were involved in bullying during one month but not the other. However, the SCBS, which was administered to the other half of the student body, asked nearly identical questions about bullying others and being a victim of bullying, but used a one month time period. Despite the differences in time period, the two instruments produced similar prevalence rates. On the BVQ, 0.5% of students reported bullying others “about once a week” and no students reported bullying others “several times a week,” Although the SCBS used a shorter time period, the rate of bullying others “about once a week” was 2.6% and again no students reported bullying others “several times a week.” For the question about being bullied by others, again there was close correspondence between the two measures, with the SCBS producing a slightly higher incidence. The corresponding percentages were 0.5% for the BVQ and 1.6% for the SCBS for being bullied “about once a week” and 2.0% for the BVQ and 2.1% for the SCBS for being bullied “several times a week.”

School discipline records and grades

The numbers of discipline referrals and suspensions for each student were obtained for the school year. The discipline referrals included all forms of violations because students who bully others engage a wide variety of rule-breaking and misbehavior (Olweus, 1993). All teachers in the school were trained to use a standard disciplinary referral form based on statewide categories and definitions of disciplinary infractions. Disciplinary referrals were entered by the school principal or assistant principal into a database that was used to prepare mandatory reports for the school division, and ultimately, the state disciplinary reporting system. In addition, year-end letter grades were obtained for four core subjects (Math, Social Studies, English, and Science). Grade point averages (GPAs) were calculated using the 4-point scale (A = 4, B = 3, C = 2, D = 1) often used in the United States.

Procedures

Prior to this study, the school had already implemented bully prevention efforts informally modeled after the Olweus Bullying Prevention Program (OBPP; Olweus, Limber, & Mihalic, 1999). In the summer preceding the study year, the principal and a teacher obtained formal training in the OBPP model and used the OBPP materials to train teachers and other staff to carry out schoolwide, classroom-level, and individual-level interventions (Olweus, 2002). At the start of the school year, a letter was sent home to parents informing them about the survey and giving them the option to decline their child's participation in the survey (no parents declined). The school did not obtain individual student consent for participation in its bully prevention program because it was a regular part of the school curriculum, conducted by the school staff, and was not initiated for research purposes. The program began with a student assembly on bullying and peer relations, followed by classroom discussions and review of school rules regarding bullying.

In previous years, the school administered the SCBS (Cornell & Sheras, 2003) to all students; however, after adopting the Olweus program, the BVQ became available as an alternative. In order to retain continuity with previous survey results, the school decided to administer both instruments. Surveys were distributed on an alternating basis so that half of the students in each classroom received one or the other of the two surveys.

Van de Looij-Jansen and colleagues (2006) recommend that, in order to foster honest responses, researchers should take extra measures to assure students that their answers are confidential. Some of their suggestions include using codes instead of names, sealing questionnaires in unmarked envelopes immediately after completion, and giving clear instructions about the method used. For the present study, each student was assigned a code number known only to a single code officer who did not have access to the completed surveys.

The surveys were administered in classrooms under teacher supervision. Each class listened to a standard set of directions for the survey broadcast on classroom televisions. Students were assured that their individual responses would not be disclosed to their teachers, parents, or anyone other than the researchers, who would not know their names.

The peer nomination forms were separated from the surveys so that names of nominated students could be tabulated by the code officer. The code officer gave researchers a tally of nominations for each student using the student's code number. In this way, the researchers could merge peer nomination and self-report survey data without knowledge of student names. The code officer also reviewed student grades and discipline records in an electronic file, and removed student names and substituted code numbers before turning the data over to the research team.

The school readministered the BVQ to the same students in the spring. We conducted a parallel set of analyses on the spring survey to see if the results were consistent.

RESULTS

Bullies

Preliminary analyses found no statistically significant gender differences (by t-test) for self-reported bullying or victimization, or peer nominations as a bully or victim, so gender was not used in further analyses. On the BVQ, the majority of students (87.6%, n = 177) reported not bullying others in the past couple of months, 10% (20) reported bullying others “once or twice,” 0.5% (1) reported bullying others “two or three times a month,” and 0.5% (1) reported bullying others “about once a week.” The 9-item criterion classified 49 students as bullies, the “at least once” criterion classified 22 students as bullies, and the Olweus standard criterion classified only 2 students as bullies.

Among the 202 students who completed the BVQ, 72% (145) received no peer nominations as bullies, 18% (37) received one peer nomination, 5% (9) received two nominations, 3% (6) received three nominations, and 2.5% (5) received four or more nominations as bullies. As shown in Table 1, the target question regarding self-reported bullying on the BVQ was significantly correlated with peer bully nominations (r = .12, p < .05) and GPA (r = −.15, p < .05), but not disciplinary violations or suspensions.

TABLE 1 Correlations Among Measures of Bullying, Grades, and Disciplinary Violations

	Self-reportedbullying	Self-reported victim	Bully nominations	Victim nominations	Discipline violations	Suspensions	GPA
Self-reported bullying	—	−.005	.118*	.016	.055	.057	−.151*
Self-reported victim		—	−.005	.422**	.101	−.060	−.122*
Bully nominations			—	.065	.359**	−.044	−.147*
Victim nominations				—	.030	−.048	−.018
Discipline violations					—	.226**	−.309**
Suspensions						—	−.072
*p < .05; **p < .01.

Comparisons of bullies and nonbullies

The group of bullies classified using the 9-item criterion had significantly higher numbers of bully nominations, t(199) = −1.80, p < .05, than nonbullies.

Using the “at least once” criterion, bullies received significantly more peer nominations than nonbullies, t(199) = −0.24, p < .05. As shown in Table 2, no other group differences were statistically significant. Bullies also had lower GPAs, t(199) = 2.33, p < .05. Because of the low number of students (n = 2) identified as bullies by the Olweus standard criterion, this criterion was not analyzed.

TABLE 2 Group Comparisons Based on Self-Report Criteria for Bullying and Victimization

	Nominations			Discipline violations			Suspensions			GPA
	M (SD)	t	d	M (SD)	t-value	d	M (SD)	t	d	M (SD)	t	d
9-item criterion		−1.80*	.28		−1.48	.24		.46	−.09		2.33*	−.41
Bully (n = 49)	0.82 (1.62)			0.66 (1.48)			0.02 (.15)			2.91 (.66)
Nonbully (n = 150)	0.43 (1.17)			0.34 (1.18)			0.04 (.29)			3.20 (.76)
“At least once” criterion		−0.24**	.39		−.78	.17		−.18	.04		1.58	−.37
Bully (n = 22)	1.14 (2.21)			0.62 (1.43)			0.05 (.22)			2.89 (.73)
Nonbully (n = 177)	0.45 (1.13)			0.39 (1.24)			0.04 (.27)			3.16 (.74)
9-item criterion		–4.01**	.55		−.84	.12		1.36	−.19		1.27	−.18
Victim (n = 96)	0.59 (1.34)			.56 (1.61)			.01 (.11)			3.04 (.79)
Nonvictim (n = 104)	0.06 (.23)			.38 (1.27)			.06 (.35)			3.18 (.73)
“At least once” criterion		–4.74**	.59		−.28	.05		.99	−.24		.92	−.14
Victim (n = 39)	0.95 (1.81)			.53 (1.70)			0 (0)			3.02 (.94)
Nonvictim (n = 161)	0.16 (.55)			.45 (1.37)			.05 (.30)			3.14 (.71)
Olweus standard criterion		–6.00**	.76		–2.18*	.41		.51	−.21		1.95*	−.50
Victim (n = 13)	1.77 (2.83)			1.33 (2.87)			0 (0)			2.71 (1.0)
Nonvictim (n = 187)	0.21 (.59)			.41 (1.28)			.04 (.27)			3.15 (.74)
*p < .05; **p < .01.

Correspondence of self- and peer-reported bullying

Using the “at least once” criterion, agreement between self report and peer nominations for bullying was 69%, with a kappa coefficient of .06, p > .05. Using the Olweus standard criterion, agreement increased to 89%, but the kappa coefficient of –.02 was not statistically significant (see Table 3 for results).

TABLE 3 Self-Report and Peer Nominations for Bullying and Victimization

Self-reportHow often have you taken part in bullying another student(s) at school the past couple of months?	Bully nominations
	0	1	2+	Total
It hasn't happened	129	34	14	177
Only once or twice	12	3	5	20
2 or 3 times a month	1	0	0	1
About once a week	1	0	0	1
Several times a week	0	0	0	0
Total	143	37	19	199
Self-reportHow often have you been bullied at school in the past couple of months?	Victim nominations
	0	1	2+	Total
It hasn't happened	142	15	4	161
Only once or twice	15	9	2	26
2 or 3 times a month	4	3	1	8
About once a week	0	1	0	1
Several times a week	0	3	1	4
Total	161	31	8	200

Victims

With regard to victims of bullying, more than three-fourths of the students (80%, n = 161) reported not being bullied by others in the past couple of months, 13% (26) reported being bullied “once or twice,” 4% (8) reported being bullied “two or three times a month,” 0.5% (1) reported being bullied “about once a week,” and 2% (4) reported being bullied “several times a week.” The 9-item criterion classified 96 students as victims, the “at least once” criterion classified 39 students, and the standard cut-off classified 13 students.

The majority (81%, n = 163) of students received no peer nominations as victims, 15% (31) received one peer nomination, 2% (4) received two nominations, 0.5% (1) received three nominations, and 1.5% (3) received five or more nominations as victims. Self-reported victimization on the BVQ was significantly correlated with peer victim nominations (r = .42, p < .01) and GPA (r = –.12, p < .01).

Comparisons of victims and nonvictims

Based on the 9-item criterion, the only significant difference was that self-reported victims had more peer nominations than nonvictims, t(200) = −4.01, p < .01. After applying the “at least once” criterion, victims continued to have significantly more peer victim nominations, t(200) = −4.74, p < .01.

Results from the Olweus standard cut-off group indicated that victims had significantly more peer nominations, t(200) = −6.0, p < .01, than nonvictims. Victims also had more discipline violations, t(200) = −2.18, p < .05, and lower GPAs, t(200) = 1.95, p < .05.

Correspondence of self- and peer-reported victimization

Using the “at least once” criterion, agreement was 81%, with a kappa coefficient of .40, p < .05. Using the Olweus standard criterion, agreement between self-report and peer nominations for victims was 92%, with a kappa coefficient of .15, p < .05 (see Table 2 for results).

The surveys were re-administered the following spring using the same instructions and administration procedures, so that the school could examine changes over time. For purposes of the present study, parallel analyses were conducted to see if there were any differences in the pattern of findings obtained in the fall. The spring analyses were generally similar to findings from the fall analyses. There were comparable numbers of bullies identified using the 9-item criterion (n = 51), “at least once” criterion (n = 21), and Olweus standard criterion (n = 5). Self-reported bullies had significantly more discipline referrals and suspensions than nonbullies only when using the “9-item” criterion.

The spring data also showed similar counts of self-reported victims using the 9-item criterion (n = 94), “at least once” criterion (n = 32), and Olweus standard criterion (n = 13). Self-reported victims had significantly more peer nominations across all three cutoff groups, and more discipline referrals and lower grades than nonvictims when the standard cutoff was applied. Detailed results are available from the authors.

DISCUSSION

Overall, there was only modest correspondence between students who identified themselves as bullies on the BVQ and peer nominations for bullying (r = .12, p < .05) and academic grades (r = −.15, p < .05), and no relationship with discipline referrals. Although the results of this study are based on the BVQ, they raise a more general concern about the use of student self-report to measure the prevalence of bullying. Cornell and Brockenbrough (2004) compared the same peer nomination form with a different self-report bullying measure and found similar results.

There was a moderately sized correlation of .42 between self-reported victimization on the BVQ and victim peer nominations. This result was slightly higher than Pellegrini's (2001) correlation of .32 for a similar comparison using the mean of nine BVQ victim items. Psychometrically, a nine-item measure should be more reliable and produce a greater range of scores than a single item, and thus could produce higher validity coefficients. The potential to improve self-report measures of bullying and victimization through multiple items rather than the single items recommended by Solberg and Olweus (2003) deserves further investigation.

Victimization overall was not correlated with disciplinary infractions, but the 13 students who identified themselves as victims one or more times per week (the Olweus standard criterion) averaged more disciplinary violations than non victims. Perhaps these violations are a consequence of frustration at being bullied. Further study is needed to determine how robust these findings are across school populations and in schools using or not using the Olweus Bullying Prevention program.

An important question that often arises when using self-report measures is how to determine which cut-off is most accurate to identify bullying and victimization. The standard cut-off recommended by Solberg and Olweus (2003) seemed to be too stringent for classifying bullies in our sample. One reason may be that the school had a low base rate for bullying because their prevention efforts were successful; however, peer nominations identified substantially more students as bullies than the BVQ. The “at least once” criterion captured more (22) self-identified bullies than the two identified with the Olweus standard criterion, but was still less than half of the 57 students nominated as bullies by their peers. Although many students did not admit bullying others, they evidently engaged in behavior that was perceived as bullying by their peers.

Students appeared to be more willing to report being victims than bullies, and self-reported victimization was somewhat more highly correlated with peer nominations than self-reported bullying. The “at least once” criterion for identifying victims was most consistent with the number of peer nominations for victims; however, only 20 of the 39 students identified by self-report received at least one peer nomination for being a victim of bullying. Of course, some bullying incidents may happen in isolation and therefore are not reported by other peers.

Another reason why there was more support for self-reported victimization than bullying may be that it is easier for a student to recognize that he or she is being bullied; a bully may not recognize that his or her aggressive behavior constitutes bullying. And as students mature, they may be increasingly unwilling to admit engaging in a socially disapproved behavior like bullying (Swearer & Cary, 2003). It may be useful to test student knowledge of bullying and whether they can distinguish bullying from other forms of peer conflict and aggression. Without follow-up interviews and further investigation, however, it is not possible to determine the accuracy of self or peer reports.

There are several limitations to this study. First, the study was based on a sample with relatively few numbers of bullies and victims. A larger sample might provide more illuminating results, particularly with regard to self-reported bullying. In addition, results of the study are based on a convenience sample from one middle school, which limits the generalizability of its conclusions to other schools. It would be useful to conduct a similar study in a series of schools, including schools without bullying prevention programs. However, the schools most likely to use the BVQ are those that are implementing a bullying prevention program. Moreover, the implementation of anti-bullying programs in schools is becoming more commonplace. Schools across the United States and throughout the world are using bullying prevention programs, so that the assessment of survey validity in such schools is increasingly relevant and important. The present study may stimulate further much-needed validity research by demonstrating that it is possible to administer the BVQ on a confidential basis and examine its relationships with external criteria.

Another limitation is that there is no true gold standard for assessing the accuracy of a student's report about bullying others or being a victim of bullying. Peer nomination surveys are well-established for measuring peer aggression, but are nevertheless vulnerable to biases in student perceptions and their understanding of bullying. The use of school discipline records provides another point of comparison, but discipline violations reflect behaviors other than bullying and can only be regarded as confirmation that the student engages in rule-breaking or misbehavior at school.

Lastly, this study administered the BVQ on a confidential basis using code numbers to identify students, which differs from the usual practice of administering the survey anonymously. One could argue that a confidential administration of the BVQ using code numbers makes it difficult to conclude that one would obtain similar results using the recommended anonymous administration. A dilemma for bullying research is that anonymous administration makes it impossible to examine the concurrent validity of the BVQ using independent criteria. However, at least three studies of other self-report surveys found that confidential administration did not result in different prevalence rates than anonymous administration (Chan et al., 2005; O'Malley et al., 2000; van de Looij-Jansen et al., 2006).

There are practical as well as methodological advantages to using a confidential survey. Confidential administration of the BVQ allows schools with anti-bullying prevention programs to conduct a more informative evaluation of their success in reducing rates of bullying. Schools could compare survey results to other sources of information on bullying, such as counselor and teacher observations and records of known bullying. In practice, the determination that bullying is taking place is made by a school counselor or administrator who interviews the students and makes a judgment, but to our knowledge there are no studies of this process.

In conclusion, this study points out the paucity of evidence regarding the validity of self-reported bullying and victimization. This study obtained evidence of modest correspondence between self reports and peer reports, discipline referrals, and academic grades, but further work is needed to demonstrate the accuracy of self-report. Researchers and practitioners are advised that given the current knowledge of the psychometric properties of bullying self-report measures, educators should not rely solely on them to assess campus needs or to monitor the outcomes of their prevention/intervention efforts. Educators interested in examining the level of bullying taking place in their school should consider a variety of procedures that will give them a broader perspective, including peer nominations, classroom discussions, and analysis of disciplinary referrals. We recommend further study of the BVQ using confidential as opposed to anonymous administration, so that critical questions regarding the validity of this widely used self-report instrument can be answered.

Acknowledgments

We thank the staff and students at the participating middle school.

Notes

1. An inquiry to Dr. Olweus in 2007 yielded a similar reply: “We have made a lot of analyses of the psychometric properties of the Questionnaire in addition to those specified in the 2003 Aggressive Behavior paper but relatively little of that has been published. We have been so much involved in the large-scale, government-supported intervention project … that we have simply not had the time to publish but a small portion of our finding …” (D. Olweus, personal communication, March 16, 2007).