Development and psychometric evaluation of an instrument measuring subjective definitions of inclusion (FEDI)

ABSTRACT

Within educational and social discourses, the term ‘inclusion’ has various meanings. In both research and practice, there is no official definition of inclusion. Nevertheless, subjective definitions affect pedagogical acts. We developed the Definitions of Inclusion Questionnaire [Fragebogen zur Erfassung subjektiver Definitionen von Inklusion] (FEDI) as an economic instrument that takes subjective definitions of inclusion into account. This paper describes the construction and first psychometric evaluation of the questionnaire instrument that 513 participants with various professional backgrounds completed. We used exploratory and confirmatory factor analyses of two different subsamples to identify the measuring model. We found a three-factor structure with acceptable-to-good fit measures and an acceptable reliability (CR from .81 to .87). Small to medium correlations of the FEDI scales with attitudes towards inclusion and teachers’ sense of efficacy suggest that discriminant validity is given. Perspectives for further research and implications for practice are discussed.

KEYWORDS:

• Research instrument
• subjective definition
• factor analysis
• questionnaire
• psychometric evaluation

Introduction

As Grosche (2015) and Piezunka, Schaffus, and Grosche (2017) argue, no distinct, accurate, and overall valid definition of inclusion (or inclusive education) exists, especially in the German context, which leads to problems for practitioners and researchers when planning, coordinating, and discussing inclusion-oriented processes (see also Werning 2010). Furthermore, discussing inclusion from different perspectives of understanding without acknowledging the differences may lead to the depreciation of others’ views (Grosche 2017a, 2017b). Although numerous papers have tried to define inclusion systematically (Ainscow, Booth, and Dyson 2006; Farrell et al. 2007; Fuchs and Fuchs 1998; Göransson and Nilholm 2014; Kavale and Forness 2000; Loreman 2014), definitions remain diverse.

Data from inclusion research containing descriptions of, attitudes towards, or beliefs about inclusion may be influenced by implicit or subjective definitions. Lüke and Grosche (2018c) showed that participants’ attitudes towards inclusion are highly influenced by the perceived attitudes of the surveying organisation. Although this relates to social desirability, it indicates that anticipated, but not explicitly given, definitions of inclusion may bias findings.

‘Subjective’ definitions refer to subjective theories that Mandl and Huber (1983, 98) define as a ‘system of cognitions that in its structure is seen as comparable to a scientific theory’ on a specific topic. Until now, the subjective definitions of inclusion have not been considered sufficiently in research (Przibilla, Linderkamp, and Krämer 2018).

Several studies have used qualitative approaches to gather data about subjective inclusion definitions. Makoelle (2014) identified three ways to define inclusion, and Przibilla, Linderkamp, and Krämer (2018) identified 27 categories for teachers’ subjective definitions of inclusion and arranged them into nine dimensions.

Another qualitative study by Piezunka, Schaffus, and Grosche (2017) focussed on definitions from expert interviews with researchers in the field of inclusion. The interviews first asked, ‘What does inclusion mean to you?’ This was followed by questions covering goals of inclusion, forms of legitimation of inclusion, distinction from integration, inclusion target group(s), inclusion feasibility, inclusion practice, and a one-sentence summary of inclusion.

These dimensions served as categories for qualitative content analysis, from which Piezunka, Schaffus, and Grosche (2017) identified four distinct definitions: (1) ‘inclusion as implementing the UN-CRPD’, (2) ‘pragmatic definition with a focus on supporting students’ individual academic progress’, (3) ‘participation/acknowledgement/well-being’, and (4) ‘inclusion as a utopia’. While these four definitions are distinct from each other, they share a consensual core (inclusion as overcoming discrimination), ‘which makes a hierarchical development within single definitions’ (Piezunka, Schaffus, and Grosche 2017, 216; authors’ translation). Definition (1) is the least complex, mostly referring to the placement of people with disabilities, and definition (4) is the most complex, describing people as individuals who cannot be divided into separate groups. These findings resemble the four definitions that Göransson and Nilholm (2014) found in their review of international literature (see Table 1), which they later replicated (2017). However, detailed examination finds differences between Piezunka, Schaffus, and Grosche (2017) and Göransson and Nilholm (2014). While both studies identify placement as the least complex and least comprehensive definition, the pragmatic definition of the former includes the meaning of both the specified and general individualised definitions of Göransson and Nilholm (2014). In contrast, the community definition of Göransson and Nilholm (2014) includes both the ‘participation/acknowledgement/well-being’ and ‘inclusion as a utopia’ definitions from Piezunka, Schaffus, and Grosche (2017).

Table 1. Comparison of the empirical findings from Piezunka, Schaffus, and Grosche (2017) and the theoretical review from Göransson and Nilholm (2014).

Piezunka, Schaffus, and Grosche (2017)	Göransson and Nilholm (2014)
(1) ‘inclusion as implementing the UN-CRPD’	(A) Placement definition – inclusion as placement of pupils with disabilities/in need of special support in general education classrooms
(2) ‘pragmatic definition with a focus on supporting students’ individual academic progress’	(B) Specified individualised definition – inclusion as meeting the social/academic needs of pupils with disabilities/pupils in need of special support
	(C) General individualised definition – inclusion as meeting the social/academic needs of all pupils
(3) ‘participation/acknowledgement/well-being’	(D) Community definition – inclusion as creation of communities with specific characteristics (which could vary between proposals)
(4) ‘inclusion as a utopia’

There is additional research from Krischler, Powell, and Pit-Ten Cate (2019) that references the framework from Göransson and Nilholm: They found that participants’ definitions corresponded with their attitudes towards inclusion. An open question asked participants to define inclusion and the answers were categorised by using the first three definitions of Göransson and Nilholm. Furthermore, participants filled out a quantitative measurement questionnaire concerning attitudes towards inclusion. Differences in attitudes across the three groups (placement definition, specified individualised definition, and general individualised definition) were analysed using ANOVA that ‘indicated differences in attitudes for people providing different definitions of inclusion’ (Krischler, Powell, and Pit-Ten Cate 2019, 639):

More specifically, people defining inclusion according to the ‘General individualised’ definition held significantly more positive attitudes than the people defining inclusion

according to the ‘Specified individualised’ definition […] and to the ‘Placement’ definition, […]. Additionally, people defining inclusion according to the ‘Specified individualised’ definition held significantly more positive attitudes than the people according to the ‘Placement’ definition […]. (Krischler, Powell, and Pit-Ten Cate 2019, 639)

Thus, the findings from Krischler, Powell, and Pit-Ten Cate (2019) imply that we must consider subjective definitions of inclusion when analysing how attitudes towards inclusion work and develop.

Scheer (2020) conducted qualitative interviews with school principals about their leadership roles for inclusive education. The results from qualitative content analysis indicated that principals who provided different definitions of inclusion also provided different reasoning for their measures to foster inclusive school development. Although the framework from Piezunka, Schaffus, and Grosche (2017) was used in this study, the ‘pragmatic definition’ had to be split into two categories (‘focused on various lines of difference’ vs. ‘focused on special educational needs’). This reflects the ‘general individualised’ vs. ‘specified individualised’ definitions from Göransson and Nilholm (2014). Beside the finding that, for actual practice, subjective definitions of inclusion seem to be powerful, this also indicates that the distinction between a ‘wide’ understanding of inclusion (‘focused on various lines of difference’ or ‘general individualised’) and a ‘narrow’ understanding of inclusion (‘focused on special educational needs’ or ‘specified individualised’) must be considered as a category of its own when developing a questionnaire to measure subjective definitions of inclusion.

In qualitative interviews with school principals, Graham and Spandagou (2011) asked the participants about their approaches to inclusive education. From their analysis, Graham and Spandagou elaborate that ‘the more culturally diverse the school, the more expansive the view of inclusive education’ (Graham and Spandagou 2011, 227). One of the main findings was that the participants’ conceptualisation of inclusion found its expression in the distinction between ‘including them’ and ‘being inclusive’. Overall, the authors conclude that ‘school principals’ attitudes towards inclusive education and their success in engineering inclusive practices within their school are significantly affected by their own conception of what “inclusion” and “being inclusive” mean’ (Graham and Spandagou 2011, 233). In another study, Salisbury (2006) combined measures of schools’ inclusiveness (for example, percentage of students educated outside the general education classroom) with qualitative interviews conducted with the schools’ principals:

In our sample, the views and commitment to inclusive education appeared to affect the decisions rendered by principals as they guided the development of their school’s service de-livery model. Several principals in our sample chose to view inclusive education as an agenda for reform, whereas others saw it as an exercise in compliance with LRE [least restrictive environment] provisions. (Salisbury 2006, 79–80)

Furthermore, they concluded that the ‘absence of a consensus definition of inclusive education in the field makes comparisons across studies difficult’ (Salisbury 2006, 81). From our point of view, this implies the need for integrating measures of participants’ conceptualisations of inclusion within empirical studies on inclusive education.

Starczewska, Hodkinson, and Adams (2012) concluded from interviews with teachers about their conceptualisation of inclusion and integration that inclusion seems to be ‘employed at the level of policy rhetoric […] [and] one of those educational buzzwords that […] says everything but says nothing’ (Starczewska, Hodkinson, and Adams 2012, 168). They found that teachers, when asked about inclusion, experienced a terminological ambiguity and struggled to provide a substantive definition. However, when asked about integration, the participants’ answers included more concrete measures. Thus, we can conclude that teachers should be provided with example statements so that they can evaluate how these statements reflect their understanding of inclusion. For research on inclusive education, this justifies the use of a rating scale questionnaire like the instrument described in our paper.

While the studies described in this section provide in-depth insights into subjective inclusion definitions and their importance, there is a need for an economic quantitative instrument that can capture differences in subjective definitions. This paper introduces the first version of the Definitions of Inclusion Questionnaire [Fragebogen zur Erfassung subjektiver Definitionen von Inklusion] FEDI (Egener et al. 2019; Scheer et al. 2020) and its psychometric properties. Initially, this questionnaire was developed to be independent of the personal and professional backgrounds of the respondents and to be used in a large variety of contexts (not only education). However, we see FEDI as an important tool within the context of inclusive school development research and in research on teacher education for inclusion. In these two contexts, the questionnaire can help evaluate if and how subjective understandings of inclusion affect teachers’, student teachers’, and principals’ attitudes and self-efficacy with respect to inclusive education, and provide actual measures for developing inclusive schools, which can be assumed according to the findings of Graham and Spandagou (2011), Krischler, Powell, and Pit-Ten Cate (2019), Salisbury (2006), Scheer (2020), and Starczewska, Hodkinson, and Adams (2012).

Research questions

This study provides the first psychometric evaluation of the FEDI, which aims to fill the persistent gap in research regarding subjective definitions of inclusion.

Since subjective definitions of inclusion should comprise several dimensions, we assume that a valid measuring instrument should have a multifactorial structure. Thus, our first question considers:

Q1: Does a multifactorial model of subjective definitions of inclusion show a better fit than a unifactorial model?

Although Varimax-rotation forces the factors themselves to be uncorrelated, one might expect small correlations (r < .2) between the final FEDI-subscales. Inclusion, especially inclusive education, is regulated via legislation, and legislation often pertains to human rights and ethical reasoning, which leads to interactions between different aspects of subjective definitions. However, since the factorial structure is evaluated during data analysis, we cannot predict the amount of intercorrelations. Therefore, we ask:

Q2: How much do the FEDI-subscales intercorrelate?

Furthermore, a valid measurement of subjective definitions of inclusion should capture descriptive cognitions rather than evaluative beliefs and attitudes. However, some interference may occur. For instance, a prior study from Krischler, Powell, and Pit-Ten Cate (2019) showed a medium main effect of participants’ definitions of inclusion on their attitudes. Furthermore, attitudes towards inclusion are associated with self-efficacy (Scheer et al. 2015; Urton, Wilbert, and Hennemann 2014; Savolainen, Malinen, and Schwab 2020). Thus, for evaluating discriminant validity, we suggest that there are small to medium correlations between FEDI scores and attitudes towards inclusion and teachers’ self-efficacy. Therefore, our third and fourth research questions are:

Q3: How do the FEDI-subscales correlate with attitudes towards inclusion?

Q4: How do the FEDI-subscales correlate with teachers’ self-efficacy?

Materials and methods

Sample and procedures

Using internal mailing lists from several German universities, we emailed an invitation to researchers, lecturers, and students from all faculties with a hyperlink to our questionnaire on LimeSurvey. Additionally, practitioners in the field of inclusive and special education recruited participants with a school-related perspective (QR code on handouts and posters). A total of 513 persons (417 female, 93 male, 3 diverse) participated. Many were university students (n = 426), of whom most were pursuing a teaching degree (n = 399). From those, the majority were studying special education (n = 179). See Tables 2 and 3 for further details.

Table 2. Participants by professional background.

	Total Sample					Subsample I (CEFA)					Subsample II (CCFA)
	Female	Male	Diverse	Total		Female	Male	Diverse	Total		Female	Male	Diverse	Total
Deputy School Principal	0	2	0	2		0	1	0	1		0	1	0	1
District School Authorities	0	1	1	2		0	0	1	1		0	1	0	1
Others	2	1	0	3		1	1	0	2		1	0	0	1
Pre-Service Teacher	10	2	0	12		5	1	0	6		5	1	0	6
Research Associate	39	11	1	51		17	8	0	25		22	3	1	26
School Principal	1	0	0	1		1	0	0	1		0	0	0	0
Teacher	3	1	0	4		1	1	0	2		2	0	0	2
Teacher Assistant	2	0	0	2		1	0	0	1		1	0	0	1
University Professor	6	4	0	10		3	2	0	5		3	2	0	5
University Student	354	71	1	426		179	34	0	213		175	37	1	213
Total	417	93	3	513		208	48	1	257		209	45	2	256

Table 3. Students from teaching degree programs by type of teaching degree.

	Female	Male	Diverse	Total
Combined: Primary and Secondary Level I Education	3	0	0	3
Primary Education	63	4	0	67
Secondary Level I Education	58	20	0	78
Secondary Level II Education	48	13	0	61
Special Needs Education	150	28	1	179
Vocational Education	7	4	0	11
Total	329	69	1	399

Measures

Subjective definitions of inclusion questionnaire (FEDI)

Item pool

To construct our instrument, we used the three most general and descriptive dimensions of inclusion that Piezunka, Schaffus, and Grosche (2017) used for their interview guidelines and formulated possible manifestations for each. These guidelines also reflect international research (Krischler, Powell, and Pit-Ten Cate 2019; Nilholm and Göransson 2017; Waitoller and Artiles 2013). We formulated the following dimensions:

Goals and aims. These are the goals and aims that a person correlates with the term inclusion, such as ‘anti-discrimination’ (prefix for items: ZA for ‘Ziele: Antidiskriminierung’), ‘effective (educational) support’ (prefix: ZF for ‘Ziele: Förderung’), ‘access to regular schools’ (prefix: ZR for ‘Ziele: Regelschule’), and ‘social and political participation’ (prefix: ZT for ‘Ziele: Teilhabe’).

Legitimation of inclusion. What a person believes provides legitimacy to inclusion, including ‘ethics’ (prefix: LE for ‘Legitimation: Ethisch’), ‘outcomes/students’ achievements’ (prefix: LO for ‘Legitimation: Outcomes’) and ‘legal’ (prefix: LR for ‘Legitimation: Rechtlich’).

Lines of difference. These are the lines of difference addressed by the term inclusion. For example, a person may feel that the term inclusion addresses ‘disability’ (prefix: DLB for ‘Differenzlinie: Behinderung’), ‘several lines of difference and/or intersectionality’ (prefix: DLU for ‘Differenzlinie: Unterschiedliche’), or ‘overcoming social construction of categories’ (prefix: DLD for ‘Differenzlinie: Dekategorisierung’).

Using the resulting ten theoretical aspects, we formulated 35 descriptive, objective statements. These statements comprise the pool of Likert-like rating items for the questionnaire. We applied a five-point rating scale with full verbalisation to the items (0 = not true, 1 = somewhat not true, 2 = partially true, 3 = rather true, 4 = true).

Pilot version

We administered the first draft to 10 university students and 20 field experts. They were asked to answer the questionnaire and give written feedback on all and/or on single items.

First revision

We revised the questionnaire based on the pilot feedback and response patterns. This version contained 39 items (see Table 4; the German items and English translation are presented in the electronic supplementary material, ESM-1.docx).

Table 4. The complete set of items generated for the FEDI linked to their underlying dimensions.

Dimension (based on Piezunka, Schaffus, and Grosche 2017)	Manifestation	Code	Item
Lines of difference	Disability	DLB1	Inclusion means that students with and without disabilities learn together.
		DLB2	Inclusion means striving to strengthen the rights of people with disabilities.
	Overcoming social construction of categories	DLD1	Inclusion means that people can no longer be divided into categories.
		DLD2	Inclusion means that the diversity of all people cannot be expressed in categories.
	Several lines of difference and/or intersectionality	DLU1	Inclusion means that different forms of diversity (disability, gender, ethnicity, language, religion etc.) are considered.
		DLU2	Inclusion means that all people, regardless of gender, origin, religion, or ability, have the same value and the same rights.
		DLU3	Inclusion means that diversity is normal.
Legitimation of inclusion	Ethics	LE1	Inclusion focuses on the concept of human dignity.
		LE2	Inclusion focuses on the idea of a fairer society.
		LE3	Inclusion focuses on the idea of universal human rights.
		LE4	Inclusion focuses on the idea of equality for all people.
	Outcomes / students’ achievement	LO1	Inclusion focuses on the cognitive development of the students.
		LO2	Inclusion focuses on the learning development of the students.
		LO3	Inclusion focuses on the social and emotional development of the students.
		LO4	Inclusion focuses on the motor development of the students.
		LO5	Inclusion focuses on the sensory development of the students.
	Legal	LR1	Inclusion means implementing the requirements of the UN Convention on the Rights of Persons with Disabilities.
		LR2	Inclusion is a legal requirement that must be met.
		LR3	Inclusion means implementing the legal right to participation and non-discrimination.
		LR4	Inclusion means implementing the legal right to education in the general school system.
Goals & aims of inclusion	Anti-discrimination	ZA1	The aim of inclusion is to reduce discrimination.
		ZA2	The aim of inclusion is that no one is excluded.
		ZA3	The aim of inclusion is that all people are treated equally
		ZA4	The aim of inclusion is that all people have equal rights.
	Effective (educational) support	ZF1	The aim of inclusion is that children receive optimal support based on their individual skills.
		ZF2	The aim of inclusion is that every child can learn at their own pace.
		ZF3	The aim of inclusion is to tailor teaching to the needs of each child.
	Access to regular schools	ZR1	In an inclusive school system, every child / adolescent has access to a general school.
		ZR2	In an inclusive school system, as few of the children with disabilities as possible attend a special school.
		ZR3	There are no special schools in an inclusive school system.
		ZR4	In an inclusive school system, all children and adolescents attend school in their neighbourhood.
		ZR5	An inclusive school system is an integrated school system.
		ZR6	An inclusive school system consists of general schools with specialised support programs.
		ZR7	In an inclusive school system, every child / adolescent has access to every general school.
	Social and political participation	ZT1	Inclusion means being part of the community.
		ZT2	Inclusion means that everyone has a say.
		ZT3	Inclusion means participating in social life.
		ZT4	Inclusion means empowering people to stand up for their rights.
		ZT5	Inclusion means being recognised as a human being.

Final version

During analysis, the questionnaire was reduced to the final 15-item version (see subsection ‘Data analysis’ for details). For this, we first eliminated all items with a relative information content H < .75. Then after a first run of factor analysis, we incorporated the five items per factor with the highest loadings into the final questionnaire (items listed in Table 5). This version is licensed under CC-BY 4.0 (Egener et al. 2019; for an English version, see Scheer et al. 2020).

Table 5. Properties of the FEDI items linked to their FEDI scales.

Item		Scale	Response Category (frequency in %)							λCEFA
			0	1	2	3	4	Mdn.	IQR	SSB	HEB	OAB	h^2CEFA	λCCFA	rit
ZR1	In an inclusive school system, every child / adolescent has access to a general school.	SSB	2.14	3.31	20.08	29.82	44.64	3	2	.705	.208	−.083	.546	.827	.722
ZR7	In an inclusive school system, every child / adolescent has access to every general school.	SSB	3.70	10.14	21.25	31.19	33.72	3	2	.762	.139	.063	.604	.713	.743
ZR3	There are no special schools in an inclusive school system.	SSB	12.67	21.64	30.02	21.25	14.42	2	2	.687	−.043	.081	.481	.516	.727
DLD2	Inclusion means that the diversity of all people cannot be expressed in categories.	SSB	2.34	7.99	16.76	33.72	39.18	3	2	.417	.367	.080	.315	.517	.558
ZR5	An inclusive school system is an integrated school system.	SSB	12.48	19.49	37.23	18.32	12.48	2	2	.529	.180	−.061	.316	.415	.638
LE2	Inclusion focuses on the idea of a fairer society.	HEB	1.56	3.70	16.96	40.94	36.84	3	1	.150	.650	.171	.474	.805	.723
ZT4	Inclusion means empowering people to stand up for their rights.	HEB	3.31	13.65	27.10	32.75	23.20	3	1	.051	.557	.132	.330	.737	.713
LE3	Inclusion focuses on the idea of universal human rights.	HEB	1.75	4.29	21.83	32.75	39.38	3	2	.146	.702	.099	.525	.722	.728
DLB2	Inclusion means striving to strengthen the rights of people with disabilities.	HEB	1.36	4.87	30.02	36.65	27.10	3	2	.045	.517	.179	.301	.534	.627
LR1	Inclusion means implementing the requirements of the UN Convention on the Rights of Persons with Disabilities.	HEB	1.95	7.02	28.27	31.97	30.80	3	2	.294	.508	−.016	.344	.409	.601
LO5	Inclusion focuses on the sensory development of the students.	OAB	9.75	26.9	46.98	13.26	3.12	2	1	.037	.023	.810	.658	.862	.809
LO4	Inclusion focuses on the motor development of the students.	OAB	13.84	31.19	43.66	8.19	3.12	2	1	.040	.104	.772	.609	.716	.764
LO1	Inclusion focuses on the cognitive development of the students.	OAB	6.82	20.86	46.39	19.69	6.24	2	2	−.013	.101	.725	.537	.719	.779
LO3	Inclusion focuses on the social and emotional development of the students.	OAB	5.07	14.42	41.72	29.24	9.55	2	1	−.055	.135	.520	.291	.624	.699
LO2	Inclusion focuses on the learning development of the students.	OAB	3.12	10.33	37.23	29.24	20.08	2	1	.060	.248	.457	.274	.573	.646

Note. SSB = school system-based perspective, HEB = human rights/ethics-based perspective, OAB = outcomes/achievement-based perspective. 0 = ‘not true’, 1 = ‘somewhat not true’, 2 = ‘partially true’, 3 = ‘rather true’, 4 = ‘true’. Mdn. = median, IQR = interquartile range, λ_CEFA = loadings from 2nd run CEFA, h²_CEFA = communalities from 2nd run CEFA, λ_CCFA = loadings from CCFA, r_it = corrected item-total-correlation.

Attitudes towards inclusive education

Attitudes towards inclusive education were measured using the ‘Professionsunabhängige Einstellungsskala zum inklusiven Schulsystem’ questionnaire (PREIS; ‘profession-independent attitudes towards inclusive education scale’) (Lüke and Grosche 2016, 2018b; Schulze et al. 2019), which was developed to measure attitudes independently from the participants’ professional backgrounds and is based on the model introduced by Eagly and Chaiken (1993). Retest reliability, r_tt = .96, (Lüke and Grosche 2018b) and internal consistency, $\alpha$ = .93, were both high (Lüke and Grosche 2018b; Schulze et al. 2019).

Teachers’ Sense of Efficacy Scale (TSES)

The TSES (Tschannen-Moran and Woolfolk-Hoy 2001) was administered to participants who had a school-related professional background (n = 420) using a German translation by Sung and Melzer (2014).

Data analysis

All data analysis occurred in the R language (R Core Team 2019) using the RStudio environment (RStudio 2019). See ESM-3.docx electronic supplementary material for the full R input. First, we evaluated the descriptive statistics of all FEDI items. Due to the ordinal level of raw data, we used category frequencies (0 = not true, 1 = somewhat not true, 2 = partially true, 3 = rather true, 4 = true), median (Mdn.), and inter-quartile range (IQR) as descriptive measures. Item statistics were computed using the descript function from the ltm-package in R (Rizopoulos 2018). Additionally, we analysed the items’ relative information content H using equation 1(1) $H_i=-{\displaystyle \frac{1}{\text{ln}{k}_i}\cdot \sum _{j=1}^{k_i}{\pi }_{ij}\cdot \text{ln}{\pi }_{ij}}$(1) as provided by Eid and Schmidt (2014), where ${\pi }_{ij}$ is the relative frequency (probability) of category j from item i, and $k_i$ is the number of categories of item i. This can be expressed as follows: (1) $H_i=-{\displaystyle \frac{1}{\text{ln}{k}_i}\cdot \sum _{j=1}^{k_i}{\pi }_{ij}\cdot \text{ln}{\pi }_{ij}}$(1) Items with H < .75 were eliminated from the questionnaire before further analysis.

Next, we split the original sample (see ESM-2.csv electronic supplementary material for the full dataset) to minimise any bias due to repeated testing. To generate two randomised but comparable subsamples, we used stratified randomisation.

In the first subsample, we conducted a categorical exploratory factor analysis (CEFA) with Varimax-rotation using the function fa from the psych-package in R (Revelle 2019). We checked the Kaiser-Meyer-Olkin and the items’ MSA based on the polychoric correlation matrix using the KMO and polychoric functions from the psych-package. To determine the number of factors to be retained, we considered Cattell’s scree test criterion (Cattell 1966), Kaiser’s criterion (Kaiser 1960) as subjective visual methods, and Velicer’s minimum average partial (MAP) criterion (Velicer 1976) and parallel analysis (Horn 1965) as objective statistical methods using the nfactors and fa.parallel functions from the psych-package. Loadings from >.4 were accepted (Field 2017). We then chose proper names for the factors according to the items’ content.

The CEFA factor solution was cross validated in subsample II using a categorical confirmatory factor analysis (CCFA) based on the polychoric correlation matrix. The model was specified and tested with the cfa-function from the package lavaan (Rosseel 2012, 2018) in R; for visualisation, we used the lavaanPlot-package (Lishinski 2018). We used DWLS as the estimator with NLMINB as the optimisation method. As fit measures, we evaluated ${\chi }^2$/df-ratio, RMSEA, SRMR, CFI, and TLI with the cut-off-values provided by Moosbrugger and Schermelleh-Engel (2012), Schermelleh-Engel, Moosbrugger, and Müller (2003), Hu and Bentler (1999), and Browne and Cudeck (1992). An LRT test for nested models (Satorra 2000) was applied to compare the one-factor and multi-factor models, using the lavTestLRT-function from lavaan.

Measurement invariance (MIV) across university students pursuing different teaching degrees was checked using the measEq.syntax-function from the package semTools (Jorgensen et al. 2019), which uses the lavaan functions. According to Svetina, Rutkowski, and Rutkowski (2020), we used theta-parametrisation, and the guidelines suggested by Wu and Estabrook (2016). According to Chen (2007), MIV is given if RMSEA increases less than .015, and CFI simultaneously decreases less than .01. Differential item functioning (DIF) was evaluated by applying a partial credit model (PCM; Masters 1982). We used the grm-function from the R-package pairwise to calculate the PCM (Heine 2020).

Since Cronbach’s $\alpha$ underestimates true reliability if factor loadings are not equal between indicators (Cho 2016), we also used composite reliability (Netemeyer, Bearden, and Sharma 2003) as shown: (2) $\text{CR}={\displaystyle \frac{{({\sum }_{i=1}^p{\lambda }_i)}^2}{{({\sum }_{i=1}^p{\lambda }_i)}^2+{\sum }_{i=1}^{p}V\left({\delta }_i\right)}}$(2) wherein ${\lambda }_i$ is the completely standardised loading for indicator i, V(${\delta }_i$) is the variance of the error term for indicator i, and p is the number of indicators.

Results

After a first round of analysis of the descriptive item statistics (see electronic supplement material ESM-4.docx for the descriptive statistics of all original items), items with H < .75 (13 items) were eliminated.

For the following round of analyses, we divided the total sample into two randomised and stratified subsamples as described above (see Table 2).

Categorical exploratory factor analysis (subsample I)

With the remaining 26 items, we ran a CEFA with Varimax-rotation based on the polychoric correlation matrix, using Subsample I (see Table 2). Kaiser-Meyer-Olkin was KMO = .81, with individual items’ MSA ranging from .61 to .89 (Mdn. = .81, IQR = .08). One item had an MSA of <.7, and Bartlett’s test was significant (K² = 160.22, df = 25, p < .001).

As Figure 1 shows, the point of inflection occurs at the third factor. According to Field (2017), Cattell (1966) argued to retain the factor at the point of inflection while Thurstone argued to consider only factors to the left of the point of inflection. In our data, two or three factors could be extracted. Kaiser’s criterion, however, indicated that two factors were sufficient. Since these criteria are discussed as subjective and unreliable, either parallel analysis or Velicer’s MAP is recommended (Field 2017, 790; Bühner 2011, 322; Wood, Tataryn, and Gorsuch 1996; O’Connor 2000; Zwick and Velicer 1982, 1986). Furthermore, extracting too many factors is preferable to extracting too few (Bühner 2011, 328). However, with ordinal data, MAP might underestimate the true number of factors (Garrido, Abad, and Ponsoda 2011). In our study, the MAP criterion supported three factors while parallel analysis supported five. Since the solutions with four and five factors (see ESM-3.docx) led to results that could not be interpreted meaningfully, we retained three factors as a compromise between the visual and statistical methods.

Figure 1. Scree plot of observed data.

After running a first round of CEFA (see electronic supplementary material ESM-5.docx for loadings and communalities of all items included), we took the five items per factor showing the highest loadings and re-ran the CEFA with the resulting 15-item set. Factors 1 and 3 had one item with a loading <.5 (see Table 5). The communalities of the items ranged from .274 to .658, with seven items having a communality <.4. Factor 1 (‘school system-based perspective’, SSB) explained 14.32% of the total variance, Factor 2 (‘human-rights/ethics-based perspective’, HEB) 13.89%, and Factor 3 (‘outcomes and achievement-based perspective’, OAB) 15.82%.

Categorical confirmatory factor analysis (Subsample II)

We used subsample II (see Table 2) to run a CCFA with the model resulting from CEFA (see Figure 2). The ${\chi }^2$-Test was significant with ${\chi }_{\left(df=87\right)}^2$ = 202.092, p < .001. However, model fit was from good to acceptable with ${\chi }^2$/df-ration = 2.323, RMSEA = .072 (90%-CI = [.059, .085]), SRMR = .078, CFI = .966, TLI = .959. The item loadings ranged from .409 to .862 (Mdn. = .713), with two items showing loadings <.5 (see Table 5). According to these findings, the factorial structure from CEFA could be reproduced with an adequate fit.

Figure 2. The model specified for the CCFA with empirical standardised coefficients and covariances.

Due to the theoretical assumption that subjective definitions of inclusion are composed by the emphasis of different possible perspectives, this model should have a better fit than a unidimensional model. The unidimensional model showed a poor fit with ${\chi }^2$/df-ration = 9.499, RMSEA = .183 (90%-CI = [.171, .194]), SRMR = .149, CFI = .776, TLI = .739; our three-dimensional model had a superior fit. An LRT test for nested models (Satorra 2000) indicated that this finding was significant with $\mathrm{\Delta }{\chi }_{\left(df=3\right)}^2$ = 191.55, p < .001.

Measurement invariance (MIV) across groups (explorative)

We used an ad-hoc-sample across different professional backgrounds because the FEDI should measure subjective definitions of inclusion independently from professional background. To ensure this, MIV must be evaluated. Since many professional backgrounds were represented by few participants, we evaluated MIV across university students studying special needs education and those studying general education.

The measurement model for all university students attending a teaching degree programme (n = 329, see Table 3) showed an acceptable fit with ${\chi }^2$/df-ration = 2.832 (${\chi }_{\left(df=87\right)}^2$ = 246.342), RMSEA = .068 (90%-CI = [.058, .078]), SRMR = .070, CFI = .967, TLI = .960, which is even better than the fit within the total subsample II. According to the cut-off values suggested by Chen (2007), configural, metric, and scalar MIV across groups of students studying special needs education (n = 150) and general education (n = 179) was given (see Table 6). As Figure 3 indicates, two items from the OAB scale (‘Inclusion focuses on the learning development of the students’ [LO2] and ‘Inclusion focuses on the motor development of the students’ [LO4]) show DIF and should therefore be considered for revisions in further studies.

Figure 3. DIF-analysis of the FEDI-items.

Table 6. MIV across university students attending a teaching degree programme for special needs education and for general education.

	χ^2	df	p	RMSEA	ΔRMSEA	CFI	ΔCFI	TLI
Configural Invariance	362.25	174	<.001	.074		.964		.957
Metric Invariance	393.56	186	<.001	.075	.001	.960	.004	.955
Scalar Invariance	393.56	183	<.001	.076	.001	.960	<.001	.954

Item and scale properties

Overall, the items showed acceptable corrected item-total-correlations (see Table 5). SSB and HEB showed a slightly skewed distribution, and OAB was slightly peaked (Figure 4), but skewness and kurtosis (Table 7) were acceptable for all three scales (Hair et al. 2017). However, the Shapiro–Wilk test indicated a significant deviation from normal distribution with W = .98, p < .001 for the scale SSB, W = .98, p < .001 for the scale HEB, and W = .98, p < .001 for the scale OAB. Since the test for normality is sensitive to sample size, the tests should not be overinterpreted. Furthermore, we found acceptable results for all scales’ internal consistency, ranging from $\alpha$ = .7 to .79 (Table 7). Composite reliability calculated from the CCFA-sample was also acceptable according to Netemeyer, Bearden, and Sharma (2003), with values ranging from CR = .81 to CR = .87.

Figure 4. Histograms of relative frequencies of the FEDI scores.

Table 7. Scale properties of the three FEDI scales.

				95%-CI
	M	SD	SE	LL	UL	Min.	Max.	Skew	Kurtosis	α	CR
SSB	2.59	0.75	0.03	2.52	2.65	0.2	4	−0.34	−0.21	.70	.81
HEB	2.87	0.67	0.03	2.81	2.93	0.6	4	−0.31	−0.25	.70	.85
OAB	2.01	0.71	0.03	1.94	2.07	0.0	4	−0.08	0.72	.79	.87

Note: SSB = school system-based perspective, HEB = human rights/ethics-based perspective, OAB = outcomes/achievement-based perspective. LL = lower limit of the 95%-CI, UL = upper limit of the 95%-CI. CR = composite reliability.

To evaluate convergent and divergent validity, we analysed the intercorrelations of the FEDI scales and their correlations with attitudes towards inclusion (PREIS) and teachers’ sense of efficacy (TSES). In our data, the SSB and OAB did not correlate. The HEB showed small correlation with both other perspectives. Furthermore, the FEDI should measure subjective definitions of inclusion separated from attitudes or other evaluative cognitions. Thus, we assumed small to medium correlations would exist between FEDI scales, attitudes towards inclusions, and teachers’ sense of efficacy, especially for definitions that imply an ethical imperative. Overall, our data had small correlations between scales and medium correlations between attitudes towards inclusion and HEB (see Table 8).

Table 8. Correlations between the FEDI scales, attitudes towards inclusion, and teachers’ sense of efficacy.

Variable	M	SD	(1)	(2)	(3)	(4)
1. SSB	2.59	0.75
2. HEB	2.87	0.67	.35**
			[.27, .42]
3. OAB	2.01	0.71	.08	.32**
			[−.01, .16]	[.24, .39]
4. PREIS	2.63	0.89	.35**	.44**	.20**
			[.27, .42]	[.36, .50]	[.12, .28]
5. TSES	6.53	0.75	.17**	.24**	.14**	.33**
			[.08, .25]	[.16, .32]	[.05, .22]	[.25, .40]

Note: SSB = school system-based perspective, HEB = human rights/ethics-based perspective, OAB = outcomes/achievement-based perspective, PREIS = attitudes towards inclusion, TSES = teachers sense of efficacy. Values in square brackets indicate the 95% confidence interval for each correlation. * indicates p < .05. ** indicates p < .01.

Discussion

CEFA and CCFA provide evidence that subjective definitions of inclusion are multi-dimensional constructs (Q1). This confirms our belief that these subjective definitions are influenced by a variety of individual factors and sub-definitions.

Furthermore, we showed MIV across different groups of university students. This is an early indication that the FEDI measures subjective definitions of inclusion independently from professional background. However, this requires further evaluation with structured sampling strategies. An initial DIF-analysis indicated that two items (LO2, LO4) did not measure equivalently between groups. This must be evaluated further with a more sophisticated sampling strategy considering different professional backgrounds.

We found small and medium correlations between the three FEDI scales and small correlations between the FEDI scales and teachers’ sense of efficacy. Attitudes towards inclusion were substantially correlated with only the HEB of the FEDI. This medium correlation can be explained by the normative imperative implied within an ethical or human rights-based perspective, because rejecting the idea of inclusion while defining it via ethical principles would mean to question one’s own ethical integrity.

Although Cronbach’s $\alpha$ was not extraordinarily high for any of the FEDI scales, we showed an acceptable amount of internal consistency, especially considering the small number of items used per scale.

Our study suggests that the FEDI is an economic instrument with acceptable reliability that can be used in inclusion research and the field. Using this instrument, practitioners can evaluate the definition of inclusion to which any participant adheres and if the teaching staff of a school share a common vision. Hence, the definitions of inclusion from the questionnaire broaden the empirical indicators for an inclusive school climate.

However, some aspects require further research evaluation. First, internal consistency and composite reliability can be used as measures of reliability but should be complemented by other methods, especially the retest method. Second, as research interest in development processes increases, the FEDI’s sensitivity to change should be evaluated. Although our findings support construct validity and discriminant validity, further efforts should be undertaken to evaluate the instrument’s validity.

Limitations

Some aspects of the study may influence the internal and external validity of our results. First, we disseminated the call for participation primarily via e-mail. Thus, we had an ad-hoc sample instead of a representative sample for a specified population. Although the stratified randomisation to build two separate samples for CEFA and CCFA is a good solution to control against repeated testing, it is still a workaround and reduces the number of participants per analysis. Therefore, replication studies must cross-validate our findings using appropriate sampling procedures.

This issue also affects the interpretation of MIV across different subgroups. We performed an explorative evaluation of MIV across two groups of university students. Since this sample was also included in the first assessment of the factorial structure, the results must be interpreted with caution and further replicated.

Finally, questionnaires on attitudes towards inclusion show a social desirability bias (Lüke and Grosche 2018a, 2018c). As our questionnaire did not use scales for controlling social desirability, we cannot completely determine if it measures subjective definitions. Thus, we would argue for analysing this possibility in other research projects.

Conclusions

As there is no clear definition of inclusion or inclusive education, researchers cannot know on what subjective constructs their inquiries are based. With the FEDI, we provide an economic quantitative measure that shows good psychometric properties and works well across different groups. We encourage cross-validation of our findings using qualitative approaches. Furthermore, the effect of subjective definitions of inclusion on other aspects, such as attitudes, self-efficacy, and knowledge, should be replicated and further evaluated.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

David Scheer

David Scheer, is a junior professor for inclusive education at the Annelie-Wellensiek Center for Inclusive Education, Heidelberg University of Education.

Lea Egener

Lea Egener, M.Ed., graduated from Paderborn University and is a student teacher at Grundschule Stieghorstschule Bielefeld.

Désirée Laubenstein

Désirée Laubenstein is a professor at the Institute for Educational Science, Paderborn University.

Conny Melzer

Conny Melzer is a professor for special needs education in the Faculty of Human Sciences, University of Cologne.