Reliability Testing of the Reasonable Adjustments for Inclusive Education Rating Scale

ABSTRACT

School inclusion of students with disabilities relies on reasonable adjustments to curriculum and class activities. The Reasonable Adjustments for Inclusive Education (RAIE) was designed to elicit reasonable adjustments from stakeholders in mainstream school inclusion for three contrived students with varied learning needs. We evaluated a scale for rating the quality of elicited reasonable adjustments across five dimensions: Agency, Authenticity, Real Learning, Strengths Based, and Inclusion. A trial (n = 5 participants) led to refinement of the scale, which was tested for inter-rater reliability with data from a randomized controlled trial (RCT) of an intervention to improve the quality of reasonable adjustments. RCT participants (10 parents, 10 educators, nine allied health professionals) provided 596 reasonable adjustments, rated for each dimension by two independent raters (n = 2980 ratings). Overall agreement was 75.3%; intraclass correlation (ICC) was .778. ICCs were moderate-good (.612–.816). Further refinement and testing with multiple raters are recommended.

KEYWORDS:

• Inclusion
• primary school
• reasonable adjustments
• reliability

The inclusion of people with disabilities in mainstream services and activities relies on reasonable adjustments to remove barriers (Shakespeare, 2018). Reasonable adjustments have been most readily understood in terms of modifications to the physical environment, such as the provision of ramps that give access to buildings by people with physical impairment. Perhaps less well understood are adjustments that enable participation of people with intellectual, social-emotional, and other impairments. Within the education sector, reasonable adjustments are mandated across many countries that have committed to the UN Convention on the Rights of Persons with Disabilities (CRPD), Article 24 (Dickson, 2019; United Nations, 2006). Although commendable, such commitment has often failed to translate to educational practice within primary and secondary schools (Iacono et al., 2019; Ineland, 2020; Mavropoulou et al., 2021).

In Australia, a key area of difficulty is in designing and implementing reasonable adjustments, which has tended to be left to teachers (Poed, 2016), who often lack the required knowledge and skills (Ineland, 2020; Mavropoulou et al., 2021). Shortfalls in meeting the needs of students with disability within mainstream settings by failing to make reasonable adjustments to curriculum and assessments has posed an inclusion barrier, which along with other practices, such as discouraging mainstream school enrollment and allowing school attendance only when teacher assistance is available for these students, has been referred to as gatekeeping, which is seemingly widespread in Australia (Poed et al., 2020). Arguably, the lack of operationalization of reasonable adjustments within legislation and policy has contributed to difficulties in their design and implementation: that is, what an adjustment might look like and what is considered reasonable have been poorly described (Mavropoulou et al., 2021; Poed, 2016).

Furthermore, although reasonable adjustments have been situated in legislation and policy, the intention that they provide a mechanism for equitable inclusion of students with disability in learning alongside peers overlaps with that of Universal Design for Learning (UDL). Curriculum developed according to UDL principles provides flexibility in proactively meeting diverse learning needs (Smith & Lowrey, 2017). Sailor (2017) argued that UDL as a school-based structural reform may not meet the needs of students who vary in the range and type of differentiated instruction. Arguably, reasonable adjustments are responses to student difficulties in accessing an existing curriculum or class activities, more broadly. In the absence of an agreed-upon understanding, in Australia, support for teachers to understand reasonable adjustments has focused on their ability to determine the level of supports required by students in their classes, motivated by the government’s needs-driven funding model. Funding allocations rely on teachers annually quantifying the level of adjustment required by each student (Dickson, 2019; Iacono et al., 2019; Mavropoulou et al., 2021). Descriptors for each of four possible levels have been provided to assist teachers in this process, but without capturing the nature of reasonable adjustments, as exampled in the following definition of a level 3 adjustment (considered substantial):

These adjustments are generally considerable in extent and may include frequent (teacher directed) individual instruction and regular direct support or close supervision in highly structured situations, to enable the students to participate in school activities. They may also include adjustments to delivery modes, significantly modified study materials, access to bridging programs, or adapted assessment procedures (for example, assessment tasks that significantly adjust content, mode of presentation and/or the outcomes being assessed). (Education Council, 2018, p. 32)

Almost exclusive reliance on teachers (they are asked to consult with parents) has precluded access to skilled input available from special educators (Iacono, McKinstry, et al., 2020) and allied health professionals (Dickson, 2019), who have been more involved in identifying a student’s category of disability (Mavropoulou et al., 2021). Valuable contributions that these specialists can offer in the education of students with varied disabilities and across the severity continuum (Department of Education and Training Victoria, 2016) may, therefore, remain unrealized. Iacono, McKinstry, et al. (2020) found that mainstream teachers prefer models of inclusion that provide them with access to this expertise, which is more likely to be available in special than mainstream schools in Australia.

The focus on level of adjustment, without defining reasonable adjustments, creates challenges in knowing what an effective adjustment might look like: that is, one that enables a student’s inclusion in the academic curriculum and social activities of the classroom or school. This challenge was addressed in a series of studies within a large project with the overall aim of supporting the design of reasonable adjustments, reported by Iacono, Spong, et al. (2020). First, three student scenarios were created to reflect characteristics of learning challenges experienced by students with disabilities. These were used to elicit reasonable adjustments from five trial participants. Then, the underlying construct of reasonable adjustments was explored by an expert panel, comprising stakeholders in the education of students with disabilities (Iacono, Spong, et al., 2020). Four panel members brought their combined experiences as parents of children with disabilities, mainstream and special educators, and allied health professionals (speech-language pathology and social work) to a series of workshops. Their discussions about the quality of reasonable adjustments were recorded and analyzed, resulting in a model comprising five dimensions, which formed the basis of a tool to elicit and rate the quality of reasonable adjustments (Iacono, Spong, et al., 2020). These dimensions are presented in Table 1. Because the data from the trial and the discussions of the expert panel were integral to the development of the tool, details of participants and procedures of these early research stages are detailed in the methods (below).

Table 1. The RAIE five dimensions of reasonable adjustments.

Dimension	Definition
Agency	The extent to which a student is actively given a choice or control in the activity through the implementation of the adjustment.
Authenticity	The extent to which the adjustment is considered a genuine learning activity, rather than something to simply occupy the student while others in the class are engaged in learning.
Real Learning	The extent to which the adjustment will address the student’s specific need(s) as identified in the scenario so that the opportunity for the student to learn is enhanced.
Strengths Based	The extent to which the adjustment provides opportunities to incorporate a student’s skill strengths and interests, as reported in the scenario, to further build and extend their learning and development.
Inclusion	The extent to which the adjustment and the activities described include or engage the student with peers in the same activities—even if additional supports are provided, or if the level or specific type of learning outcome differs.

In this study, we describe and evaluate the tool developed to elicit and rate the quality of reasonable adjustments along each of the five dimensions within the context of primary school inclusion of students with varied disabilities and severity levels. Data were from the trial administration of the Rating Adjustments for Inclusive Education (RAIE), and then a randomized controlled trial (RCT). The RCT tested the efficacy of a web-based intervention designed to improve the quality of reasonable adjustments by stakeholders in the education of students with disability (Iacono, Spong, et al., 2020). In the RCT study, participants were parents, teaching staff, and allied health professionals who were randomly allocated to an intervention group (with access to an on-line intervention) and a control group (without access to the intervention). They provided reasonable adjustments in response to student scenarios at three time points: pre-post, and about 2 months later. Full details of this study are provided in Spong et al. (2023).

The specific aim of the current study was to determine inter-rater reliability: that is, whether two raters could achieve consistency in judgments about the quality of reasonable adjustments generated by stakeholders in the school inclusion of students with disability using the rating scale.

Reasonable Adjustments for Inclusive Education (RAIE) development

The RAIE provides information and instructions needed to elicit reasonable adjustments from relevant stakeholders, such as teaching staff, allied health professionals, and parents. Each reasonable adjustment is then rated using a scale as a measure of its quality. The RAIE is based on the social model of disability, with a focus on inclusion through removing or reducing physical, attitudinal, communication, and other societal barriers experienced by people with disability (Shakespeare, 2018). It comprises student scenarios and, for each of these, their individual education plans (IEP) and description of a class lesson, and a rating scale.

The RAIE scenarios are of three students – Sienna, Parri, and Elang, ages 8, 11, and 9 years, respectively (Iacono, Spong, et al., 2020). These scenarios were created to reflect student characteristics that might arise from combinations of impairments in cognition, physical abilities, and social-emotional skills. Students were described according to their family situations, personalities and strengths, and challenges experienced in academic learning and social interactions, but without naming specific disabilities. In this way, a strengths-based approach was followed to avoid deficit-based categorization (Mavropoulou et al., 2021).

Within each scenario, the student was described as having a student support group (SSG), who had developed an IEP, as per the policy in Australia (Howes, 2017; Poed, 2016). The IEP is meant to be developed collaboratively with members of the SSG, comprising teachers, parents, and others involved in addressing the student’s needs (Howes, 2017). In each scenario, three potential goals from the student’s IEP were listed (Arthur-Kelly & Nielands, 2017).

Following the description of each student and IEP, details are provided of a class lesson taken from an Australian Curriculum learning area (Australian Curriculum Assessment and Reporting Authority, n.d.) at a grade level aligned with the student’s chronological age. For Sienna, the learning area was Literacy and Health Physical Education Grade 2; for Parri, it was Mathematics Grade 6; and for Elang, it was Science Grade 3. Completion of the RAIE involves providing three reasonable adjustments for each student that would allow participation in the class lesson described. The scenario for Parri is provided in the Appendix as an example and the full RAIE can be accessed from Iacono et al. (2021).

Methods

Ethical approval

Approval for the study was obtained from the La Trobe University Human Research Ethics Committee (HREC, Approval Number HEC18445). Permission was also obtained from two educational jurisdictions in Victoria, Australia, who required evidence of HREC approval. All participants provided signed informed consent.

Design

Data were obtained from both the RAIE trial and the RCT. The trial was conducted to practice administration of the RAIE and obtain data for the development of the scale that would be used to rate reasonable adjustments.

The RCT comprised three data collection points for each participant, regardless of whether they had been randomly allocated to intervention or control (no intervention) groups: pre-intervention, approximately 4 weeks (time given to complete intervention) post-intervention, and approximately 2 months at follow-up. At pre- and post-intervention, participants responded to the student scenarios of Sienna and Parri, and also to that of Elang at follow-up (see Iacono, Spong, et al., 2020). The current study relied on quantitative descriptive approaches, with a focus on rating reliability of the RAIE.

Trial

Participants

Trial phase participants were recruited through local mainstream schools contacted directly by researchers or through word-of-mouth. They comprised one parent, two educators, and two allied health professionals (speech pathology and occupational therapy). Each participant had experience in supporting students with disabilities in schools.

Procedures

Participants were emailed an electronic copy of the student scenarios and target learning activities, with instructions to consider reasonable adjustments that could support each student’s inclusion in the activity. In a phone interview approximately one week later, a researcher asked the participant to provide up to three reasonable adjustments for each scenario. The researcher wrote them down, prompting for details about who would be involved, what each person would be doing, and where the activity would occur. These interviews were audio-recorded to enable the accuracy of the written record to be checked.

Rating scale development

An iterative process was followed to develop the rating scale for scoring the reasonable adjustments generated first by a researcher (contrived), then by trial participants. Initially, a single 5-point scale was used. Each point on the scale reflected the extent to which a reasonable adjustment was inclusive according to the social model of disability (i.e., reduced barriers to participation alongside peers without disability), anchored by 1 (least inclusive) and 5 (most inclusive). This scale was distributed to the four expert panel members (Iacono, Spong, et al., 2020) who were asked to use it to rate 13 contrived reasonable adjustments. Instructions to panel members included that they base their ratings on the reasonable adjustment provided within the scenario context without inferring beyond that information, such as by considering the reasonable adjustment as preparation for some unspecified future activity. Expert panel members discussed their attempts in a workshop, during which they expressed difficulty in applying the scale. It was through analysis of their discussion that the five dimensions emerged (Table 1) (Iacono, Spong, et al., 2020). Despite attempts to operationalize the points on the scale by describing the extent to which each dimension was addressed, panel members continued to have difficulty in applying the scale and reaching consensus on the score for each reasonable adjustment.

Based on panel member feedback during a second workshop, the scale was reworked as separate 5-point scales for each dimension. This scale was then applied by two of the researchers to 12 reasonable adjustments provided by the trial participants. Given the early phase of development and the ordinal nature of the scale, a lenient criterion for agreement between raters was used, whereby rater scores that were the same or within 1-score point of each other were considered an agreement. For example, a score of 3 by both researchers or 3 by one and 4 by the other were both agreements; differences of more than one point (e.g., 3 and 5, respectively) were disagreements. Independent scoring resulted in an overall agreement of 72% with a range of 40%-100% across dimensions. These researchers then reached consensus through discussion, refining the operational definitions of each point on the scale for each dimension. This formed the final version of the scale. Operational definitions for each point for each dimension are provided in Iacono et al. (2021).

This revised scale was then applied by a research assistant who had not been involved previously in the study. She was a speech-language pathologist experienced in supporting students with language learning needs in mainstream classrooms. She was provided with all elements of the RAIE and scoring guidelines (including the operational definitions), and participated in a training session with a researcher. She rated the trial participant reasonable adjustments. Agreement (within one point) between this research assistant and the first researcher was 87% with discrepancies resolved through discussion.

RCT-generated reasonable adjustments

Participants

Three groups of participants were targeted for recruitment to the RCT study: parents, education staff (teachers and teacher assistants), and speech-language pathologists and occupational therapists (allied health professionals). Although students with disabilities themselves are considered integral to the SSG (Advocate for Children and Young People, 2020), they were excluded from the study because (a) the cases were hypothetical and (b) their direct participation raised ethical and practical challenges that could not be addressed within the scope of the study.

Participants were recruited from Victorian primary schools (the first year of compulsory school to Grade 6) located within the geographic area of the researchers’ university campus location, which were contacted directly by phone from contact details on publicly available listings on the internet. This process was supplemented by a snowballing strategy once successful recruitment began, ending when surplus to the required number was obtained to accommodate potential drop-out from the RCT. Of 37 regional schools contacted, 13 agreed to distribute information about the study (participant information and consent forms) to their school communities (teachers and teaching assistants, parents, and allied health professionals). Potential participants contacted a researcher directly for further information and returned completed consent forms to indicate their intention to participate. Participants in the RCT were 10 parents, 10 educators, and nine allied health professionals, all having experience of supporting students with disabilities (see Iacono, Spong, et al., 2020, for further details).

Data collection

A process similar to that for the RAIE trial was followed, which was written into a protocol and adhered to by all researchers to maintain fidelity. Only those procedural aspects of relevance to the current study are reported here.

The researchers who conducted the interviews checked the accuracy of their written recording of participant reasonable adjustments by listening to a digital recording of the data collection session, inserting a time into the Word™ document to mark the beginning of their reading out of the reasonable adjustment to the participant and the participant’s response (correcting, confirming, or clarifying). The researchers then uploaded each document and digital recording, identified with a code, to a university server.

Extraction and checking of data

A researcher extracted the reasonable adjustments for each data collection episode from the Word documents. This process was completed in four batches, each of which captured data from the RCT participants and time points as far as possible, given that participants progressed through individually: that is, data collection points varied across participants according to when each entered the study. In total, 596 reasonable adjustments were generated and included in the reliability analyses.

The extraction process was completed primarily by one researcher, who had not been involved in the data collection nor knew the codes that blinded the data collection point, participant group (intervention/control), and stakeholder type (parent, educator, allied health professional). Each reasonable adjustment was extracted to a table in another Word document with its code. Fifty-four reasonable adjustments were randomly selected to check that the researchers who conducted the interviews had accurately written each reasonable adjustment as expressed by the participant. Another researcher listened to the digital recording identified by the time point recorded in the document to coincide with the point at which participants heard the reading of their suggested reasonable adjustment and provided confirmation or otherwise. No changes to any of the 54 adjustments were made other than an occasional correction of grammar. To ensure that the researcher who extracted the reasonable adjustments had not changed content or meaning, another researcher, similarly blind to key aspects of the study and codes, completed this process independently for six reasonable adjustments; no discrepancies in extraction, other than minor differences in editing for grammar were noted.

Rating of reasonable adjustments

The two raters (researcher and research assistant involved in rating the trial data) independently completed the rating of all RCT data (i.e., each of 5 dimensions across 596 reasonable adjustments). Batches (n = 4) of reasonable adjustments were prepared once sufficient data had been generated and extracted, while ensuring each contained data from across time points. The exception was the first batch that did not contain follow-up data. The four batches were rated separately to break up the task and enable its progress as data were collected and extracted. To facilitate the rating process, all reasonable adjustments were transferred within their batch to an on-line Qualtrics survey with the scale presented for each dimension underneath. This presentation enabled the researchers to score each dimension efficiently.

Data preparation and analyses

Scores from both raters were exported from Qualtrics into an Excel™ file and tagged with the code for each reasonable adjustment to enable later analysis according to select variables. Point-by-point agreement (within one point) between raters and intraclass correlations (ICC) were calculated as indicators of overall reliability and were also calculated across dimensions (collapsing dimension ratings across scenarios) and scenarios (collapsing scenario ratings across dimensions) to identify potential variations along these variables. For the ICC calculations, a two-way random affects model using the consistency option in SPSS v26 was chosen to determine reliability between the two raters with systematic error taken into account (Koo & Li, 2016).

Results

Results are presented in Table 2, which indicate that point-by-point reliability was above 70% overall, and across dimensions and scenarios, with the exceptions being the dimension of Real Learning. Most ICCs were above .75, categorized as good, with the exceptions being the dimensions of Agency and Real Learning, and the scenario of Sienna, which were in the moderate range (.50–.75) and no ICC was classified as excellent (Koo & Li, 2016).

Table 2. Interrater reliability indices across two raters.

Component	Point-by-Point Total Agreement %	ICC	ICC 95% Confidence Interval
Overall (n = 2980)	2245 75.3%	.778**	.761–.793
Dimension (n = 596)
Agency	476 79.9%	.665*	.606–.715
Authenticity	478 80.2%	.755**	.713–.792
Real Learning	399 66.9%	.612*	.545–.670
Strengths Based	462 77.5%	.765**	.724–.800
Inclusion	430 72.1%	.803**	.768–.832
Scenario
Sienna (n = 1295)	937 72.4%	.742*	.713–.769
Parri (n = 1265)	995 78.7%	.816**	.794–.835
Elang (n = 420)	313 74.5%	.767**	.717–.807

* = Moderate interrater agreement; ** = good interrater agreement (Koo & Li, 2016).

Figure 1 provides a visual representation of the extent to which the two raters agreed across RAIE scores, using the first rater (researcher) as the reference point (i.e., the extent to which the second rater [research assistant] assigned the same score or within 1 point), with 1 representing a poor score across dimensions (combined) and 5 a strong score. As evident in Figure 1, most agreement was obtained for scores of 1 and 4, while neither rater tended to assign a score of 5; most disagreement was for the score of 1. Table 3 provides examples of reasonable adjustments for which the raters agreed on a 4 or 5 for at least one or more dimensions.

Figure 1. Rater agreement and disagreement across the five RAIE dimensions (1 = least; 5 = most).

Table 3. Example of scores agreed across raters for reasonable adjustments generated for Parri.

Reasonable Adjustment	Dimension	Score	Operational Definition^a
1. Parri to walk with the group around the school to see landmarks. The teacher to explain distance, going from one place to another, distance, perimeter. Students to use an iPad to take notes (count buildings, perimeter in meters, how far it is between buildings) or tally what they find, and note how far one place is from the other (e.g., using clicker to mark physical distance, or count buildings). The objective is to use Parri’s strengths and interests (walking), to provide a real life experience of the task, to make Parri’s specific goal to understand perimeter, distance, and counting rather than the group goal, which is broader and more abstract.	Authenticity	5	Scores of 5 or 4 indicate that the RA is an activity judged to be a genuine classroom or lesson-related activity (5) or fairly genuine (4) and provides the student with an opportunity to learn.
	Agency	3	Score of 3 indicates that the RA may or may not give the student some choice or control, but it is difficult to know in the context of the details provided in relation to the scenario content.
	Real Learning	4	Scores of 5 or 4 indicate that the RA addresses specific (5) or general (4) learning needs of the student, and may including specific strategies with potential to enhance the student with disability’s learning.
	Strengths Based	5	Scores of 5 or 4 indicate that the RA is an activity that directly (5) or indirectly (4) builds on the student’s strengths/interests or incorporates one or more strengths/interests into the activity.
	Inclusion	5	Scores of 5 or 4 indicate that the RA involves an activity that fully (5) or mostly (4) includes the student with peers and in the same activities.
2. Prior to the task, the whole class will role-play the activity, walking around the school perimeter and looking at the different landmarks, paths and trees that they will use during the task. Parri will choose who he works with (with teacher support), and will also have a visual calendar with pictures of what will happen during the activity, and what students need to do. The teacher will check on the group. The objectives are to provide full warning of what will happen during the activity, set expectations around the task, and for Parri to be involved and learn.	Authenticity	5	Scores of 5 or 4 indicate that the RA is an activity judged to be a genuine classroom or lesson-related activity (5) or fairly genuine (4) and provides the student with an opportunity to learn.
	Agency	4	Scores of 5 or 4 indicate that the RA gives the student direct (5) or indirect (4) opportunity to make a choice or have control in the activity.
	Real Learning	4	Scores of 5 or 4 indicate that the RA addresses specific (5) or general (4) learning needs of the student, and may including specific strategies with potential to enhance the student with disability’s learning.
	Strengths Based	3	Score of 3 indicates that the RA may or may not build on a strength/interest, but it is difficult to know in the context of the details provided in relation to the scenario content.
	Inclusion	5	Scores of 5 or 4 indicate that the RA involves an activity that fully (5) or mostly (4) includes the student with peers and in the same activities.

a = provided for the score assigned from the RAIE manual; RA = reasonable adjustment.

Discussion

The development of the RAIE was motivated by a lack of attention in the literature and policies on the quality of reasonable adjustments designed to support the inclusion of students with disability in mainstream primary education (Iacono et al., 2019; Mavropoulou et al., 2021). The question of what a quality reasonable adjustment might look like was informed by discussions of a panel representing stakeholders experienced in the support of students with disability (Iacono, Spong, et al., 2020). They considered a reasonable adjustment should capture the dimensions of authenticity, student agency, whether an opportunity for real learning was consistent with a student’s learning needs, the extent to which a student’s strengths had been considered, and, finally, the extent to which a student was included in the academic or social milieu of the classroom. The student scenarios were designed to depict information about students and their learning tasks that would enable participants to generate reasonable adjustments addressing each of these dimensions; raters judged the extent to which this had been achieved.

The overall results of the reliability analysis indicated that the RAIE could be rated with good agreement between raters, one of whom was closely involved in the RAIE development and the other, independent of this process. By including a naïve rater, we tested the extent to which access to the RAIE and operational definitions could be applied to the task of rating the reasonable adjustments provided by trial participants, and then again, from the RCT participants. This process was found to be relatively successful for three of the dimensions and two of the scenarios, with the exceptions of Agency and Real Learning dimensions, and the Sienna scenario, for which only moderate consistency was obtained. The drop in scoring consistency (i.e., ICC scores that accounted for systematic variability) across these variables may have reflected differences across the scenarios in terms of details that would enable participant-generated reasonable adjustments to incorporate the dimension or raters to evaluate them, in that the same reasons may apply to both tasks of generating and rating.

It is possible that the dimension Real Learning, for example, required further detail within each scenario to capture how well a reasonable adjustment resulted in learning for the student depicted. The operational definition for a high score on this dimension for the first exampled reasonable adjustment (Table 3) refers to specific strategies that offer the potential to enhance Parri’s learning by breaking down the activity and providing him and his peers with a means to record what they find as they walk around the school perimeter. In contrast, providing an opportunity for agency may present challenges in a classroom by relying on the student to be given some choice within learning activities most often determined by a teacher. Individual student choice may be difficult to provide other than at a micro-level (i.e., offering a choice within a teacher-chosen activity), without impinging on the potential for other dimensions, such as inclusion. Too much choice, for example, could mean that the student is no longer working alongside peers or is working on a task unrelated to the learning objective of the session (which would also impinge on the activity’s authenticity). In the second example in Table 3, this balance between Agency and Inclusion has been achieved in that Parri is given a choice within a whole-group activity by choosing who he works with. However, there is little in the form of specific steps or teaching and learning strategies relating to the potential for Real Learning, hence the score of 3 for this dimension. These considerations may have hampered consistency in scoring, with raters perhaps struggling to avoid projecting beyond the context, as may be required to judge the potential for Real Learning or to make sure a rating for one dimension was independent of those for other dimensions.

Designing and judging reasonable adjustments to increase a student’s potential to learn and build on their strengths could rely on participants who generated reasonable adjustment and the raters who judged them to know more about each student’s underlying skills in the scenario. Information from assessments, such as of speech and language, or cognitive levels were deliberately omitted, as were labels for their disabilities. This omission was to ensure that the RAIE was aligned with the social model of disability, the foundation for reasonable adjustments as strategies to remove barriers to inclusion (Shakespeare, 2018). A further reason was to focus the task on the classroom environment and the students’ functional needs, while reducing possible beliefs about limitations in their ability to learn in an inclusive setting (Poed et al., 2020), and how a student’s category of disability might dictate their school setting (Mavropoulou et al., 2021). Rater differences in terms of their familiarity with the characteristics of the students depicted in the scenarios potentially resulted in differences in ratings, most evident for Sienna, who had characteristics and learning challenges consistent with intellectual disability, and far less for Parri, whose learning challenges were more aligned with being on the autism spectrum. It is possible that the raters brought their respective understanding of both the students’ learning needs and dimensions, resulting in disagreements at least in some instances.

Implications

The RAIE has potential for use in research in which the quality of reasonable adjustments is the dependent variable. This initial investigation into the reliability of the RAIE provides direction for further development of and research into the scenarios and the rating scale for application in both research and educational practice. Such work needs to be guided by the primary function of the RAIE as a measure of the quality of reasonable adjustments developed by those who are key to the support and primary school education of students with disabilities. In this way, a key practice implication of this study and potential uses of the RAIE, which includes the student scenarios, are in terms of shifting the narrative about reasonable adjustments from their level to their nature and quality, with the latter operationally defined along each of five dimensions. Our research into developing good quality reasonable adjustments addresses concerns raised about a failure to define them, which has created opportunities for poor quality reasonable adjustments and, in turn, function to exclude students with disability from quality learning opportunities (Iacono et al., 2019; Poed, 2016; Poed et al., 2020). Within educational practice, the RAIE may be most useful in providing a basis for reflection and comparison across key stakeholders in the education of students with disability on the extent to which suggested adjustments provide opportunities for agency, real learning, and inclusion, and are authentic and strengths-based.

In terms of further research, the findings suggest that the RAIE was a reasonably reliable measure in terms of agreement between two raters, but further analysis is needed. Suggestions include obtaining ratings from multiple raters with varying experience with students with disabilities, but perhaps using a refined tool in which the dimensions are restricted to Authenticity, Strengths Based, and Inclusion, given that good consistency in agreement was obtained for these. Rather than ignore the extent to which a reasonable adjustment addresses the potential for Real Learning, another mechanism for tapping into this dimension may be needed, such as incorporating information about how the student engaged in the activity. The other option is not to change the scenarios or scale, but rather to systematize the training of raters to improve reliability, such as through multiple examples of reasonable adjustments with low to high scores across dimensions. The potential to implement this strategy is facilitated by the large database of reasonable adjustment ratings generated from this study. Exclusion of those with poor agreement across raters still provides over 2,000 ratings distributed across scenarios and dimensions. Further testing of only those ratings with good agreement across multiple raters will enable identification of the underlying constructs of the RAIE and its dimensions. Further, research with large numbers of raters will enable the response scale to be tested, such as through Rasch analysis to determine the extent to which scores measure a dimension and whether certain scores along the scale should be combined (Pallant & Tennant, 2007).

For the current study, RAIE ratings were analyzed according to the scenarios and dimensions; for the RCT, scores included analyses according to time points (pre-, post-, and intervention follow-up) for each scenario and intervention versus control groups. Decisions about how dimension ratings were analyzed (e.g., within participant types, control versus intervention groups, or within scenarios) were based on the aims and research questions. Such questions may not be relevant in educational or training contexts, in which skill development of individuals is often the focus. Further research into the response scale will reveal the validity of combining rating scores from individuals within each dimension (Pallant & Tennant, 2007), which can serve to measure the effectiveness of training in judging the quality of reasonable adjustments (e.g., pre- to post- administration) and demonstrate understanding of each dimension. It may be that comparing scores according to dimensions will reveal strengths and areas for training in relation to dimensions, while comparison of an individual or stakeholder groups’ ratings across scenarios may highlight difficulties in meeting scenario learner needs.

Conclusion

The development and preliminary testing of the RAIE provides an early attempt to refocus understanding of reasonable adjustments and supports from the extent to which teachers make changes to their usual ways of teaching to considerations of how to remove barriers to inclusion of students with varied learning needs. Our research indicated that measurement of reasonable adjustments as a multi-dimensional construct met with moderate-good agreement in scoring. Future research into dimensions incorporated into the RAIE and improving the reliability of scoring is warranted. Such research can also serve to maintain a focus in the published literature, with implications for practice, regarding how to best support varied student learning needs.

Acknowledgments

Thanks are extended to the participants in the study and to Ms Emily Greaves for her assistance with rating the reasonable adjustments and to Dr. Oriane Landry for her assistance in early parts of the study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was supported by La Trobe University Building Healthy Communities Research Focus Area funding.

Appendix

Scenario of Parri

Parri is 11 years old. He is in a 6th-grade class in a mainstream school. Parri lives with his mother in a unit just a few streets from his school. She walks him to school every day. Parri does know how to find his school and they use the same route every day because Parri hates to divert from it. Still, his mother is concerned that Parri might get teased by other children when he gets to the school gate. They don’t usually see other students during the walk to school, because students tend to catch a school bus or be driven to school by a parent.

Parri loves to engage in what seem to be conversations with adults, including any teachers on the playground or in a classroom. He can get quite excited during these conversations, often jumping up and down and raising his voice. Others struggle to understand what Parri is trying to tell them, as he talks very fast. They can understand him, however, when he uses short phrases or sentences, but these often don’t seem relevant to the situation. For example, a favorite phrase that he often repeats is “No more for you, young man.” He will sometimes walk into other classrooms and begin a conversation with the teacher using this phrase.

Parri is not very interested in the books that are being used in his class, choosing instead to read only books about insects. He is able to read these aloud; when asked questions about what he has read, however, he is unable to answer them. He can also count the number of insects appearing in a book, starting at the first page and ending with the last. His teacher is unsure about his ability to understand what he is reading and he is also unclear about what Parri might understand from what is said to him. Sometimes he does follow instructions; at other times, Parri seems to “do his own thing,” which can get him into trouble for noncompliance.

Parri is fascinated by small insects and will often carry them in from the playground, put them on the floor and then follow them around watching their movements. He becomes distressed when asked to take them outside again.

Parri loves working on the computer and enjoys interactive games, but it is difficult to shift his attention to other activities when the lesson moves on. During these times, he can become agitated, and jump up and down and raise his voice. When the class is engaged in other activities at tables, Parri will often walk around the perimeter of the classroom.

During recess and lunchtime, Parri often can be found walking the perimeter of the playground, stopping only to watch insects; sometimes, he can be heard counting them. At these times, other students may make fun of him.

Parri has a Student Support Group, who have designed an Individual Learning Plan. Examples of goals from his plan follow:

1. Parri will verbally answer three questions about a book on insects, relating to “what,” “where,” and “who” three consecutive times over three different days.

2. Parri will consistently and independently transition from one classroom activity to the next after looking at a visual calendar depicting all pre-recess activities over three successive days.

3. Parri will verbally greet teachers in the playground with “Hi Mr or Miss [name]” in response to the teachers greeting him first and giving him the spoken model, on five consecutive occasions for two days and involving at least two different teachers.

Curriculum Learning Area: Mathematics

The teacher has a large aerial map of the school, and has drawn grid lines so as to divide the school into equal-size squares. Students have been asked to work in threes. Each group has been given an A4 size copy of the aerial map on which to draw their grid lines and three school landmarks written on cards (e.g., buildings, playground features, paths, trees). Each group must work out a grid reference system and identify the location of each landmark according to its grid reference. They are to write this grid reference on the card with the name of the landmark. The students will then come together in the class to share their work and decide if they can create an overall grid map with the locations of all the landmark features marked according to a common grid reference system.