Classifying the evidence for evidence-based classification in Paralympic sport

1

A central component of the Paralympic mission is to “ … enable Para athletes to achieve sporting excellence” (International Paralympic Committee, 2019, p. 21). Para sport classification systems perform two functions that are critical for the realisation of the IPC’s mission. First, a sport’s classification system defines the minimum impairment required to be eligible to compete in that Para sport. In this way, classification is fundamental to Para sport because, in a literal sense, it provides a framework for determining who is (and who is not) a Para athlete. Second, a classification system groups athletes into sport classes which control for the impact of impairment on the outcome of competition to ensure that, as far as possible, sporting excellence determines which athlete or team is ultimately victorious (International Paralympic Committee, 2015; Tweedy & Vanlandewijck, 2011). Without classification it is not possible to hold legitimate Para sport competitions, or to differentiate the very best athletes from all others.

It is only 10 years since the conceptual framework for evidence-based classification was outlined in the IPC Position Stand on the background and scientific principles of classification in Paralympic sport (Tweedy & Vanlandewijck, 2011). The position stand stated that each sport should develop its own classification system based on empirical evidence that demonstrates the association between impairment and performance in that sport. This represented a significant departure point from existing classification systems which often were developed on the basis of opinion and were similar across sports with very different demands.

Given the relatively short time since the publication of that original position stand, it is with great pleasure that we are able to co-edit this special issue of 19 scientific articles on evidence-based classification in Para sports. Ten years ago the existence of such a special issue could not have been envisaged, and indeed, would not have been possible without significant, dedicated contributions from the whole Movement: the IPC and its member organisations, Para athletes, coaches, classifiers, and the scientific community who have produced the research. The body of research presented in this special issue of the Journal of Sports Sciences has a number of striking features. It has remarkable breadth, covering evidence-based classification for athletes with physical, intellectual, and vision impairment. And, as we are about to explain, the research papers cover not only each of the steps required in the development of a sport-specific system of classification, but have begun to address issues relevant to the effective implementation of those new systems. Overall, this body of research materially demonstrates that the Movement is progressing to the point where classification decisions will not only be valid and reliable, but also transparent and defensible.

Classifying classification research

A five-step process has previously been proposed designed to describe the research required for the development of an evidence-based system of classification in Para sports (see inside large square in Figure 1; Tweedy et al., 2016). First appearing in the IOC Handbook for Training and Coaching the Paralympic Athlete, the five-step process has been used to guide the development of evidence-based classification in a range of sports, including several of the research projects described in this special issue. The five steps can be summarised as follows. In Step 1, the sport and the eligible impairment types are identified. A sport can select whichever impairment types it chooses to offer for its own competitions, but for the purposes of competition in Paralympic sport (i.e., in competitions governed by the International Paralympic Committee), impairment types must be selected from a list of ten eligible types (see Table 7.1 in Tweedy et al., 2016). In Step 2, a theoretical model should be developed to identify the determinants of performance in that sport. The two core undertakings in this step are to identify (i) the key tasks that an athlete must perform to optimise their performance in that sport (e.g., a swimmer must dive, swim, navigate, turn, and streamline), and (ii) the factors that govern (or limit) the performance of those tasks (e.g., anthropometric, strength-related, cognitive, psychological, visual abilities). Step 3 then requires the development of ways to measure both the impairment (Step 3a) and the determinants of performance (Step 3b). Appropriate tests may already exist, or they may need to be developed. It is important to note that tests of impairment as well as tests of determinants of performance may be laboratory or field based. To become part of the classification process, the tests should be as complex as needed to measure to construct of interest, but as simple as possible so they can be used in practice. Accurate field tests, validated through comparison with laboratory golden standards, would best fit the practical process of classification. Step 4 then uses those tests to assess the relationship between impairment and performance in that sport. Experimental studies are required that test the level of impairment of athletes (or even simulated impairment in some cases, see Krabben et al., 2021; Ravensbergen et al., 2016) and then compare the results with the measures of sport performance for those athletes. From those findings, Step 5 then requires the development of minimum impairment criteria and class profiles for the sport. Conceptually, minimum impairment criteria are required for each of the impairment types eligible for the sport and should be set so that an athlete who just meets those criteria will have a demonstrable disadvantage in the able-bodied version of the sport. Sport classes should balance two criteria: within a class, the competitive disadvantage experienced by athletes with the greatest impairment should be minimised; and the difference in performance standard between classes should be maximised (e.g., Connick et al., 2018). The minimum impairment criteria and sport classes should be developed using data from a sample of athletes that is representative of the typical athlete population in that sport in terms of size, skill level, sex, and diversity of impairment types.

Figure 1. Conceptualisation of the process required for evidence-based classification in Para sport. The Development Phase (blue) contains the existing five-step process for the research required to develop an evidence-based system of classification (Tweedy et al., 2016). The Translational Phase (green) addresses the implementation of the new classification system. The Monitoring Phase (orange) outlies the need for periodic monitoring of the classification system

Despite the welcome growth in classification research, there has at times been ambiguity in the degree to which some studies provide value and, in particular, whether they address one of the five steps in the classification research process. Given this concern, we believe it is worthwhile for researchers to clearly articulate which step(s) they seek to address. Therefore, here we briefly introduce each of the 19 papers in this special issue and indicate which of the five steps we believe is addressed by each paper, noting that some papers may go beyond the step(s) we identified.

First, Fliess Douer et al. address Step 1 by attempting to clarify what should be the eligible impairment types in wheelchair basketball. Those familiar with Para sports may be aware of the considerable controversy in the degree to which the classification system for wheelchair basketball meets the requirements of the IPC Athlete Classification Code, in particular with the inclusion of impairment types not eligible under the Code. Fliess Douer et al. use the Delphi method to show that a group of experts in wheelchair basketball agree on a new definition for the aim of classification in wheelchair basketball, but also show ongoing disagreement about how to define the eligible impairments for the sport, providing vital insights into those disagreements.

Van Biesen et al. address both Steps 1 and 2 of the five-step process by outlining a conceptual model of sport-specific classification for Para athletes with intellectual impairment. The paper adds to existing position stands for sport-specific classification (Mann & Ravensbergen, 2018; Tweedy & Vanlandewijck, 2011) by extending the conceptual basis of classification specifically to athletes with intellectual impairment. Van Biesen et al. outline the current approach for determining eligibility for competition, and highlight important research challenges for the future.

Runswick et al. also address Step 2 by developing a conceptual model of the determinants of performance in 5-a-side football for athletes with vision impairment. Using the Delphi method, Runswick et al. canvas the views of a group of experts in the sport to uncover the aspects of vision impairment that the experts felt were most likely to impact performance in football (e.g., visual acuity, motion perception), and also the determinants of performance in the sport likely to be affected by vision impairment.

Almost half of the papers in this special issue address Step 3, the step that requires researchers to develop valid measures of impairment (Step 3a) and of determinants of performance (Step 3b). Six papers specifically address Step 3a. Hutchinson et al. conduct a systematic review of the measures of impairment employed for classification in wheelchair sports, specifically identifying measures of strength, coordination, and range of motion. Maia et al. evaluate an objective method of assessing motor coordination specifically designed for Para swimming using accelerometry (for movement smoothness) and video analysis (for rhythm and accuracy). Also in Para swimming, Smith et al. test the reliability of an instrumented method of evaluating trunk impairment using assessments of trunk coordination, range of motion, and strength. Given that measures of impairment should not change as a result of training, Wilson et al. examine the degree to which measures of upper limb strength, range of movement, and coordination change as a result of swimming training. Encouragingly, results show little change in the measures of impairment despite marked improvements in swimming performance. Altmann et al. evaluate the construct validity of new measures of arm coordination suitable for wheelchair rugby, with tests used to assess coordination in the shoulder, elbow, forearm, wrist, and fingers. Finally, Lemmey et al. investigate measures of intellectual impairment that could provide useful to determine whether eligible athletes with intellectual impairment should, in the future, be separated into multiple competition classes rather than the single class currently used for all competition for athletes with intellectual impairment.

In addressing Step 3b, the special issue contains papers that introduce measures of determinants of sport performance suitable for athletes with physical, intellectual, and vision impairment. Carabello et al. identify measures of performance in the sport of sailing that would be suitable for future work to uncover the impairment-performance relationship in the sport. Khudair et al. investigate the usefulness of measures of pacing as an indicator of performance in the sport of basketball for athletes with intellectual impairment. And Kons et al. introduce new measures of technical variation in attacking moves in judo to help better establish the relationship between vision impairment and performance in the sport.

Step 4 requires researchers to assess the relationship between impairment and performance in a given Para sport, and the number of studies that address Step 4 in the special issue reflects the significant progress made in reaching the latter stages of the evidence-based classification process. Mason et al. examine the association between arm strength and performance in wheelchair rugby, with the results suggesting that arm strength provides a valid means of clustering athletes in the sport. Reina et al. investigate the relationship between coordination and performance in cerebral palsy football. Using three tests of coordination, Reina et al. show an association between impairment and match physical load in football. Latham et al. show that the loss of peripheral vision does not appear to impact performance in shooting for athletes with vision impairment, concluding that visual field testing is not necessary for classification in the sport.

Four studies not only address the association between impairment and performance in a sport (Step 4), but also address Step 5 by proposing minimum impairment criteria or class profiles for their sport. Van der Linden et al. investigate the association between measures of impairment and performance in RaceRunning, a sport in which athletes with limited ability to walk use a three-wheeled frame to propel themselves. The authors show the emergence of two clusters of athletes that significantly differ both in the measures of impairment and race speed. Two papers use the simulation of impairment to propose new minimum impairment criteria in sports for athletes with vision impairment. Stalin et al. simulate impairment in skilled alpine and Nordic skiers, concluding that less impairment should be necessary to qualify to compete in those sports than is currently the case. Krabben et al., on the other hand, investigate the association between impairment and performance in judo and conclude that more vision impairment should be required to qualify to compete in judo than is presently the case.

The final paper in the special issue does not fit into the five-step process, but in doing so reflects the degree to which classification research has progressed, and helps to highlight important avenues for future research. Having established and implemented a new evidence-based system of classification in shooting for athletes with vision impairment (Allen et al., 2016, 2018; Myint et al., 2016), Allen et al. conduct a qualitative study to evaluate stakeholder opinions about the new system of classification in shooting. Given that this (and potentially other) papers do not fit into the existing five-step process, we now revisit that process and position it in what we consider to be important considerations for future research in evidence-based classification.

Implementing the findings from classification research

Until recently, the question of what processes should be followed once an evidence-based system of classification has been developed has not been relevant – the prospect was simply too far in the future. For this reason, early publications that aimed to operationalise the Position Stand and describe a process that researchers could follow to develop evidence-based methods of classification finished at the point where a validated system had been produced (Tweedy et al., 2014, 2016). In particular, the five-step process for research needs outlined in the IOC Handbook for Training and Coaching the Paralympic Athlete stopped at the point where recommendations had been put forward for minimum impairment criteria and a class structure. However, as the prospect of evidence-based classification systems becomes a reality in some Para sports, questions arise about how to implement the findings. Here we seek to position the existing framework within a wider conceptualisation of evidence-based classification for Para sport by breaking down the process into three broad phases: a development phase; a translational phase; and a monitoring phase (see Figure 1). The development phase for classification research simply encompasses the existing five-step process required for developing evidence-based methods of classification (Tweedy et al., 2014, 2016). The translational and monitoring phases go beyond development to describe what needs to occur once a new system is developed.

The translational phase is necessary when the new classification system is implemented by the sport. This step can be surprisingly challenging and requires careful planning. Translational research is a branch of science in its own right, designed to uncover how new policies or procedures can be translated into practice with fidelity (Tweedy et al., 2018). Within Para sports, the translational phase typically commences with a period of consultation to communicate the proposed changes to the sport membership and to gather their feedback. The research findings emerging from the development phase need to be presented in a way that is relatable to the wider membership, most of whom will have no scientific background, with communication required typically in the form of a written report and/or presentations. These recommendations should include an assessment of the impact of the changes on the existing body of athletes, including estimates of the proportion of athletes likely to change sport class, those who may be no longer eligible, and those who will become eligible to compete. Sports may convene an implementation committee, and will often host consultation sessions where the findings can be explained to the membership and to gather feedback. The report and recommendations may require clarification on the basis of the feedback and, in some cases, additional research may be required.

The process of implementation can be controversial and evoke strong reactions, particularly if the proposed changes will impact the eligibility of some athletes to compete or result in a change in their sport class. Potential consequences for impacted athletes include the perception that their impairment is no longer recognised or that their previous achievements have been undermined. Some athletes may even lose their funding from their national sport federation. Sports who change their classification criteria need to consider contingency plans to care for the well-being of these athletes. Interventions may include education, care, and transition programmes to encourage athletes into other Para sports that offer opportunities for those with that level and type of impairment. Researchers should look to work with sports to better understand the impact of changes on affected athletes and the type of support services that would best help athletes through this potentially difficult transition.

Because athletes and their national federations require certainty about each athlete’s eligibility well in advance of the Paralympic Games, the implementation requires long-term planning. A new classification system may result in the inclusion of new athletes who were previously ineligible (and who may be much less trained in the sport) and, conversely, previously eligible athletes may become ineligible. Accordingly, for most it would be inappropriate to change classification rules immediately before a major competition such as the Paralympic Games. Long lead-in times do sometimes have unavoidable consequences though. For instance, a proposed change for the 2028 Los Angeles Paralympic Games might be publicly available for membership consultation before the 2024 Paris Games. This could result in some athletes taking part in the 2024 games in the knowledge that they may not be eligible to compete beyond that date. Again, researchers and sports need to prepare for the adverse controversy those changes might introduce.

Do all sports need to do research for the translational phase? Our answer is “no”, but research would be invaluable for understanding how best to communicate changes in classification to athletes. Communication Scientists, for instance, work to identify the most effective ways to enhance the understanding and acceptance of changes in health policies (e.g., to reduce smoking, enhance COVID vaccination rates), and much can be learned from those fields to assist in the effective communication of changes to classification. Research can address, for instance, the arguments that would best convince athletes of the need for change (e.g., centred on fairness, or the value of medals), the best format of communication given the target age-group (e.g., by public information sessions, email, social media), and the adjustments in messaging required given some forms of impairment (e.g., visual and intellectual impairment). The article by Allen et al. in this special issue is an example of research designed to evaluate the implementation of the new classification system for shooting for athletes with vision impairment. The experiences of key stakeholders as a result of the implementation can be canvassed to improve the success of future changes not only in shooting, but also in other sports that implement their own sport-specific system of classification.

In the Monitoring Phase, the new classification system should be periodically evaluated to determine whether further modifications are necessary following implementation. If changes are required, then the five-step process may need to begin again. This might seem disheartening, because it implies that evidence-based classification is an ongoing cycle that is never complete. However, in reality, monitoring should be a reiterative process in which successive versions of the cycle become faster and simpler, acting like a funnel, with the magnitude of any necessary changes diminishing with time. If a “ground truth” classification system is assumed to exist which represents the fairest possible system of classification for any given sport, then successive iterations of an evidence-based classification system should move progressively closer to that ground truth. Subsequent iterations should be faster given that progressively smaller changes should be required, and also given that valid measures of impairment and performance will already exist, and that the size of the dataset on which decision making is based increases.

There are certain reasons why early rounds of research might not arrive at a “ground truth” classification system. In particular, the impact of changes to classification can sometimes be difficult to predict when a new system is introduced. For instance, it can be difficult to anticipate how well previously ineligible athletes will perform when they become eligible for a Para sport. It may have become clear from classification research that a group of athletes exists who should be eligible to compete because their impairment impacts performance in the sport. However, if those athletes were largely untrained because of a lack of access to resources and coaching then it can be difficult to predict their true potential when given appropriate training and resources. Indeed, they might perform better than the existing Para athlete population and require their own sport class. Similarly, it can be difficult to evaluate the potential of athletes who remain eligible but were under-represented in the previous classification system because that system disadvantaged them. They too might not have had access to the same resources and coaching as others and might improve give equal access.

Other instances may lead to a change in the ground-truth classification system and require the five-step process to be revisited. If new rules and/or equipment were to be introduced to a sport, then those changes could fundamentally alter the nature of the impairment-performance relationship in the sport and therefore require changes to the classification system (Tweedy & Vanlandewijck, 2011). For instance, a wheelchair sport might change the type of strapping an athlete is able to use to fasten themselves into their wheelchair. If the new strapping reduced the degree to which wheelchair performance relied on a particular body part (e.g., the pelvis), then the classification system might need to be modified to reduce the degree to which eligibility and/or class allocation relies on an assessment of the functional ability of that body part. Similarly, new equipment (e.g., new prosthetic limbs) might improve athlete functionality by better compensating for impairment and, as a result, change the nature of the impairment-performance relationship. Further, a sport federation might decide to make the sport available to athletes with impairment types that were not eligible in that sport’s existing classification system. Athletes and their sports will continue to evolve, and it remains vital to consider the ongoing impact of those changes on classification if they change the nature of the impairment-performance relationship.

Research is encouraged and indeed necessary during the Monitoring Phase to evaluate the efficacy of a new classification system. Researchers might examine stakeholder satisfaction with the new system (e.g., athletes, coaches, spectators), or might empirically investigate the nature of the impairment-performance relationship under the new system. However, research validating the new classification system should not simply compare sport performance across the new sport classes. On the one hand, comparisons across sport classes might reveal no difference between the performance of some classes, indicating that those classes could be combined. On the other hand, however, this approach would not uncover any differences in performance profiles that exist within sport classes. It might be that a class should be split into multiple sport classes because of a significant relationship between impairment and performance within that class, yet this relationship will not be uncovered by simply comparing the mean performance of different classes. Access to measures of impairment for the athletes within each class is necessary to demonstrate whether impairment is related to performance within those classes (e.g., see Krabben et al., 2020).

The wealth of research in this special issue attests to the rapid development of evidence-based classification in Para sports. The next frontier for researchers will be to determine how best to translate those research outcomes into successful classification systems and to ensure their ongoing validity. Throughout this process, thoughtful implementation is vital. Even the best evidence-based system will have limited acceptance if the sport community does not understand the reason for implementing change, and how the new classification system was established. Similarly, research is needed to monitor the efficacy of the new system to ensure it remains valid as the sport evolves and new equipment and medical treatments become available. Effective implementation and monitoring of new sport-specific systems of classification will only be achieved through continued engagement between researchers and the Para sports community.