The challenge of complexity: research quality in assistive technology

The field of assistive technology is growing globally, thanks to increased investment in awareness and research from advocacy organizations, national and international non-governmental organizations, academic partners, and others. We are witnessing this growth and increased awareness in the increase in relevant research, resolution 71.8 on Improving Access to Assistive Technology at the World Health Assembly in 2018 (World Health Assembly, 2018), and the publication of the Global Report on Assistive Technology in 2022 from the World Health Organization and UNICEF (World Health Organization, 2022).

With this growth comes an increased recognition of the need for high-quality research to support evidence-informed assistive product development, service delivery standards and processes, and improved public policy. The recent publication of the WHO Wheelchair Provision Guidelines (World Health Organization, 2023) highlights the challenge of limited high-quality research to support many of the recommendations in the document, despite wheelchair research being one of the more developed areas of evidence in the field.

Assessment of research quality in health-related fields has historically been based largely on the perceived strength of the methodology used to conduct the research, with considerations for how the research was carried out, and by whom. In general, randomized controlled trials (RCTs) are perceived to be a higher quality of evidence when creating guidelines, while observational studies are perceived to be of lower quality of evidence. A variety of additional factors are considered in the critical appraisal of research which may further impact the perception of quality. We perceive systematic reviews and meta-analyses to provide the highest levels of evidence for decision-making. These levels of evidence are largely based on a medical model of research, and an assumption that an intervention may have a single causative effect on an outcome.

However, in complex interventions, like assistive technology provision, adherence to this model of evidence quality is problematic. Assistive technology provision is multi-faceted, individualized, and contextual. Even an intervention that looks the same on the surface – the provision of a specific product to address a specific functional limitation – will vary significantly in its application. The person’s needs, preferences, and functional capacity, existing skillset of the person, the product and its personalized specifications, set-up of the product and training around its use, role of the service provider, support networks around the person, and social, physical, and institutional environments in which the product is used all will have an impact on the resulting outcomes. To measure and evaluate outcomes of any one of these is challenging. To address the potential contribution of each of them in the context of a controlled trial is nearly impossible. Furthermore, placebo-controlled trials are impossible in assistive technology provision, and other forms of comparison between assistive products – or no assistive product at all – are often unethical. Evaluating outcomes associated with service delivery models is equally problematic.

Assistive technology is not alone in this challenge. In fact, a study looking at the grading of evidence quality in Cochrane Reviews found that complex interventions were regularly scored below simpler interventions for evidence quality, and that no complex interventions at all achieved a “high” quality evidence rating (Movsisyan et al., 2016). This presents significant challenges when we evaluate evidence using this approach for clinical guidelines and public policy. It is especially problematic when we focus on the existence of these levels of evidence for funding decisions which govern whether or not entire populations will get access to the assistive technology they need.

As a field, we are at a crossroads, where we must begin to define what constitutes high-quality evidence in the face of complexity. There is a role for all of us not only in defining this but also in promoting this definition among evidence-users, including public policy makers. The Assistive Technology journal is committed to continuing to publish high-quality research in the field, and recognizes the contributions of clinical trials, observational research, qualitative studies, and other methodologies to our overall understanding of the impacts and outcomes associated with assistive technology provision. We also remain committed to publishing high-quality research syntheses in the form of systematic and other reviews.