A systematic review on how to conduct evaluations in community-based rehabilitation

Abstract

Purpose: Community-based rehabilitation (CBR) must prove that it is making a significant difference for people with disabilities in low- and middle-income countries. Yet, evaluation is not a common practice and the evidence for its effectiveness is fragmented and largely insufficient. The objective of this article was to review the literature on best practices in program evaluation in CBR in relation to the evaluative process, the frameworks, and the methods of data collection. Method: A systematic search was conducted on five rehabilitation databases and the World Health Organization website with keywords associated with CBR and program evaluation. Two independent researchers selected the articles. Results: Twenty-two documents were included. The results suggest that (1) the evaluative process needs to be conducted in close collaboration with the local community, including people with disabilities, and to be followed by sharing the findings and taking actions, (2) many frameworks have been proposed to evaluate CBR but no agreement has been reached, and (3) qualitative methodologies have dominated the scene in CBR so far, but their combination with quantitative methods has a lot of potential to better capture the effectiveness of this strategy. Conclusions: In order to facilitate and improve evaluations in CBR, there is an urgent need to agree on a common framework, such as the CBR matrix, and to develop best practice guidelines based on the literature available and consensus among a group of experts. These will need to demonstrate a good balance between community development and standards for effective evaluations.

Implications for Rehabilitation

• In the quest for evidence of the effectiveness of community-based rehabilitation (CBR), a shared program evaluation framework would better enable the combination of findings from different studies.

• The evaluation of CBR programs should always include sharing findings and taking action for the sake of the local community.

• Although qualitative methodologies have dominated the scene in CBR and remain highly relevant, there is also a call for the inclusion of quantitative indicators in order to capture the progress made by people participating in CBR programs.

• The production of best practice guidelines for evaluation in CBR could foster accountable and empowering program evaluations that are congruent with the principles at the heart of CBR and the standards for effective evaluations.

• Community-based rehabilitation
• evaluation process
• framework
• methods of data collection
• program evaluation

Introduction

People with disabilities represent approximately 15% of the world’s population and are among the poorest and most marginalized of many communities [1]. In 1978, in an attempt to decrease the burden of disability in low- and middle-income countries, the World Health Organization (WHO) launched a strategy called community-based rehabilitation (CBR) [2]. CBR is now implemented in more than 90 countries and is defined as an inclusive community development strategy, which aims at the equalization of opportunities, rehabilitation, poverty reduction, and social inclusion of the population living with a disability [3]. It is critical to evaluate existing CBR programs and to demonstrate their effectiveness in order to promote their sustainability and ongoing financial support, as well as the development of new programs based on the lessons learned from more than 30 years of experience in CBR. Yet, program evaluation does not seem to be common practice in the field, nor always congruent with standards for effective evaluations.

A review on the effectiveness of CBR programs by Finkenflugel et al. [4] came to an unfortunate conclusion: evidence on the effectiveness and on the conditions under which CBR programs are most effective remains fragmented and largely insufficient. These authors retrieved few program evaluations on the effectiveness of CBR documenting both implementation and outcomes and highlighted that rigorous controlled studies on the efficacy of CBR were extremely rare. In addition, the wide range of outcomes analyzed in the different studies make it almost impossible to get a good general idea of the efficacy and effectiveness of CBR. Ultimately, there is very little consensus on how program evaluation and evaluative research should be conducted in order to be as close as possible to scientific standards, while remaining in harmony with CBR philosophy and the context of its implementation in low- and middle-income countries.

As Boyce and Ballantyne [5], we believe that CBR will not survive unless better and more systematic program evaluation systems are used to document the outcomes and the effectiveness of this strategy. This systematic review aims to document: (1) the characteristics of the process that could be followed in the evaluation of CBR programs, (2) the way in which a framework or classification model could help frame the choice of outcome measures, and (3) the characteristics of the data collection methods that could be privileged. This study proposes a thorough look at the recommendations available in the literature and represents the first step in the preparation of best practice guidelines for program evaluation in CBR.

Method

Search strategy

A systematic search was conducted by the first author (MG) on the main rehabilitation databases (CINAHL, Embase, MEDLINE, PsychINFO, and Scopus) with the keywords “community-based rehabilitation” AND “program evaluation” (OR evaluative research OR process OR evidence OR evidence-based OR framework OR classification model OR conceptual model OR methods). Because of the confusion present in the literature between rehabilitation happening in the community and CBR as a community development strategy, no synonym was included for the first key word. The years 1994–2011 were searched as 1994 is the year when the first joint position statement on CBR was published [3]. Articles published in this period are considered more representative of CBR today. In addition, a manual search was done on the WHO website. Finally, the references of the included articles were reviewed.

Inclusion and exclusion criteria

The selection of the articles that could provide indication on the process, models, and methods to be privileged in CBR program evaluation was conducted by two researchers who looked at all titles and abstracts, and read the full articles when in doubt. Disagreements on inclusion were resolved through discussion. Inclusion criteria were: (1) talking about CBR, as defined in the joint position paper in 1994 or 2004 (i.e., community development strategy), (2) having evaluation as a major theme, more precisely how evaluations should be done, (3) published in English or French. Articles reporting program evaluations without explicitly reflecting on the methodologies used were excluded. Peer-reviewed articles and guidelines were included, but book chapters, articles in newsletters, and articles published in journals without archives were excluded. Consistent with the suggestions of Kuipers and Harknett [6] and Mitchell [7], both program evaluation and evaluative research were considered; program evaluations can contribute to the evidence base if process and outcomes are described together, and if the outcomes that stand out in varied contexts and programs are verified using analytic study designs.

Methodological quality assessment of the studies

In order to make judgment on whether the recommendations for program evaluation represent a high level of evidence, two researchers assessed the methodological quality of all included studies, using tools appropriate for each type of study. The CASP tool was used to evaluate the methodological quality of observational qualitative studies [8], the AMSTAR scale for systematic reviews [9], the STROBE statement checklist for observational quantitative studies [10], and the AGREE II instrument for practice guidelines [11]. No appraisal tool was used to assess the quality of editorials (experts’ opinions) since these are generally considered to be the lowest possible level of evidence. The two researchers compared their assessment of the methodological quality of each article and reached consensus for the global rating of each article through discussion.

Both researchers also evaluated the strengths and weaknesses of each document in relation to the congruence with best practices in program evaluation and CBR. The four standards for effective evaluations from the Centre for Disease Control and Prevention (i.e., utility, feasibility, propriety, accuracy) were used as reference for best practices in program evaluation [12]. The principles of CBR applicable to program evaluation (i.e., inclusion, participation, empowerment) served as reference for the CBR philosophy [2].

Results

Articles and documents included

Some 373 articles were retrieved from the databases and 13 documents from the WHO website. In total, 22 documents met all inclusion criteria, two of which were found on the WHO website, and one was added after revision of the reference lists of the included articles (Figure 1). These included observational studies, practice guidelines, literature reviews and editorials. The characteristics of the included documents and the results of the quality assessments are provided in Appendix A. A list of the excluded articles is provided in Appendix B.

Figure 1. Flow chart for the selection of the articles.

Characteristics of the program evaluation process

Table 1 provides a detailed description of the characteristics of the evaluation process found in the nine documents that provided specific suggestions on program evaluation processes that are most relevant in CBR. These were divided in three categories that emerged from the findings: people who should be involved in the evaluation process, steps to follow, and time when the evaluation should take place.

Table 1. Findings about the characteristics of the evaluation process for CBR.

	Characteristics of the evaluation process
	People who should be involved	Steps to follow	Time to evaluate
Boyce and Ballantyne [5]	People with disabilities and other key stakeholders in the community should be involved in all the steps. They should be empowered through coaching and training. Evaluations should be as participatory as possible.	Decide whether or not to evaluate, determine the objectives and indicators, select coordinators, choose methods, write the proposal, prepare and test the methods, collect the information, analyze the information, and report the results.	–
Cornielje et al. [13]	–	–	Importance of baseline survey (before implementation).
Mitchell [7]	–	The last step should be to draw attention to CBR by reporting the results publicly.	–
Price and Kuipers [14]	The participation of the community is essential to ensure local relevance.	The process should be iterative and involve cycles of action and reflection.	–
Thomas [15]	–	–	Importance of collecting baseline data (before implementation).
Velema et al. [16]	–	–	Importance of collecting baseline (before implementation) and follow-up data on people who completed the program.
WHO and IDC [17]	CBR programs should engage in self-reflection (self-assessment).	The last step should be to report findings and meet community members to make decisions to improve the program.	Incorporate monitoring and self-assessment in regular activities of CBR programs.
WHO et al. [2]	Both internal and external evaluations can be valuable, but a combination of both approaches would ideally be used.	1. Focus the evaluation (purpose and first questions) 2. Collect information (to answer the questions) 3. Analyze the information and draw conclusions (statistical calculations or thematic analyses) 4. Share findings and take action (write report, meet community members, influence decisions, act on the results)	Importance of collecting baseline data when implementing CBR programs. Specific points in time during the project cycle are relevant for evaluation (e.g., midway through the implementation, immediately after completion of the project, some time afterwards).
Zhao and Kwok [18]	There are many possibilities, including self-evaluation, mutual evaluation (exchange between two CBR programs), higher authorities evaluation, and external evaluation.	1. Make a detailed evaluation plan 2. Collect materials and conducting investigation 3. Analyze results 4. Make evaluation report and give suggestion 5. Feedback of results and integration into practice	At regular intervals, including baseline survey before implementation, periodical evaluation during implementation, eventual evaluation when the program ends, and follow-up evaluation some time after.

In relation to who should be involved in the evaluation, the findings of this literature review suggest that the participation of the local community, including people with disabilities, is critical to ensure local relevance [5,14]. They also propose that self-assessment and external evaluations are suitable options [2,17,18].

Concerning the steps to follow when conducting CBR program evaluation, the need to start by focusing the evaluation process came out [2,5,18]. Two guidelines suggested that relevance, efficiency, effectiveness, sustainability, and social impacts could be used to focus the evaluation question [2,18]. Another element recommended by many authors is the completion of the evaluation cycle with reporting the findings and taking actions to improve the program [2,5,7,14,17,18].

As for the time when it is most appropriate to conduct an evaluation, the findings indicate that evaluation should be part of the regular activities of CBR programs, from the planning phase up to the follow-up upon completion of the program [17,18]. Many authors highlighted that the lack of baseline data in many CBR evaluations make it very challenging to capture the change happening over time [2,13,15,16].

Framework: process and outcomes used in program evaluation

A total of 13 classifications for CBR were identified in this review, including a broad variety of potential process and outcome measures, as well as indicators. They were not all presented as models for evaluation by the authors, but all included different categories on what should be measured when evaluating CBR programs. The different frameworks are presented in Table 2, along with process and outcome measures when available. The suggestions of the authors are divided in two categories: those associated with the implementation of the program and those associated with the outcomes of CBR.

Table 2. Frameworks proposed in the CBR literature and their associated process and outcome measures.

PROCESS: Framework categories and measures	OUTCOMES: Framework categories and measures
Adeoye et al. [19]
Implementation: role of people with disabilities, role of family, role  of community, role of health services, role of educational services,  role of vocational services, role of social services, challenges to  implementation ˆ Leadership role, formation of self-help groups, program coverage, participation in planning-implementing, access to health services, health education, home based care/intervention, access to educational services and equipments, inclusive education Policies, strategies, activities, and challenges to: community development, rehabilitation, equalization of opportunities, and social inclusion. ˆ Policies: mainstreaming, participation, implementation of laws and guidelines, implementation of disability policy ˆ Strategies: participation in planning-implementation, promoting community awareness, changing community attitudes, sign language training, promoting/advocating, increasing program coverage ˆ Activities: income-generating activities, sustainability, collecting baseline data, technical aids and services, participation in community, participation in family, employment of people with disabilities, personal development and skills, access to justice, access to employment, social integration, inclusive education, leadership role of people with disabilities ˆ Challenges: sustainability, unemployment, inadequate equipments, non-implementation of disability policies, inadequate access to CBR officials, communication barrier, discrimination, sexual abuse	*We placed the framework in the process category in order to represent the way they look at policies, strategies, activities and challenges. However, some elements may belong to both process and outcome evaluations if we pay attention to the verbatim used by the authors to provide supportive evidence.
Chung et al. [20]
CBR-related program content ˆ Advocacy, networking, involvement of relatives and families, of CBR manager, of CBR workers Participant empowerment/governance ˆ Leadership, self-help and mutual help, autonomy in program Community ownership ˆ Community support and recognition, national/government level support, collaboration and support among different sectors Program management and development ˆ Compliance to relevant service standards, ethical practices, management issues, sustainability, continuous growth and development	Participants outcomes ˆ Functional independence, education, economic independence, participation in family and community life, physical change, spiritual change, early detection and intervention
Cornielje et al. [13] *Categories originally proposed by Cornielje et al. [21]
Locus of power Commitment to involve others Type of response	Restoration of quality of life
Evans et al. [22]
Interpersonal quality Technical quality Management quality	–
Mannan and Turnbull [23]
–	Quality of life Family quality of life
McColl and Paterson [24]
CBR programs ˆ Aims ˆ Beneficiaries ˆ Strategies Supportive structure under CBR ˆ Human resources ˆ Structural resources ˆ Attitudes	–
Mitchell [7]
Service delivery Community involvement Technology transfer Organization and management	–
Pal and Chaudhury [25]
–	Parental adjustment ˆ Feeling of rejection and resettlement ˆ Positive feelings toward child ˆ Negative aspects of maternal mental health ˆ Feelings of guilt and self-blame
Sharma [26]
Strengths Weaknesses Opportunities Threats	–
WHO et al. [2] *Examples of outcomes for the 25 elements are available. Here, examples associated with the 5 components are presented.
Factors that promote sustainability Effective leadership, partnerships, community ownership, using local resources, considering cultural factors, building capacity, financial and political support.	Health: promotion, prevention, medical care, rehabilitation, assistive devices Ex.: Improved knowledge about good health, health sector awareness, access to health care and rehabilitation services, improved collaboration across sectors. Education: early childhood, primary, secondary and higher, non-formal, lifelong Ex.: Access to learning, accessibility of local schools, awareness of community that people with disabilities can learn, advocacy to facilitate inclusive education. Livelihood: skills development, self-employment, wage employment, financial services, social protection Ex.: Access to skills development, microfinance and social protection, equal work opportunities for women with disabilities, inclusion in poverty reduction strategies. Social: personal assistance, relationships-marriage and family, culture and arts, recreation-leisure and sports, justice Ex.: Access to personal assistance options, individual support plans, informal support for families, training availability for assistants. Empowerment: advocacy and communication, community mobilization, political participation, self-help groups, disabled people’s organizations Ex.: Informed choices and decisions, active participation in family and community, removal of barriers in community, creation of groups of people with disabilities.
WHO and IDC [17]
Transfer of skills to community level, program activities	Impact on individuals, community mobilization, opportunity for education, opportunity for work, involvement of disabled people
Wirz and Thomas [27]
Service delivery: program planning and management, financial and people management, training, sustainability	Maximizing potential: functional independence, education, economic independence, inclusion, leadership roles in the community, participation in/ownership of programs Ex.: Halting progression of impairment, improved mobility, improved ADL, improved communication skills, improved orientation and mobility skills training by parents, mobility aids, hearing aids, removal of physical barriers, reduced attitudinal barriers, special education or inclusive education attendance, skills and work placements acquisition, earnings through income-generation activities and self-employment, availability of credit groups, equal access to community services, leadership roles in the community, management of self-help groups. Environment: family attitudes, family involvement, community attitudes, inclusion of people with disabilities
Zhao and Kwok [18]
Evaluation of management: government role, NGO role, other sectors, communities involvement Evaluation of implementation: services in medical rehabilitation, in educational rehabilitation, in vocational rehabilitation	Evaluation of social impacts of CBR: attitude changes

Four articles suggest that CBR would certainly benefit from using outcomes and indicators derived from classification models [13,15,27,28]. Thomas [15] suggested that the desirable outcomes proposed in the CBR Guidelines for each element of the CBR matrix should be used to guide the choice of outcomes and indicators in CBR evaluations. Wirz and Thomas [27] believe that a bank of ready-to-use indicators derived from classification systems is needed while others argue that it is best to create a framework and indicators that meet the specific needs of the program to be evaluated [19,20].

Methods of data collection in program evaluation

Of the 22 documents, 15 had specific recommendations on the characteristics of the methods of data collection that are most relevant for CBR (see Table 3). Most authors agree that it is usually best to use more than one method to enable triangulation [2,6,18,20,22,26]. The findings highlight that traditional qualitative methodologies have been used extensively in CBR evaluations [29], but some authors argue that mixed methods and quantitative methodologies have more potential to contribute to demonstrating the effectiveness of CBR programs [13,18,27]. One point of disagreement in the literature lies in the need for tools that would be relevant in all contexts or some that would be context-specific.

Table 3. Characteristics of the methods of data collection in CBR.

	Suggestions about the characteristics of the methods
Adeoye et al. [19]	Fifteen-statement questionnaire in which participants needed to say if they agreed or not with each on a five-point Likert scale, developed after qualitative interviews in the local community in Uganda. Tools need to be context-specific in order to be valid and useful.
Cornielje et al. [13]	Need to combine qualitative and quantitative methods: “Evaluations should be as systematic and objective as possible; but as participatory and subjective where necessary.” (p.40) Best to develop indicators that meet the specific needs of the program.
Chung et al. [20]	Structure interviews and observations using a template of key points that should be evaluated.
Evans et al. [22]	Use of a tracer approach to document the quality of the process of medical rehabilitation within CBR, including individual interviews, observations, and file review. They acknowledge that it requires a lot of concentration and exhaustive note taking.
Kuipers and Harknett [6]	Best to use more than one method and multiple sources to enable triangulation. Participatory techniques are particularly well suited for CBR as they facilitate the participation of people with disabilities.
Mannan and Turnbull [23]	Propose using quality of life scales.
Pal and Chaudhury [25]	Suggest using a parental adjustment measure with a five-point Likert scale. Tools need to be context-specific in order to be valid and useful.
Sharma [29]	Propose that different qualitative methods of data collection can be relevant (ex: interviews, focus groups, nominal groups, diaries, participatory techniques). Good moderators, the use of audiorecording, and homogeneous groups are suggested to optimize the use of focus groups. Individual interviews are frequently used, and allow the gathering of more personalized data although they are time consuming.
Sharma [30]	Recommends using focus groups even more often for evaluation and instrument development.
Sharma [26]	Best to use more that one method to enable triangulation. In the CBR literature, methods used to collect and analyze the information are frequently not explicitly described upon.
WHO and IDC [17]	Qualitative methodologies are relevant and include: focus groups, interviews, observations, document review, questionnaires, nominal groups, videos or photographs analysis, and diaries.
WHO et al. [2]	Best to use more than one method to collect information in order to enable triangulation. Both qualitative and quantitative methodologies are relevant. They suggest choosing from these methodologies: focus groups, interviews, observations, document review, questionnaires, record review, individual assessments, or surveys.
Velema et al. [16]	Proposed two flow charts and tested their use; one to look at service delivery and another one to examine the role of the environment. Concluded that these could be powerful tools to facilitate the process as they covered most elements of the reports. However, they also realized that information about organizational competency, linkages with other programs and sectors, quality of services, sources of incomes, and changes in community attitudes were not picked up by the flow charts, but were often required during program evaluation.
Wirz and Thomas [27]	Mixed methods and quantitative methodologies have a lot of potential to contribute to demonstrating the effectiveness of CBR. Vital need for a bank of quantitative indicators in CBR evaluations.
Zhao and Kwok [18]	Combine qualitative and quantitative methods. Include archival data and first hand field findings. Try to avoid functional assessments methods used in rehabilitation centers (too sophisticated and hard to master by community staff). Methods need to be simple, practical, and easy to use. Provide suggestions of tables focusing on the ability to do the activity or not, in association with the principal difficulty (e.g., hearing).

Discussion

The review conducted provides insight into what should be considered best practices in CBR program evaluations. First, the need for the evaluative process to be focused on taking action in light of the results is certainly congruent with the utility and propriety standards for effective evaluations and the empowerment principle in CBR [2,7,14]. However, it appears that only Boyce and Ballantyne [5] have emphasized the need for the process itself to be empowering, so that the local community can become the leader of their evaluation. Based on our field experiences in CBR, we believe that CBR would gain from applying some of the principles proposed by Fetterman and Wandersman [31] to design and engage in empowering evaluations that would foster greater community ownership and sustainability.

Second, the lack of consensus on which classification model to use contributes to a very fragmented evidence-base for CBR, as there is no common language [4]. The authors of three articles suggest that CBR needs a common framework from which to derive outcomes for evaluations [15,27,28] while two other groups propose that a localized framework would be better suited to a bottom-up approach like CBR [19,20]. We propose that evaluators need to join forces towards a common framework enabling the combination of findings in the quest for evidence for CBR effectiveness, but that this framework must be clearly defined, comprehensive, and flexible to enable local adaptability. We agree with Thomas [15] that the CBR matrix could be an important piece of this framework since it was developed with the contributions of experts and managers of the field, and because each element is now explicitly described in the CBR Guidelines. The other frameworks available in the literature could help determine the most relevant evaluation questions while providing additional ideas for the process and outcome measures that can be appropriate for the particular context. Table 4 presents a proposition for a shared framework for program evaluation in CBR. It integrates the CBR matrix and the highlights from the other frameworks. It is structured in the respect of the hierarchy of evaluations [32] in order to facilitate the development of effective evaluations. The idea would be situate evaluations on both axes, the component and element of the CBR matrix evaluated, and the type(s) of question(s) studied (i.e., relevance, process, outcomes or cost). Further validation with experts and on the field is now needed.

Table 4. Proposed framework for program evaluation and evaluative research in CBR.

	Relevance of the program: Fit between program and community	Implementation process: How the program actually operates	Outcomes: Desirable and unintended	Cost: Benefit and effectiveness
	Context, policies, attitudes, needs, services	Activities, strategies, strengths, weaknesses, what the program does to be sustainable and efficient	Short, medium, and long term outcomes for: people with disabilities, families, and communities	Costs for achieving a given outcome, for achieving tangible results
CBR Matrix
Health
Education
Livelihood
Social
Empowerment

Although qualitative methods have dominated the scene in CBR program evaluation for now, and remain highly relevant, the need to incorporate quantitative indicators to capture the progress made by people participating in CBR programs is identified explicitly in three articles [13,27] and is certainly congruent with a quest for greater accuracy in evaluations. The challenge is to determine how to do so without losing flexibility in evaluations, and without renouncing to the richness and power that qualitative methodologies can have in engaging local communities. That being said, it is clear that the most important is for the methods to be suitable for the local community.

Most of the recommendations from the documents included in this review were highly congruent with the CBR principles of inclusion and participation, and the context of its implementation in resource-poor areas, but not always completely so with the empowerment principle and with the standards for effective evaluations and good research. This weakness, along with the lack of common language and framework in CBR before the publication of the CBR Guidelines, certainly has posed serious threats to the recognition and advancement of this strategy. Even though this review does not provide level 1 evidence for one type of evaluation over another, it certainly offers insights into what could be considered best practices in the field and highlights the need to move towards a culture of evaluation where reflection and action are incorporated in the regular activities of CBR programs. We believe that CBR now requires best practice guidelines for program evaluation, clarifying what is the proper balance between community development and standards for good evaluation and research. Ideally, these guidelines would be based on the evidence found in this systematic review and consensus among a group of experts, and would suggest a clear, detailed, and empowering process to follow, with clear links to a shared framework and potential methods of data collection. A field validation of the guidelines would be essential to ensure applicability.

Conclusion

In conclusion, we strongly believe that it is more than time for the CBR community to engage in program evaluation and in the production of best practice guidelines for accountable and empowering evaluations. Since moderate levels of evidence in favor of particular practices are available, experts in program evaluation in CBR need to come to a consensus on what are best evaluative practices in the context of CBR. Further research and evaluation on the conditions under which CBR programs are most effective for different populations is definitely needed, in the respect of both CBR principles and standards for effective research and evaluation. Hopefully, these can be situated within the framework proposed in this manuscript. We wish that this review provided guidance on how to demonstrate the effectiveness of this strategy, in order to foster the institutionalization of existing CBR programs showing positive results and the development of new CBR programs to address the needs of the population living with a disability in resource-poor areas.

Declaration of interest

This article is included in the first author’s doctoral thesis in rehabilitation sciences at the University of Ottawa, Canada. She is supported financially by a Vanier Scholarship from the Canadian Institutes of Health Research, an Excellence Scholarship from the University of Ottawa and a Doctoral scholarship from the Canadian Occupational Therapy Foundation. She also received an Ontario Graduate Scholarship from the Ministry of Training, Colleges and Universities at the beginning stages of the preparation of this manuscript. The authors report no conflicts of interest.

Acknowledgements

Special thanks go to Jessica Morin who helped in the selection and evaluation of the articles.

Appendix A

Table A1. Characteristics of the included articles and highlights of the quality assessments.

				Highlights of the quality assessment
Article	Type of article	Objective	Methods	Rating: consensus (tool)	Congruence with evaluation and research standards	Congruence with CBR principles
Adeoye et al. [19]	Observational study (qualitative)	Develop an evaluation tool for CBR	Qualitative analysis of core elements from the joint position statement (JPS, 2004) Followed by a case study in Uganda (interviews, focus groups, document review).	CASP: 8/10	(−) Tool is only useful in one specific context in Uganda (−) Little information about the actual tool (+) Feasible to implement (+) Accurate in local context	(+) Grounded in CBR values and terminology through the JPS (+) Participation of the local community
Boyce and Ballantyne [5]	Editorial	Suggest ways to develop CBR through evaluation	X	X	(−) Accuracy: Not consensus among a group of experts, nor opinion of respected authorities	(+) Process suggested is coherent with CBR values of inclusion, participation and empowerment
Chung et al. [20]	Observational study (qualitative)	Development of a framework for CBR evaluations in China	Qualitative analysis of core elements of CBR defined from key texts and reports of CBR evaluations Tested for relevance and appropriateness in five Chinese CBR programs (interviews, observations, document review)	CASP: 9/10	(−) Only tested in China, little generalization possible	(+) Development process is congruent with inclusion and participation (+) Content is closely associated with CBR concepts (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Cornielje et al. [13]	Editorial	Share lessons learned during evaluations of CBR programs	X	X	(+) Lessons learned based on more than one experience (−) Accuracy: Not consensus among a group of experts, nor opinion of respected authorities (+) Accuracy: propose to combine objective and subjective measures	(+) Lessons are coherent with CBR philosophy
Evans et al. [22]	Observational study (qualitative)	Propose tracer approach to evaluate quality of CBR medical rehabilitation	Application of the method in West Bengal, India (document review, observations, interviews)	CASP: 7/10	(−) Lack of clarity in the article (−) Not totally clear if the tracer approach was developed and-or tested with CBR programs or other types of rehabilitation services	(+) Relevance to the field of CBR and respectful of context (low-cost)
Finkenflugel et al. [28]	Literature review	Examine the classification models used for evaluations in CBR	Used search results of previous study [4] and complemented with other documents	AMSTAR: 3/11	(−) Poor methodology (relied on old search, inclusion and exclusion criteria are not clear, no quality assessment for the included articles) (−) Lack of integration of the findings of the different articles reviewed	(+) Relevance of study to the field of CBR
Finkenflugel et al. [4]	Literature review	Establish the evidence base for CBR	Search in five databases from 1978–2002 Only articles in English and reporting CBR experiences in low- and middle-income countries	AMSTAR: 5/11	(+) Comprehensive search (−) Poor account of the quality of included articles	(+) Clear relevance for the field of CBR (−) Confusion between CBR and rehabilitation in the community showed in the included articles
Kuipers and Harknett [6]	Editorial	Contribute to the debate about evidence-based practice in CBR	X	X	(−) Does not present consensus among a group of experts, nor opinion of respected authorities (−) Arguments for their propositions are weak	(+) Congruent with CBR values and the context of its implementation
Mannan and Turnbull [23]	Literature review	Examine literature on evaluations in CBR	Not provided (search, inclusion criteria)	AMSTAR: 2/10	(−) Search strategy and inclusion criteria are not provided (+) Utility: suggestion of tools for outcome measurement	(+) Congruence of propositions with CBR the need to include people with disabilities and their families
McColl and Paterson [24]	Observational study (qualitative)	Develop a framework for CBR programs	Key informants representing CBR programs were interviewed	AMSTAR: 8/11	(+) Rigorous development and involvement of stakeholders from different programs	(+) Relevance for CBR (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Mitchell [7]	Literature review	Review research on CBR	Not provided (search, inclusion criteria)	AMSTAR: 1/11	(−) Methodology for the review is not provided (+) Suggestions congruent with utility and propriety evaluation standards (reporting the results)	(+) Relevance and coherence of propositions with current CBR definitions
Pal and Chaudhury [25]	Observational study (quantitative)	Develop an instrument to measure parental adjustment in a CBR program in India	Development of the measure from attitude statements heard in the area Interviews with parents of children attending the program using these statements to which they answered using a five-point Likert scale	STROBE: 16/22	(+) Accuracy: Rigorous analysis (−) Accuracy: Further validation is needed (−) Utility: Generalization is limited as the tool is relevant to the Indian rural context only	(+) Congruence with bottom-up approach in CBR as it is grounded in local experience
Price and Kuipers [14]	Editorial	Propose action research as suitable for CBR evaluations	X	X	(−) Accuracy: Does not present consensus among a group of experts, nor opinion of respected authorities (+) Excellent respect of the utility standard as the goal is program improvement	(+) Congruence with CBR values of participation and empowerment and its context (+) Proposition promotes local acceptability and empowerment
Sharma [29]	Literature review	Review literature on the role of qualitative approaches in CBR evaluations	Not provided (search, inclusion criteria)	AMSTAR: 0/11	(−) Metholodogy for search is not described (+) Feasibility of using qualitative methodologies in CBR evaluations (+) Utility of qualitative approaches to gather the perspectives of people	(+) Congruence of proposition with context of implementation of CBR
Sharma [30]	Literature review	Examine the extent to which focus groups have been used in CBR	Search on MEDLINE, inclusion and exclusion criteria provided.	AMSTAR: 2/11	(+) Relevance of review to give insights into data collection methods	(+) Using focus groups for evaluation and intervention is congruent with empowerment (−) Inclusion of many articles associated with rehabilitation programs in the community
Sharma [26]	Literature review	Analyze the extent to which CBR has been evaluated	Search on MEDLINE, for CBR and evaluation. Qualitative analysis using a framework of Strengths, Weaknesses, Opportunities and Threats (SWOT)	AMSTAR: 3/11	(+) Suggestions congruent with scientific standards (−) Review is restricted to one database	(+) Congruence of suggestions with CBR values of participation (−) Confusion between CBR and rehabilitation in the community showed in the included articles
Thomas [15]	Editorial	Contribute to debates around CBR and present the CBR Guidelines	X	X	(−) Does not present consensus among a group of experts, nor opinion of respected authorities	(+) Relevance of propositions for optimal use of the Guidelines
Velema et al. [16]	Observational study (qualitative)	Validate the use of flow charts in evaluations of CBR programs	Comparison of information reported in five program evaluations reports with information that can be collected with flow charts	CASP: 7/10	(−) All reports studied from evaluations conducted by the authors (+) Utility of flow charts to structure data collection and gather most relevant information	(+) Relevance of suggesting a tool to structure the evaluative process (−) Lack of clear distinction between rehabilitation programs in the community and CBR
WHO and IDC [17]	Guidelines	Produce guidelines on self-assessment and monitoring of CBR	Not clear: based on key texts and mention that they were prepared with the collaboration of the WHO and IDC	AGREE II global rating: 2/7	(+) Scope and purpose of guidelines clearly described and relevant (−) Weak methodology to produce the guidelines	(+) Self-reflection for assessment is congruent with empowering programs to take the lead of their evaluations
WHO et al. [2]	Guidelines	Strengthen CBR programs	Based on consensus among a group of experts on the CBR matrix which provided the structure At least two authors involved for each section Extensive field validation Review planned in 2020	AGREE II global rating: 5/7	(−) Not sufficiently grounded in evidence (−) Section on program evaluation is limited (+) The views of people with disabilities have been sought (propriety) and emphasize the importance of sharing findings and taking action (utility)	(+) High relevance for CBR implementation (+) Clear definitions and structured around a potential common matrix (+) Grounded on experience around the world (+) Accessible in 3 languages
Wirz and Thomas [27]	Observational study (qualitative)	Review usefulness of indicators proposed in CBR evaluations	Two authors read 10 evaluation reports (both published and privately circulated) Analyzed the information included to generate categories and indicators.	CASP: 7/10	(+) Rigorous analysis by two independent authors (−) Choice of evaluation reports is not explained	(−) Propose indicators associated with only 1/3 dimensions (+) Bank of indicators can be useful to CBR evaluations (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Zhao and Kwok [18]	Guidelines	Provide guidance for accountable evaluations	Reviewed key texts from the literature and consulted colleagues and people with disabilities.	AGREE: 3/10	(−) Not clear how best practices were generated (+) Suggestions are congruent with utility, feasibility, accuracy standards for evaluation	(+) Proposition concerning self-assessment is congruent with the participation and empowerment principles

Appendix B

List of excluded studies (available upon request sent to corresponding author).