A systematic review on how to conduct evaluations in community-based rehabilitation

Figures & data

Figure 1. Flow chart for the selection of the articles.

Table 1. Findings about the characteristics of the evaluation process for CBR.

	Characteristics of the evaluation process
	People who should be involved	Steps to follow	Time to evaluate
Boyce and Ballantyne [5]	People with disabilities and other key stakeholders in the community should be involved in all the steps. They should be empowered through coaching and training. Evaluations should be as participatory as possible.	Decide whether or not to evaluate, determine the objectives and indicators, select coordinators, choose methods, write the proposal, prepare and test the methods, collect the information, analyze the information, and report the results.	–
Cornielje et al. [13]	–	–	Importance of baseline survey (before implementation).
Mitchell [7]	–	The last step should be to draw attention to CBR by reporting the results publicly.	–
Price and Kuipers [14]	The participation of the community is essential to ensure local relevance.	The process should be iterative and involve cycles of action and reflection.	–
Thomas [15]	–	–	Importance of collecting baseline data (before implementation).
Velema et al. [16]	–	–	Importance of collecting baseline (before implementation) and follow-up data on people who completed the program.
WHO and IDC [17]	CBR programs should engage in self-reflection (self-assessment).	The last step should be to report findings and meet community members to make decisions to improve the program.	Incorporate monitoring and self-assessment in regular activities of CBR programs.
WHO et al. [2]	Both internal and external evaluations can be valuable, but a combination of both approaches would ideally be used.	1. Focus the evaluation (purpose and first questions) 2. Collect information (to answer the questions) 3. Analyze the information and draw conclusions (statistical calculations or thematic analyses) 4. Share findings and take action (write report, meet community members, influence decisions, act on the results)	Importance of collecting baseline data when implementing CBR programs. Specific points in time during the project cycle are relevant for evaluation (e.g., midway through the implementation, immediately after completion of the project, some time afterwards).
Zhao and Kwok [18]	There are many possibilities, including self-evaluation, mutual evaluation (exchange between two CBR programs), higher authorities evaluation, and external evaluation.	1. Make a detailed evaluation plan 2. Collect materials and conducting investigation 3. Analyze results 4. Make evaluation report and give suggestion 5. Feedback of results and integration into practice	At regular intervals, including baseline survey before implementation, periodical evaluation during implementation, eventual evaluation when the program ends, and follow-up evaluation some time after.

Table 2. Frameworks proposed in the CBR literature and their associated process and outcome measures.

PROCESS: Framework categories and measures	OUTCOMES: Framework categories and measures
Adeoye et al. [19]
Implementation: role of people with disabilities, role of family, role  of community, role of health services, role of educational services,  role of vocational services, role of social services, challenges to  implementation ˆ Leadership role, formation of self-help groups, program coverage, participation in planning-implementing, access to health services, health education, home based care/intervention, access to educational services and equipments, inclusive education Policies, strategies, activities, and challenges to: community development, rehabilitation, equalization of opportunities, and social inclusion. ˆ Policies: mainstreaming, participation, implementation of laws and guidelines, implementation of disability policy ˆ Strategies: participation in planning-implementation, promoting community awareness, changing community attitudes, sign language training, promoting/advocating, increasing program coverage ˆ Activities: income-generating activities, sustainability, collecting baseline data, technical aids and services, participation in community, participation in family, employment of people with disabilities, personal development and skills, access to justice, access to employment, social integration, inclusive education, leadership role of people with disabilities ˆ Challenges: sustainability, unemployment, inadequate equipments, non-implementation of disability policies, inadequate access to CBR officials, communication barrier, discrimination, sexual abuse	*We placed the framework in the process category in order to represent the way they look at policies, strategies, activities and challenges. However, some elements may belong to both process and outcome evaluations if we pay attention to the verbatim used by the authors to provide supportive evidence.
Chung et al. [20]
CBR-related program content ˆ Advocacy, networking, involvement of relatives and families, of CBR manager, of CBR workers Participant empowerment/governance ˆ Leadership, self-help and mutual help, autonomy in program Community ownership ˆ Community support and recognition, national/government level support, collaboration and support among different sectors Program management and development ˆ Compliance to relevant service standards, ethical practices, management issues, sustainability, continuous growth and development	Participants outcomes ˆ Functional independence, education, economic independence, participation in family and community life, physical change, spiritual change, early detection and intervention
Cornielje et al. [13] *Categories originally proposed by Cornielje et al. [21]
Locus of power Commitment to involve others Type of response	Restoration of quality of life
Evans et al. [22]
Interpersonal quality Technical quality Management quality	–
Mannan and Turnbull [23]
–	Quality of life Family quality of life
McColl and Paterson [24]
CBR programs ˆ Aims ˆ Beneficiaries ˆ Strategies Supportive structure under CBR ˆ Human resources ˆ Structural resources ˆ Attitudes	–
Mitchell [7]
Service delivery Community involvement Technology transfer Organization and management	–
Pal and Chaudhury [25]
–	Parental adjustment ˆ Feeling of rejection and resettlement ˆ Positive feelings toward child ˆ Negative aspects of maternal mental health ˆ Feelings of guilt and self-blame
Sharma [26]
Strengths Weaknesses Opportunities Threats	–
WHO et al. [2] *Examples of outcomes for the 25 elements are available. Here, examples associated with the 5 components are presented.
Factors that promote sustainability Effective leadership, partnerships, community ownership, using local resources, considering cultural factors, building capacity, financial and political support.	Health: promotion, prevention, medical care, rehabilitation, assistive devices Ex.: Improved knowledge about good health, health sector awareness, access to health care and rehabilitation services, improved collaboration across sectors. Education: early childhood, primary, secondary and higher, non-formal, lifelong Ex.: Access to learning, accessibility of local schools, awareness of community that people with disabilities can learn, advocacy to facilitate inclusive education. Livelihood: skills development, self-employment, wage employment, financial services, social protection Ex.: Access to skills development, microfinance and social protection, equal work opportunities for women with disabilities, inclusion in poverty reduction strategies. Social: personal assistance, relationships-marriage and family, culture and arts, recreation-leisure and sports, justice Ex.: Access to personal assistance options, individual support plans, informal support for families, training availability for assistants. Empowerment: advocacy and communication, community mobilization, political participation, self-help groups, disabled people’s organizations Ex.: Informed choices and decisions, active participation in family and community, removal of barriers in community, creation of groups of people with disabilities.
WHO and IDC [17]
Transfer of skills to community level, program activities	Impact on individuals, community mobilization, opportunity for education, opportunity for work, involvement of disabled people
Wirz and Thomas [27]
Service delivery: program planning and management, financial and people management, training, sustainability	Maximizing potential: functional independence, education, economic independence, inclusion, leadership roles in the community, participation in/ownership of programs Ex.: Halting progression of impairment, improved mobility, improved ADL, improved communication skills, improved orientation and mobility skills training by parents, mobility aids, hearing aids, removal of physical barriers, reduced attitudinal barriers, special education or inclusive education attendance, skills and work placements acquisition, earnings through income-generation activities and self-employment, availability of credit groups, equal access to community services, leadership roles in the community, management of self-help groups. Environment: family attitudes, family involvement, community attitudes, inclusion of people with disabilities
Zhao and Kwok [18]
Evaluation of management: government role, NGO role, other sectors, communities involvement Evaluation of implementation: services in medical rehabilitation, in educational rehabilitation, in vocational rehabilitation	Evaluation of social impacts of CBR: attitude changes

Table 3. Characteristics of the methods of data collection in CBR.

	Suggestions about the characteristics of the methods
Adeoye et al. [19]	Fifteen-statement questionnaire in which participants needed to say if they agreed or not with each on a five-point Likert scale, developed after qualitative interviews in the local community in Uganda. Tools need to be context-specific in order to be valid and useful.
Cornielje et al. [13]	Need to combine qualitative and quantitative methods: “Evaluations should be as systematic and objective as possible; but as participatory and subjective where necessary.” (p.40) Best to develop indicators that meet the specific needs of the program.
Chung et al. [20]	Structure interviews and observations using a template of key points that should be evaluated.
Evans et al. [22]	Use of a tracer approach to document the quality of the process of medical rehabilitation within CBR, including individual interviews, observations, and file review. They acknowledge that it requires a lot of concentration and exhaustive note taking.
Kuipers and Harknett [6]	Best to use more than one method and multiple sources to enable triangulation. Participatory techniques are particularly well suited for CBR as they facilitate the participation of people with disabilities.
Mannan and Turnbull [23]	Propose using quality of life scales.
Pal and Chaudhury [25]	Suggest using a parental adjustment measure with a five-point Likert scale. Tools need to be context-specific in order to be valid and useful.
Sharma [29]	Propose that different qualitative methods of data collection can be relevant (ex: interviews, focus groups, nominal groups, diaries, participatory techniques). Good moderators, the use of audiorecording, and homogeneous groups are suggested to optimize the use of focus groups. Individual interviews are frequently used, and allow the gathering of more personalized data although they are time consuming.
Sharma [30]	Recommends using focus groups even more often for evaluation and instrument development.
Sharma [26]	Best to use more that one method to enable triangulation. In the CBR literature, methods used to collect and analyze the information are frequently not explicitly described upon.
WHO and IDC [17]	Qualitative methodologies are relevant and include: focus groups, interviews, observations, document review, questionnaires, nominal groups, videos or photographs analysis, and diaries.
WHO et al. [2]	Best to use more than one method to collect information in order to enable triangulation. Both qualitative and quantitative methodologies are relevant. They suggest choosing from these methodologies: focus groups, interviews, observations, document review, questionnaires, record review, individual assessments, or surveys.
Velema et al. [16]	Proposed two flow charts and tested their use; one to look at service delivery and another one to examine the role of the environment. Concluded that these could be powerful tools to facilitate the process as they covered most elements of the reports. However, they also realized that information about organizational competency, linkages with other programs and sectors, quality of services, sources of incomes, and changes in community attitudes were not picked up by the flow charts, but were often required during program evaluation.
Wirz and Thomas [27]	Mixed methods and quantitative methodologies have a lot of potential to contribute to demonstrating the effectiveness of CBR. Vital need for a bank of quantitative indicators in CBR evaluations.
Zhao and Kwok [18]	Combine qualitative and quantitative methods. Include archival data and first hand field findings. Try to avoid functional assessments methods used in rehabilitation centers (too sophisticated and hard to master by community staff). Methods need to be simple, practical, and easy to use. Provide suggestions of tables focusing on the ability to do the activity or not, in association with the principal difficulty (e.g., hearing).

Table 4. Proposed framework for program evaluation and evaluative research in CBR.

	Relevance of the program: Fit between program and community	Implementation process: How the program actually operates	Outcomes: Desirable and unintended	Cost: Benefit and effectiveness
	Context, policies, attitudes, needs, services	Activities, strategies, strengths, weaknesses, what the program does to be sustainable and efficient	Short, medium, and long term outcomes for: people with disabilities, families, and communities	Costs for achieving a given outcome, for achieving tangible results
CBR Matrix
Health
Education
Livelihood
Social
Empowerment

Table A1. Characteristics of the included articles and highlights of the quality assessments.

				Highlights of the quality assessment
Article	Type of article	Objective	Methods	Rating: consensus (tool)	Congruence with evaluation and research standards	Congruence with CBR principles
Adeoye et al. [19]	Observational study (qualitative)	Develop an evaluation tool for CBR	Qualitative analysis of core elements from the joint position statement (JPS, 2004) Followed by a case study in Uganda (interviews, focus groups, document review).	CASP: 8/10	(−) Tool is only useful in one specific context in Uganda (−) Little information about the actual tool (+) Feasible to implement (+) Accurate in local context	(+) Grounded in CBR values and terminology through the JPS (+) Participation of the local community
Boyce and Ballantyne [5]	Editorial	Suggest ways to develop CBR through evaluation	X	X	(−) Accuracy: Not consensus among a group of experts, nor opinion of respected authorities	(+) Process suggested is coherent with CBR values of inclusion, participation and empowerment
Chung et al. [20]	Observational study (qualitative)	Development of a framework for CBR evaluations in China	Qualitative analysis of core elements of CBR defined from key texts and reports of CBR evaluations Tested for relevance and appropriateness in five Chinese CBR programs (interviews, observations, document review)	CASP: 9/10	(−) Only tested in China, little generalization possible	(+) Development process is congruent with inclusion and participation (+) Content is closely associated with CBR concepts (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Cornielje et al. [13]	Editorial	Share lessons learned during evaluations of CBR programs	X	X	(+) Lessons learned based on more than one experience (−) Accuracy: Not consensus among a group of experts, nor opinion of respected authorities (+) Accuracy: propose to combine objective and subjective measures	(+) Lessons are coherent with CBR philosophy
Evans et al. [22]	Observational study (qualitative)	Propose tracer approach to evaluate quality of CBR medical rehabilitation	Application of the method in West Bengal, India (document review, observations, interviews)	CASP: 7/10	(−) Lack of clarity in the article (−) Not totally clear if the tracer approach was developed and-or tested with CBR programs or other types of rehabilitation services	(+) Relevance to the field of CBR and respectful of context (low-cost)
Finkenflugel et al. [28]	Literature review	Examine the classification models used for evaluations in CBR	Used search results of previous study [4] and complemented with other documents	AMSTAR: 3/11	(−) Poor methodology (relied on old search, inclusion and exclusion criteria are not clear, no quality assessment for the included articles) (−) Lack of integration of the findings of the different articles reviewed	(+) Relevance of study to the field of CBR
Finkenflugel et al. [4]	Literature review	Establish the evidence base for CBR	Search in five databases from 1978–2002 Only articles in English and reporting CBR experiences in low- and middle-income countries	AMSTAR: 5/11	(+) Comprehensive search (−) Poor account of the quality of included articles	(+) Clear relevance for the field of CBR (−) Confusion between CBR and rehabilitation in the community showed in the included articles
Kuipers and Harknett [6]	Editorial	Contribute to the debate about evidence-based practice in CBR	X	X	(−) Does not present consensus among a group of experts, nor opinion of respected authorities (−) Arguments for their propositions are weak	(+) Congruent with CBR values and the context of its implementation
Mannan and Turnbull [23]	Literature review	Examine literature on evaluations in CBR	Not provided (search, inclusion criteria)	AMSTAR: 2/10	(−) Search strategy and inclusion criteria are not provided (+) Utility: suggestion of tools for outcome measurement	(+) Congruence of propositions with CBR the need to include people with disabilities and their families
McColl and Paterson [24]	Observational study (qualitative)	Develop a framework for CBR programs	Key informants representing CBR programs were interviewed	AMSTAR: 8/11	(+) Rigorous development and involvement of stakeholders from different programs	(+) Relevance for CBR (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Mitchell [7]	Literature review	Review research on CBR	Not provided (search, inclusion criteria)	AMSTAR: 1/11	(−) Methodology for the review is not provided (+) Suggestions congruent with utility and propriety evaluation standards (reporting the results)	(+) Relevance and coherence of propositions with current CBR definitions
Pal and Chaudhury [25]	Observational study (quantitative)	Develop an instrument to measure parental adjustment in a CBR program in India	Development of the measure from attitude statements heard in the area Interviews with parents of children attending the program using these statements to which they answered using a five-point Likert scale	STROBE: 16/22	(+) Accuracy: Rigorous analysis (−) Accuracy: Further validation is needed (−) Utility: Generalization is limited as the tool is relevant to the Indian rural context only	(+) Congruence with bottom-up approach in CBR as it is grounded in local experience
Price and Kuipers [14]	Editorial	Propose action research as suitable for CBR evaluations	X	X	(−) Accuracy: Does not present consensus among a group of experts, nor opinion of respected authorities (+) Excellent respect of the utility standard as the goal is program improvement	(+) Congruence with CBR values of participation and empowerment and its context (+) Proposition promotes local acceptability and empowerment
Sharma [29]	Literature review	Review literature on the role of qualitative approaches in CBR evaluations	Not provided (search, inclusion criteria)	AMSTAR: 0/11	(−) Metholodogy for search is not described (+) Feasibility of using qualitative methodologies in CBR evaluations (+) Utility of qualitative approaches to gather the perspectives of people	(+) Congruence of proposition with context of implementation of CBR
Sharma [30]	Literature review	Examine the extent to which focus groups have been used in CBR	Search on MEDLINE, inclusion and exclusion criteria provided.	AMSTAR: 2/11	(+) Relevance of review to give insights into data collection methods	(+) Using focus groups for evaluation and intervention is congruent with empowerment (−) Inclusion of many articles associated with rehabilitation programs in the community
Sharma [26]	Literature review	Analyze the extent to which CBR has been evaluated	Search on MEDLINE, for CBR and evaluation. Qualitative analysis using a framework of Strengths, Weaknesses, Opportunities and Threats (SWOT)	AMSTAR: 3/11	(+) Suggestions congruent with scientific standards (−) Review is restricted to one database	(+) Congruence of suggestions with CBR values of participation (−) Confusion between CBR and rehabilitation in the community showed in the included articles
Thomas [15]	Editorial	Contribute to debates around CBR and present the CBR Guidelines	X	X	(−) Does not present consensus among a group of experts, nor opinion of respected authorities	(+) Relevance of propositions for optimal use of the Guidelines
Velema et al. [16]	Observational study (qualitative)	Validate the use of flow charts in evaluations of CBR programs	Comparison of information reported in five program evaluations reports with information that can be collected with flow charts	CASP: 7/10	(−) All reports studied from evaluations conducted by the authors (+) Utility of flow charts to structure data collection and gather most relevant information	(+) Relevance of suggesting a tool to structure the evaluative process (−) Lack of clear distinction between rehabilitation programs in the community and CBR
WHO and IDC [17]	Guidelines	Produce guidelines on self-assessment and monitoring of CBR	Not clear: based on key texts and mention that they were prepared with the collaboration of the WHO and IDC	AGREE II global rating: 2/7	(+) Scope and purpose of guidelines clearly described and relevant (−) Weak methodology to produce the guidelines	(+) Self-reflection for assessment is congruent with empowering programs to take the lead of their evaluations
WHO et al. [2]	Guidelines	Strengthen CBR programs	Based on consensus among a group of experts on the CBR matrix which provided the structure At least two authors involved for each section Extensive field validation Review planned in 2020	AGREE II global rating: 5/7	(−) Not sufficiently grounded in evidence (−) Section on program evaluation is limited (+) The views of people with disabilities have been sought (propriety) and emphasize the importance of sharing findings and taking action (utility)	(+) High relevance for CBR implementation (+) Clear definitions and structured around a potential common matrix (+) Grounded on experience around the world (+) Accessible in 3 languages
Wirz and Thomas [27]	Observational study (qualitative)	Review usefulness of indicators proposed in CBR evaluations	Two authors read 10 evaluation reports (both published and privately circulated) Analyzed the information included to generate categories and indicators.	CASP: 7/10	(+) Rigorous analysis by two independent authors (−) Choice of evaluation reports is not explained	(−) Propose indicators associated with only 1/3 dimensions (+) Bank of indicators can be useful to CBR evaluations (−) Now that CBR guidelines provide clear definitions for matrix, less relevant
Zhao and Kwok [18]	Guidelines	Provide guidance for accountable evaluations	Reviewed key texts from the literature and consulted colleagues and people with disabilities.	AGREE: 3/10	(−) Not clear how best practices were generated (+) Suggestions are congruent with utility, feasibility, accuracy standards for evaluation	(+) Proposition concerning self-assessment is congruent with the participation and empowerment principles

Boyce W, Ballantyne S. Developing CBR through evaluation. Asia Pacific Disabil Rehabil J 2000;2:69–83

 Google Scholar

Cornielje H, Velema JP, Finkenflügel H. Community based rehabilitation programmes: monitoring and evaluation in order to measure results. Leprosy Rev 2008;79:36–49

 PubMed Web of Science ®Google Scholar

Mitchell R. The research base of community-based rehabilitation. Disabil Rehabil 1999;21:459–68

 PubMed Web of Science ®Google Scholar

Price P, Kuipers P. CBR action research: current status and future trends. Asia Pacific Disabil Rehabil J 2000;2:55–68

 Google Scholar

Thomas M. Reflections on community-based rehabilitation. Psychol Dev Soc 2011;23:277–91

 Google Scholar

Velema J, Finkenflugel H, Cornielje H. Gains and losses of structured information collection in the evaluation of ‘rehabilitation in the community’ programmes: ten lessons learnt during actual evaluations. Disabil Rehabil 2008;30:396–404

 PubMed Web of Science ®Google Scholar

WHO and IDC [Internet]. Guidelines for conducting, monitoring and self-assessment of community-based rehabilitation programmes: using evaluation information to improve programmes Geneva: WHO Press; 1996. Available from: http://www.who.int/disabilities/cbr/en/ [last accessed 12 Apr 2013]

 Google Scholar

World Health Organization (WHO), United Nations Educational, Scientific and Cultural Organization (UNESCO), International Labour Organization (ILO), International Disability and Development Consortium (IDDC). [Internet]. Community-based rehabilitation: CBR Guidelines. Geneva: WHO Press; 2010. Available from: http://www.who.int/disabilities/cbr/en/ [last accessed 12 Apr 2013]

 Google Scholar

Zhao T, Kwok J. Evaluating community based rehabilitation: guidelines for accountable practice. 1999. Available from: http://unipd-centrodirittiumani.it/public/docs/34011_planning.pdf [last accessed 12 Apr 2013]

 Google Scholar

Adeoye A, Seeley J, Hartley S. Developing a tool for evaluating community-based rehabilitation in Uganda. Disabil Rehabil 2011;33:1110–24

 PubMed Web of Science ®Google Scholar

Chung EY, Packer TL, Yau M. A framework for evaluating community-based rehabilitation programmes in Chinese communities. Disabil Rehabil 2011;33:1668–82

 PubMed Web of Science ®Google Scholar

Cornielje H, Nicholls P, Velema J. Making sense of rehabilitation projects: classification by objectives. Leprosy Rev 2000;71:472--85

 Google Scholar

Evans PJ, Zinkin P, Harpham T, et al. Evaluation of medical rehabilitation in community based rehabilitation. Soc Sci Med 2001;53:333–48

 PubMed Web of Science ®Google Scholar

Mannan H, Turnbull AP. A review of community based rehabilitation evaluations: quality of life as an outcome measure for future evaluations. Asia Pacific Disabil Rehabil J 2007;18:29–45

 Google Scholar

McColl MA, Paterson J. A descriptive framework for community-based rehabilitation. Can J Rehabil 1997;10:297–306

 Google Scholar

Pal DK, Chaudhury G. Preliminary validation of a parental adjustment measure for use with families of disabled children in rural india. Child: Care Health Dev 1998;24:315–24

 PubMed Web of Science ®Google Scholar

Sharma M. Evaluation in community based rehabilitation programmes: a strengths, weaknesses, opportunities and threats analysis. Asia Pacific Disabil Rehabil J 2007;18:46–62

 Google Scholar

Wirz S, Thomas M. Evaluation of community-based rehabilitation programmes: a search for appropriate indicators. Int J Rehabil Res 2002;25:163–71

 PubMed Web of Science ®Google Scholar

Kuipers P, Harknett S. Guest editorial. Considerations in the quest for evidence in community based rehabilitation. Asia Pacific Disabil Rehabil J 2008;19:3–14

 Google Scholar

Sharma M. Viable methods for evaluation of community-based rehabilitation programmes. Disabil Rehabil 2004;26:326–34

 PubMed Web of Science ®Google Scholar

Sharma M. Using focus groups in community based rehabilitation. Asia Pacific Disabil Rehabil J 2005;16:41–50

 Google Scholar

Finkenflugel H, Cornielje H, Velema J. The use of classification models in the evaluation of CBR programmes. Disabil Rehabil 2008;30:348–54

 PubMed Web of Science ®Google Scholar

Finkenflugel H, Wolffers I, Huijsman R. The evidence base for community-based rehabilitation: a literature review. Int J Rehabil Res 2005;28:187–201

 PubMed Web of Science ®Google Scholar