A systematic review to identify system-related elements that can be used to evaluate community-based rehabilitation (CBR) programmes

Introduction

Community-based rehabilitation (CBR) was launched in 1978 as a model for provision of rehabilitation services in remote areas in developing countries [1,2]. The majority of people living with disabilities (PLWDs), about 15% of the world’s population, live in developing countries [3,4] where rehabilitation services are inaccessible [5,6]. The absence of effective interventions that enable adequate rehabilitation and integration of PLWDs affects their livelihoods, those of their families and community [7], making disability a community issue that requires adequate stakeholder participation to address the factors affecting PLWDs.

Existing system barriers to accessing rehabilitation services enlarge the gap between the need for and provision of rehabilitation services [5,8]. This mismatch adversely affects the functional and health states of PLWDs denying them the fundamental right to community participation [5,9]. In order to address the inequalities faced by PLWDs, coordinated, evidence-informed interventions need to be put in place at all levels of the system [5,9]. Systems thinking if applied to CBR, will lead to both technical and allocative efficiency leading to better health outcomes [10]. Systems analysis requires consideration of the interaction of the system elements and the context in which the programme operates [11]. The elements of a system are the different components that interlink to define the behaviour of the system [11]. The organisation of CBR programmes is a product of the system that is in place and affects the extent to which they achieve their objectives [12,13].

Although CBR has been practiced for a long time and is implemented in more than 90 countries, there is very little evidence that demonstrates its effectiveness [4,12,14]. Existing reports describe the CBR process rather than evaluate effectiveness [15,16]. Where research has been done, indicators that assess impact are lacking [5,16,17]. There is little evidence in the form of systematic reviews; hence, the field of CBR has been described as ‘data rich but evidence poor’ [16,18,19]. The lack of evidence on the effectiveness of CBR programmes may eventually lead to reduced support for the CBR strategy [20]. Consequently, there is need for the development of robust indicators for evaluation and research purposes to ensure continued support for CBR [17,20]. Critics of the common set of indicators for CBR evaluation have singled out the heterogeneity of CBR programmes as a limitation to their use [21,22]. Others, however, acknowledge that there are different areas of focus in CBR but suggest a comprehensive set of indicators from which one can choose to enable the evaluation of the different components of CBR [4,16].

This systematic review was aimed at identifying system-related elements that have been used in CBR programme evaluations in order to contribute to an evaluation standard for CBR. The objectives were to determine the types of elements used in CBR evaluation literature and to document the system-related elements that were used to evaluate CBR programmes.

Methods

A qualitative systematic review was done. A systematic search was conducted on four data bases (SCOPUS, PubMed, PsycINFO and Medline) and on the WHO website for the period between 2005 and 2015. Grey literature was searched on Google scholar. Keywords used were community-based rehabilitation, evaluation, indicators, framework, monitoring and disability. Articles in English on CBR evaluation from all over the world were included. Systematic and literature reviews, book chapters and articles from newspapers were excluded.

Titles of articles obtained though the literature search were read through by the principal reviewer. Articles that mentioned CBR evaluation had their abstracts retrieved. Abstracts were read through independently by principal reviewer and two secondary reviewers. Each reviewer selected appropriate abstracts. Full articles were then retrieved and read through by the three reviewers. Articles relevant to community-based rehabilitation evaluation were selected. Disagreements were discussed among the three reviewers and agreement reached for inclusion or exclusion of the abstracts and papers under consideration. References of the chosen articles were then screened to identify more articles for inclusion.

Methodological quality was assessed by the researcher and checked by a second person using the Critical Appraisal Skills Programme for qualitative studies and the Strengthening the Reporting of Observational Studies checklist for observational studies. The Evans appraisal tool was used to determine the level of evidence for the included articles [23]. Inductive content analysis was used to identify elements used in CBR evaluation [24]. Emerging themes were identified manually, using open coding and recorded. Constant comparison was used to identify and classify the elements into categories [25,26]. A three-step process was used:

1. Principal reviewer identified and categorised emerging themes into higher order headings [25,27].

2. Validation was done by researcher and two secondary reviewers. A second process of constant comparison and categorisation was done to come up with a final list of elements and categories.

3. Abstraction process was done by the panel of three reviewers to identify major themes relating to system-related elements for CBR emerging from the articles. A list of identified system related elements from the articles was recorded [4,18,24,25,28].

The system-related elements were compared to existing frameworks for validation.

Results

Articles were selected as shown in Figure 1, and a total of 1734 titles were perused from which 49 abstracts were obtained. Thirty-four were excluded for various reasons and 16 full articles were obtained. Duplicates were removed leaving nine articles for inclusion in the review. Three articles were obtained from the reference lists of the included studies bringing the total studies for possible review to 12.

Figure 1. Flow chart for article selection.

The level of evidence for studies was good for three studies, fair for seven and poor for two studies. These two studies with poor evidence were excluded leaving a sample size of 10. Table 1 gives a summary of the overall ranking of the 12 studies.

Table 1. Ranking of studies across all scales (n = 12).

Author	Study design	Assessment tool	Score	Level of evidence (Evans [23])
Chung et al. [20]	Qualitative study	CASP	10	Good
Roe et al. 2014 [32]	Qualitative study	CASP	4	Poor (poor methodological quality)
Mijnarends et al. 2011 [33]	Qualitative case study	CASP	9	Fair
Raja et al. 2008 [34]	Qualitative case study	CASP	8	Good
Mol et al. 2014 [35]	Qualitative study	CASP	10	Fair
Grandisson et al. 2014 [14]	Qualitative study	CASP	10	Fair
Adeoye et al. 2011 [36]	Qualitative study	CASP	10	Fair
Reimer and LeNavenec 2005 [37]	Qualitative study	CASP	6	Fair (fair methodological quality)
Chappell and Johannsmeier 2009 [38]	Qualitative	CASP	10	Fair
Higashida 2014 [39]	Qualitative	CASP	10	Fair
Cornieljie et al. 2008 [40]	Expert opinion	None		Poor
Hartman-Maeir et al. 2007 [41]	Quantitative study	STROBE CHECKLIST	19	Good

Inductive content analysis to identify elements used for CBR evaluation was carried out on the 10 articles. Forty-four elements were identified by the principal reviewer and checked by two other reviewers (Table 2).

Table 2. Showing the identified elements and their frequency across reports (n = 10).

Element	Chung et al. 2011 [20]	Mijnarends et al. 2011 [33]	Raja et al. 2008 [34]	Mol et al. 2014 [35]	Grandisson et al. 2014 [14]	Adeoye et al. 2011 [36]	Reimer and LeNavenec 2005 [37]	Hartman-Maeir et al. 2007 [41]	Chappell and Johannsmeier 2009 [38]	Higashida 2014 [39]
Communication/information system	✓	✓				✓			✓	✓
Training empowerment for field staff	✓	✓	✓		✓	✓			✓
Impact on disabled people/improved quality of life	✓	✓	✓	✓	✓	✓	✓	✓	✓
Impact on families of PWDs	✓	✓	✓	✓	✓	✓	✓	✓	✓
Family involvement in rehabilitation	✓	✓	✓	✓		✓	✓		✓
Impact on community	✓	✓	✓	✓	✓	✓	✓		✓
Changing community attitudes	✓	✓		✓		✓			✓
Impact of training on staff and community/capacity building	✓	✓	✓	✓	✓
Staff performance	✓	✓	✓		✓				✓	✓
Good balance between income and expenditure	✓	✓	✓		✓	✓
Accessibility of services	✓		✓		✓	✓	✓
Utilisation of services	✓					✓
Empowerment of people with disabilities	✓	✓	✓	✓	✓	✓				✓
Participatory process	✓	✓	✓	✓	✓					✓
Structure of management of human resources	✓	✓				✓
Networking/collaboration	✓	✓	✓		✓				✓	✓
Advocacy	✓	✓	✓	✓		✓			✓	✓
Addressing poverty	✓	✓	✓	✓	✓	✓			✓	✓
Early detection and intervention	✓			✓	✓
Stable human resources	✓	✓	✓			✓			✓	✓
Continuous entry of new service users	✓				✓
Quality assurance/M&E	✓	✓	✓		✓
National/government level support	✓	✓	✓							✓
Training of community	✓	✓	✓			✓			✓	✓
Baseline study/needs assessment	✓	✓	✓			✓	✓	✓		✓
Management information system		✓	✓			✓				✓
Management/Leadership	✓	✓	✓
Use of local resources		✓	✓	✓						✓
Integration into society	✓	✓	✓	✓	✓	✓		✓	✓	✓
Behavioural function							✓
Cognitive function							✓	✓
Psychological health				✓			✓	✓
Emotion/mood							✓	✓
Physical function				✓			✓	✓	✓
Social interaction		✓		✓			✓	✓	✓	✓
Social/physical environment				✓			✓
Participation in religious activities				✓						✓
Level of independence				✓				✓	✓
Mobility								✓	✓
Communication								✓
Muscle strength								✓
Global recovery								✓
Self-awareness									✓
Identification										✓

A further abstraction and categorisation process of the forty-four CBR evaluation elements was done by the principal reviewer. Checking and re-categorisation was done together with two other reviewers revealing 25 system evaluation elements. Three main categories were identified as categories that inform the system in CBR evaluation namely service delivery, management and community participation with the overall support from government or national structures which influence policy (Figure 2).

Figure 2. Showing identified system-related elements.

Discussion

A wide range of elements were identified from the 10 CBR programmes reviewed making this a heterogeneous set. This variety is expected since CBR is a strategy that is based on the needs of PLWDs and programmes choose which aspects to focus on. The elements and domains identified in this review could be aligned to existing frameworks namely CBR matrix [1,29] and WHO health systems building blocks [30] and were comparable to Pina’s health system domains and elements [31]. The reviewed studies focussed on the health, social and empowerment domains of the CBR matrix, a few elements covered livelihoods and none covered education [1]. The large pool of elements identified from the few studies included in this systematic review is evidence of the complexity of CBR programmes as guided by the CBR matrix. Identified system elements which have been validated through these frameworks can be used in CBR evaluations and contribute to the formulation of indicators for CBR evaluation and help address the problem of the lack of indicators in CBR [5,16].

It is important to note that the identified elements are relevant to health systems and that they are aligned to the health system domains using both the building blocks of health systems [30] and the domains by Pina et al. [31]. Thus, CBR as a programme can be domiciled in any ministry, for example social services, education or health with collaboration between the other ministries [1] making this approach to system identification appropriate.

Limitations of the review

Inclusion of literature ranked as fair on the level of evidence was the main limitation, thereby compromising the validity of the results of the review. The small sample size and differences in focus across reports made it difficult to establish the common elements used in CBR evaluations. The absence of grey literature which constitutes a major source of CBR programme evaluation information was another limitation this review.

Conclusion and recommendations

There still is a paucity of literature on CBR evaluation with qualitative methods being predominantly used. The quality of CBR literature is generally fair. Twenty-five system evaluation elements were identified from the current literature addressing outcomes in the health, empowerment and social domains of the CBR matrix. Further research is required in the area of monitoring and evaluation for CBR to develop a comprehensive evidence-based tool kit for CBR. From this study, the education and livelihood domains require further work within CBR.

Disclosure statement

No potential conflict of interest was reported by the authors.