2,302
Views
1
CrossRef citations to date
0
Altmetric
Original Article

The role of evaluation in iterative learning and implementation of quality of care interventions

, , , &
Article: 1882182 | Received 17 May 2020, Accepted 22 Dec 2020, Published online: 20 Jun 2021

ABSTRACT

Background: The Lancet Global Health Commission (LGHC) has argued that quality of care (QoC) is an emergent property that requires an iterative process to learn and implement. Such iterations are required given that health systems are complex adaptive systems.

Objective: This paper explores the multiple roles that evaluations need to play in order to help with iterative learning and implementation. We argue evaluation needs to shift from a summative focus toward an approach that promotes learning in complex systems. A framework is presented to help guide the iterative learning, and includes the dimensions of clinical care, person-centered care, continuum of care, and ‘more than medicine. Multiple roles of evaluation corresponding to each of the dimensions are discussed.

Methods: This paper is informed by reviews of the literature on QoC and the roles of evaluation in complex systems. The proposed framework synthesizes the multiple views of QoC. The recommendations of the roles of evaluation are informed both by review and experience in evaluating multiple QoC initiatives.

Results: The specific roles of different evaluation approaches, including summative, realist, developmental, and participatory, are identified in relationship to the dimensions in our proposed framework. In order to achieve the potential of LGHC, there is a need to discuss how different evaluation approaches can be combined in a coherent way to promote iterative learning and implementation of QoC initiatives.

Conclusion: One of the implications of the QoC framework discussed in the paper is that time needs to be spent upfront in recognizing areas in which knowledge of a specific intervention is not complete at the outset. This, of course, implies taking stock of areas of incompleteness in knowledge of context, theory of change, support structures needed in order for the program to succeed in specific settings. The role of evaluation should not be limited to only providing an external assessment, but an important goal in building evaluation capacity should be to promote adaptive management among planners and practitioners. Such iterative learning and adaptive management are needed to achieve the goals of sustainable development goals.

Responsible Editor

Jennifer Stewart Williams

Background

Evaluations of quality of care in a sustainable development goal (SDG) era

In its recent publication, The Lancet Global Health Commission (LGHC) on ‘High-Quality Health Systems in the SDG Era’ [Citation1] poses the following provocative question: ‘What should a high-quality health system look like in countries with resource constraints and competing health priorities that aspire to reach the SDGs?’ The LGHC’s response to this question is informed by a recognition that health systems are complex adaptive systems and that such systems can resist change and can be impervious to isolated interventions [Citation2]. Further, the LGHC argues that ‘quality of care is an emergent property that requires shared aims among all health system actors, favorable health system foundations, and is honed through iterative efforts to improve and learn from successes and failures’ [Citation1].

Throughout this paper, the question we focus on is: What should evaluation’s role be in such iterative learning? The notion of quality of care (QoC) as an emerging aspect in complex adaptive systems demands a more comprehensive discussion around the role of evaluations in helping systems move toward greater quality in an iterative way. Such thinking about quality is especially critical in resource-constrained settings and where ideas of best practices might be difficult to implement given the constraints and complexities of specific settings [Citation3,Citation4].

Given the varieties of quality of care (QoC) interventions, this paper is not intended to provide a ‘how to’ do evaluation. We work in the field of evaluating complex interventions and our view is that the ‘how to’ do evaluations might depend on both the needs of stakeholders and the complexities of the interventions. Instead, our goal here is to raise awareness of the types of questions and alternative evaluation approaches can help with in planning, translating knowledge, and estimating impacts of QoC interventions. There are multiple purposes of evaluation; there is a need for the field of QoC program planners and implementers to be aware of these multiple evaluation approaches. We believe becoming aware of the wide variety of evaluation purpose and approaches available will help improve the quality of evaluation of QoC interventions [Citation5].

How can a scientific approach to evaluation help in moving toward such an iterative approach? In recent years, the field of evaluation [Citation3,Citation6,Citation7] has undergone a shift in thinking from a purely summative exercise (‘does an intervention work?’) toward a recognition that evaluations in complex systems [Citation7–9] require a focus on helping an intervention or a system learn to develop more completely. Developmental evaluation [Citation10,Citation11], for example, accepts that evidence-based interventions typically need further development to adapt to specific contexts – the evaluation itself is helpful in aiding such development. Similarly, realist evaluators [Citation12,Citation13] explore the specific contexts and mechanisms that are necessary for interventions to work.

Recent discussions about QoC have increasingly revolved around the question of incorporating a multi-dimensional view of quality. The literature has also seen a growth in scholarly debate around related concepts of dignity and equity [Citation14]; person-centered care [Citation15–17]; and rights-based approaches [Citation18]. Returning to an emergent view of QoC, the current paper focuses on how evaluations can help come to terms with such a multi-dimensional view of QoC that is grounded in the need for iterative learning. In our view, this means moving away from a narrow ‘what works’ evaluation to a role of evaluation that is sensitive to the development of QoC over time.

Building an iterative-learning system needs to go beyond a series of commissioned evaluations done by external organizations; instead, the focus needs to be how different parts of systems at various levels – national, state, district, regional, community, and facility can learn about ongoing improvement. There is a need to move from debates about best methods toward a clearer view of pathways of system-level learning.

Outline of paper

This paper discusses what it means, from an evaluation perspective, to think about quality as an emergent, multi-dimensional construct that requires an iterative process to learn and implement [Citation3]. To facilitate this, we propose a framework for QoC and focus our discussion on maternal health. While several QoC frameworks exist in the literature, not all of these frameworks provide a blueprint for approaching QoC as a multi-dimensional construct. The LGHC has argued that the sheer breadth of quality deficits in maternal and neonatal care means incremental solutions will be insufficient; the framework presented in this paper can help position the evaluations to create a more robust iterative process to achieve high quality, rather than exist as a single incremental change as they often do.

Quality of care in maternal health

The need for an iterative evaluative approach to improving QoC becomes important when one considers the successes and failures of the millennium development goals (MDGs). The fourth and fifth MDGs were the global reduction of child mortality and the improvement of maternal health [Citation19]. Despite the significant progress made toward these goals, every year more than 35,0000 women die due to complications related to pregnancy and childbirth [Citation20]. Significant resources have been allocated to interventions designed to lower maternal mortality in low resource countries. Much of this effort has focused on increasing rates of antenatal care, ensuring that deliveries are attended by skilled birth attendants, and making sure that these deliveries occur in a healthcare facility. After years of hard work and millions of dollars in investment, a growing body of evidence suggests that resource poor areas with increased antenatal care and skilled birth attendants are not seeing the expected proportional drop in maternal mortality [Citation21].

Following the failure to reach the MDGs’ maternal mortality goals, the creation of the SDGs in 2015 included a target for maternal mortality of less than 70 deaths per 100,000 live births [Citation22]. A blueprint for reaching this goal was published in the Lancet Maternal Health Series in 2016. A key step in achieving this goal is improving the quality, and not simply the QoC [Citation23]. In discussing an approach to QoC evaluation, we will focus on a discussion on its application to maternal health, using examples related to this field.

Methods

Three sources of information informed the development of this paper:

(a) Literature reviews of the QoC interventions and evaluations of complex intervention experience.

(b) Based on the experience of authors. Many of the authors work in the field of evaluating complex interventions and are based in the Evaluation Centre for Complex Health Interventions in Toronto. One of the co-authors has been a leader in evaluation for close to three decades and has experience in conducting evaluations of complex interventions in many international contexts.

(c) The framework and the approaches have been presented in multiple Evaluation Centre webinars with leaders in the field of evaluation in attendance. The recommendations in this paper have been informed by the feedback received in these webinars.

Quality of care frameworks

Existing models of QoC and the need for a new framework

We searched the literature for conceptual frameworks related to QoC. While there are many potential frameworks related to maternal health and QoC in general, we chose well-established frameworks that have been previously used to evaluate maternal health interventions for inclusion here.

There are several prominent models for QoC in the literature. Some are simple, and while succinct, are not easily operationalizable, such as the definition offered by the Institute of Medicine, ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current and professional knowledge’ [Citation24,Citation25].

Definitions or models that define quality using individual dimensions or components begin to recognize the complexity and multidimensionality of QoC. The most cited QoC model is the Donabedian model, which divides QoC into structure (i.e. the factors that affect the context of care, such as facility, human resources, and training), process (i.e. the actions that comprise healthcare, both what is delivered and how) and outcomes (i.e. the effects of the healthcare on the individual or population) [Citation26].

Other prominent models include that of Hulton et al. [Citation27], which proposed 10 elements as a framework for maternal QoC and separated these 10 elements into two components: (1) the quality of the provision of care (human and physical resources, the referral system, maternity information systems, use of appropriate technologies, internationally recognized good practice, and management of emergencies) and (2) the quality of the care as experienced by users (human and physical resources, cognition, respect, dignity, equity, and emotional support) [Citation27].

In their review of the existing literature, Raven et al. [Citation28] examined current frameworks used to approach QoC in maternal and newborn health and identified several focus areas. Current models include those based on assessing QoC from the user’s experience of care, the client’s perspective, patients’ rights and providers’ needs, and input–output-outcome models [Citation29].

A framework that assesses QoC must also recognize the interdependence of the various dimensions of quality. This is critical in deciding which levers to push and, therefore, where to invest in order to drive improvements in quality. For example, in India, an intervention that provided cash transfers to women to deliver their babies in healthcare facilities failed to achieve an adequate level of skilled birth attendance. In this case, the primary barriers identified included abusive staff and a lack of person-centered care [Citation30]. The intervention incentivized technical quality (facility delivery, attendance of skilled birth attendant), but failed to recognize a connection between technical quality and person-centered care.

Consistent with a view of complex adaptive systems, QoC is not a measure of a single interaction in the healthcare system but is rather a result of multiple interactions at various levels of care (primary, secondary, tertiary). QoC must also be viewed as dynamic, rather than episodic, in nature, as it extends beyond isolated interactions within the healthcare system and must encompass the supports and resources available outside the healthcare system because these shape a patient’s interaction with the healthcare system. For example, a recommendation of bedrest in treating pre-eclampsia may be unrealistic for a rural woman living in poverty who has no support and little choice but to continue with her daily activity. In a low resource setting, quality varies not just between countries, but also within countries, within states, within cities, and even within individual facilities.

Finally, we argue that a framework for quality must recognize that not only is quality dynamic, but there is a dynamic interdependence between the components of quality. For example, if good quality care is a function of the care given and the care experienced, we know that this contributes to engagement with care over time [Citation31]. We also know that increased engagement leads to better quality care [Citation32]. As the various dimensions of quality improve, a robust framework must provide the user direction to other dimensions that may be affected by this change and may be prudent to measure.

In the next section, we synthesize the above definitions into a novel framework, while also exploring the implications for evaluation of each of the dimensions.

Towards a multidimensional view of quality of care

Based on the existing literature and our experience with QoC evaluations in resource poor settings, we propose the following framework consisting of four core domains: (i) person centered care, (ii) continuum of care, (iii) more than medicine, and (iv) technical quality. The framework allows us to map theoretical relationships, create testable propositions, and help inform the development of an evaluation framework (see ). Our framework has been informed by the House of Care model from the UK [Citation33] that focuses on coordinated care for patients with long-term conditions.

Figure 1. Proposed framework for quality of care

Figure 1. Proposed framework for quality of care

In what follows below, we describe both the conceptual implications and the implications for evaluation for each of these core domains.

Person-centered care

Person-centered care is the delivery of care that involves patients in shared decision-making with health care providers. It takes into consideration their values and priorities and treats them as an active contributor to their health, rather than a passive recipient of healthcare [Citation34]. When viewing healthcare as more than just a series of interactions with a healthcare provider, it becomes clear that without the participation of the patient, patients are less likely to follow direction, take prescribed medication, or participate in positive activities, such as attending regular checkups or getting immunized [Citation35]. The importance of person-centered care has been shown to lead to improved health outcomes and utilization, particularly in areas of socio-economic, cultural, or religious diversity [Citation36]. Improvements in person-centered care measures show a correlation with ‘increased adherence to modern family planning methods, […] shorter labor, better coping with pain, decreased incidence of operative birth, increased incidence of spontaneous vaginal delivery, increased maternal satisfaction, less anxiety, and increased rates of breastfeeding initiation’ [Citation17].

In their 2017 Person-Centered Care Framework for Reproductive Health Equity, Sudhinaraset and colleagues outline eight patient reported measures around dignity, autonomy, privacy/confidentiality, communication, social support, supportive care, trust, and health facility environment [Citation17]. Monitoring implementation using such a framework and conducting periodic assessments of the person-centered policies and practices among healthcare staff and healthcare facilities might be vital to changing the landscape of clinical practice in regions with poor maternal outcomes.

Continuum of care

Health care delivery is frequently siloed. Healthcare providers dealing with one issue often lack a comprehensive picture of a patient’s health and are often unaware of what other providers have planned. A lack of integration across the continuum of care results in, at best, wasted resources and redundant care; at worst, it can result in harm. For example, efforts to ensure high rates of skilled birth attendance may result in little improvement in neonatal and maternal mortality if there exists a gap in connecting newborns to timely and repeated follow-up care within the first weeks of life, when they are most vulnerable.

Maternal and neonatal care must be viewed not as a stand-alone entity, but instead as interwoven in the framework of existing healthcare and women’s homes and community. A continuity or continuum of care must exist throughout a woman’s lifecycle, from adolescence, pregnancy, childbirth, and childhood. As Kerber argues, ‘Saving lives depends on high coverage and quality of integrated service-delivery packages throughout the continuum, with functional linkages between levels of care in the health system and between service-delivery packages, so that the care provided at each time and place contributes to the effectiveness of all the linked packages’ [Citation40].

A care continuum also suggests minimizing programming silos as these can affect health system resilience [Citation41]. Perinatal mortality through indirect causes, such as HIV/AIDS, malaria, and non-communicable diseases are on the rise and comprehensive care packages should be linking existing services for these diseases with maternity services for optimal efficiency and cost-effectiveness [Citation23]. Having such routine, reliable, comprehensive care that follows a person throughout her life-cycle will promote ‘the adoption of healthy behaviors and empower individuals and families to demand quality services’ [Citation40], which, in turn, can perpetuate the use of, and a reliance on, the health system.

Epidemiological and health systems contexts need to form the bedrock of continuum of care measures for maternal and neonatal healthcare. Care fragmentation is prevalent in developing countries simply because ‘human-resource capacity, health-facility infrastructure, supply systems, financial resources, government stewardship, district-level management, and monitoring’ are poorly coordinated [Citation40]. Some basic administrative measures that can strengthen care continuity include linking data through the use of technology, where there is capacity, or training providers to keep accurate medical records [Citation42,Citation43]. It is vital that national and subnational bodies ensure that continuum of care measures not merely test service utilization, but also the quality and level of integration between services [Citation44,Citation45]. This is critical so that women and children are not lost from one medical episode or instance of service utilization to the next.

More than medicine

We know that community-based care has the potential to lower maternal mortality [Citation51,Citation52]. Even in the most conventional clinical setting, good anticipatory care needs to be informed by an understanding of the types of supports available to the client. The dimension of ‘more than medicine’ extends beyond structured community resources to include understanding of healthcare recipients’ partner, family, and other interpersonal supports. Currently, in India, the health sector is largely absent from the social milieu of the perinatal period [Citation53], despite good social support being a predictor of lower maternal morbidity [Citation54].

More than medicine in a maternal care setting mobilizes a ‘multipronged attack from all conceivable fronts – using both obstetric and non-obstetric interventions’ [Citation55]. For example, development interventions through primary education, microcredit, and women’s empowerment initiatives have seen a reduction in violence against women, increased family-planning practice, and improved children’s nutritional status [Citation55].

Measures of ‘more than medicine’ could include whether the health care practitioners have promoted community health resources, assessed the needs, goals, and limitations of patients, established an understanding of community resources, and aligned these resources with patient and population needs [Citation56]. Health system stakeholders could also look for the presence of ‘peer support services, advocacy services, structured education, [and] coaching programmes’ [Citation56] targeted at women’s education, empowerment, health, and nutrition education. Mobilization of civil society groups and the voluntary sector could also be markers of robust community support structures, while continued advocacy on behalf of women and mothers would be instrumental in shifting local priorities to addressing challenges unique to women.

Technical quality

At its simplest, technical quality refers to safe, accurate, and well-timed routine and emergency clinical interventions done by competent staff who are trained according to internationally recognized and evidence-based guidelines of care delivery. However, an assessment of technical quality must consider more than just the provision of a clinical procedure or the competency of the provider. For example, knowledge of the appropriate use of uterotonic drugs is meaningless without a procurement system to ensure readily available supply of those drugs. Similarly, training in the appropriate administration of antibiotics will have limited impact without the presence of a robust accountability system to ensure the behavior continues beyond the observation period of an intervention. Technical capacities vary dramatically between facilities and across locales based on the resilience of their physical and human resources, adherence to good practice, and the presence of robust accountability systems.

From a clinical standpoint, the WHO recommends eight signal functions for obstetric care that all facilities should meet [Citation60]. Additional comprehensive services would include facilities being able to perform surgeries (e.g. cesarean section) and perform blood transfusions [Citation60]. High rates or mortality and morbidity, despite technical quality improvements, potentially suggest that other dimensions are deficient. Emphasis on measures of efficiency through appropriate levels of funding, the cost-effectiveness of interventions, and effective administration are important in establishing and sustaining quality improvements [Citation61].

Discussion

This paper has been informed by an understanding that views of QoC need to be informed by a recognition that quality is a dynamic, emergent aspect of complex adaptive systems [Citation2]. We have argued that evaluations need to promote knowledge of how quality can emerge given the complexities of context, the dependence of interventions on underlying systems, and the necessity for adaptation of interventions to specific contexts. Our primary argument is that we need to enhance our toolbox of evaluation approaches that are currently used to understand and enhance QoC. Participatory evaluation approaches [Citation38,Citation39], for example, can help incorporate patient/client voices as part of better understanding the heterogeneities of person-centered care. A realist evaluation [Citation12,Citation13] approach, on the other hand, can assist in better understanding how interventions need to adapt to specific contexts in order to trigger specific mechanisms that can impact outcomes. The developmental evaluation approach [Citation10,Citation11], moreover, recognizes that interventions being implemented in specific contexts are often incomplete in their knowledge of context. From a developmental evaluation perspective, the evaluation itself plays an important role in helping such interventions adapt to specific contexts.

The paper has identified the following implications for evaluating QoC:

  1. Recognition of the multiple purposes of evaluation: This paper draws attention to the multiple purposes of evaluation in understanding and enhancing QoC. Returning to the formulation of QoC as an emergent property, we believe that evaluations are most useful if they are situated within a learning system [Citation62,Citation63] that pays attention to improvements over time. Such improvements can be achieved not merely by providing evidence of good QoC, but also by promoting thinking about what constitutes good QoC, developing theories of change to address context-specific impediments to QoC [Citation64–67], and developing an enhanced theoretical understanding of the heterogeneous pathways by which good QoC can impact individuals [Citation68].

  2. Clarifying the role of context in implementing QoC interventions: In our experience working with complex interventions, it is rare that interventions are clearly fungible across contexts. Given the challenges of explaining variations in QoC across contexts, we argue for a realist evaluation perspective; the focus of realist evaluation is on the contexts and mechanisms that generates the outcomes. Realist evaluation falls into the class of theory-driven evaluation approaches that seeks to explain ‘what works for whom, in what circumstances, in what respects, and how?’ [Citation69]. describes some of the key tenets of realist evaluation and why it is particularly suited for iterative learning for QoC. Evaluation, thus, needs to promote thinking about how interventions can be implemented with high ‘fidelity’ across multiple contexts [Citation65,Citation67,Citation70]. In our view, context (and support structures) are often under-theorized in most descriptions of QoC interventions. We suggest that evaluations need to promote a more formal approach to thinking about the role of context and support structures in facilitating or hampering the implementation of interventions. Incorporating context has important implications for how programs are planned and implemented. For example, a good theory of change will need to describe the infrastructure and institutional supports that are necessary for interventions to be successful [Citation71].

  3. Minimal thresholds of QoC: One of the additional advantages to thinking evaluatively about the multiple dimensions of QoC is the attention such thinking brings to the minimum standards for each of the dimensions. For instance, even though there has been a lot of research focus on the minimum threshold for good clinical care [Citation72], there has not been much research done on the thresholds for person-centered care. Abusive care, for example, can lead to clients dropping out of treatment altogether. A focus on minimal standards of care also can shed light on the capacities of health systems needed to provide high-quality care. For example, severely over-stretched health professionals might not be in a position to provide good levels of person-centered care because of time constraints placed on interactions with patients.

  4. Developing a research agenda for QoC: Another strength of thinking evaluatively about the multiple dimensions of QoC is that it can lead toward generating hypotheses that can be explored in the future research and evaluation. Some examples of longitudinal evaluation questions that emerge from the framework presented in this paper include:

    • Can more favorable trajectories of person-centered care lead to an enhanced likelihood of the safe delivery of healthy babies?

    • What is the relationship between clinical care and person-centered care in high-quality health systems?

    • Do increased supports at home and in the community lead to a greater likelihood of safe deliveries?

Table 1. Relevance of realist evaluation for iterative learning

Towards principles of evaluating quality of care

Following the core principles enunciated by the LGHC, this paper argues that evaluations of QoC initiatives also need to be informed by some critical principles that move beyond a clinical view of QoC. Following the QoC framework presented in this paper, we argue that evaluations of QoC interventions should be guided by the following principles:

  1. Pay attention to clients’ journeys: Our conceptualization of quality recognizes that measuring the quality of a single episode of care needs to be embedded within the client’s journey. This follows from a recognition in LGHC that health systems are ‘for people.” It is important to recognize that the client herself is a co-producer of her/his own health outcomes [Citation73]. This, of course, means asking important questions of the intervention, such as whether the intervention empowers the client as a co-producer of her own health [Citation73] and whether a client feels that the intervention gives her a voice to state her preferences and choices [Citation73].

  2. Respect and dignity matter: As part of any multi-dimensional measurement of QoC, we think that there needs to be a greater focus on the principles of respect and dignity [Citation74–76]. It is important for an evaluation to explore if an intervention treated a woman with dignity and respect. It is likely that respectful care can be a means to better long-term outcomes.

  3. High-quality care is anticipatory: Our conceptualization is also driven by a recognition that high-quality care needs to anticipate key events in a client’s life, and also anticipate constraints and barriers. Evaluations of QoC also need to interrogate whether care is anticipatory [Citation77]. High-quality care is driven by a view of key long-term outcomes that need to be achieved – for example, a safe birth. We need to think longitudinally in terms of the trajectory of care, as it is a journey made up of multiple key events over a period of time. In the context of maternal health, the central focus thus needs to be on what is needed to ensure a safe delivery and a healthy baby and mother over time.

  4. High QoC pays attention to contexts: There are multiple types of contexts that might influence whether a client received good quality care. Such care can be grouped into what we characterize as the 5I’s – which refers to Infrastructural, Institutional, Interpersonal, Individual, and Intersectional considerations [Citation78–80]. In our experience, variations in the 5I’s can lead to considerable heterogeneities [Citation81,Citation82] in the QoC women experience. Given the heterogeneities that are introduced by the realities of the 5I’s, providing quality care is rarely a mechanical process. For example, when a health care professional is faced with a high-risk pregnancy, attention needs to be given to determine the types of care that client can receive in her home, the types of supports she has available, the type of preferred interpersonal communication that will facilitate adherence to treatment, the types of work obligations she might have, and the types of transportation she has access to in order to get to a district-level facility that can deliver high-quality care for potential complications. Evaluations of QoC initiatives need to pay attention to the 5I’s and explore the role of multiple contexts in leading to variations in QoC.

  5. Quality of care is system-aware

All health care providers are embedded in larger systems [Citation83–87]. Delivering high quality care is, therefore, dependent on many systemic factors including having a well-functioning human resources system, a procurement system that pays attention to shortages of key medicines, and so on. Ideally, an evaluation of QoC initiatives will be able to provide feedback on role of system-level factors in enhancing the efficacy of QoC interventions.

Towards system-level learning

Taking a systems perspective to QoC, LGHC (1) has argued that ‘… evidence from health and other sectors shows that complex adaptive systems can thrive if actors within the system have a shared vision, clear rules, and space to allow evolution and learning.’ An important question for evaluation is: How can a learning system promote such evolution and learning? describes some questions that comprehensive evaluations that are both context and capacity-aware and guided by system-level learning need to explore. We recognize that realistically a number of the questions are not relevant to most QoC interventions; yet, we think such a list of questions will help the planning and knowledge translation of QoC interventions. Addressing some of the questions can also enhance the utility and relevance of evaluations.

Table 2. Questions to guide a comprehensive approach to QoC evaluations

Limitations and potential strengths

We recognize that this expanded view of evaluation will carry with it some limitations and potential objections. One concern we anticipate to the multiple purposes of evaluation described in this paper is that it might inadvertently be making evaluation harder than it needs to be by focusing on data in multiple dimensions. While we recognize that taking the multiple dimensions of QoC seriously does imply an additional data burden, we think the ‘learning pay-offs’ associated with such additional data will more than make up for any burdens created by such a focus on multiple dimensions of QoC. This paper was a response to how the evaluation community can be responsive to the recent LGHC on high-quality health systems. The iterative view recommended by the LGHC would require, in our view, a much more diverse and pluralistic approach to evaluation and data collection. We also think that taking such a pluralistic view of evaluation itself also will help move the field away from a mechanical view of evaluation in which products often are developed without serious learning occurring. Additionally, the view espoused in this paper, driven by our experiences working in developing country settings, is that interventions are not often transportable across contexts. The adaptations required for interventions to be ‘adapted’ into different contexts does imply the need to think evaluatively on how best can interventions be implemented across contexts.

We recognize implementing the conceptual ideas discussed in this paper will remain challenging. Operationalizing many of the ideas, such as information systems for ‘more than medicine’ or discuss recent innovations in data linkages, or more recent advances in analytical methods will require another paper with a narrower focus on a specific intervention. We acknowledge the absence of discussion on measurement, data collection, data linkages, and analysis as a limitation of this paper.

Conclusion

We have argued that an expanded view of evaluation can help shift current definitions of QoC as a primarily clinical phenomenon toward a multidimensional view that prioritizes the respect for and dignity of the client. We also suggest that it is vital that a view of quality does not treat the individual as a passive player in the production of good health outcomes. Instead, a view of quality needs to pay attention to the critical role played by the client in the co-production of health. Thoughtful evaluations can play an important role in helping to move toward an expanded conceptualization of quality. This paper has relevance for the role of evaluation in helping meet the SDG goals of maternal health. High-quality health systems will be critical to meeting the SDG targets. There is a need for a discourse around the multiple roles that evaluation can play in enhancing the likelihood that the SDGs focus on ‘no one left behind’ can be successful.

Ethics and consent

This study does not involve any human subjects.

Paper context

This paper explores the role of evaluation in building iterative learning to achieve the Lancet Global Health Commission’s recommendations for achieving high-quality health systems through iterative learning. The multiple role evaluation can play in achieving such iterative learning is described by presenting a novel quality of care model that was adapted to maternal health settings. Key dimensions of the model include person-centered care, clinical care, continuum of care, and ‘more than medicine.’

Acknowledgments

The authors wish to thank Dr Rachael Gibson for help in helping shape the thinking of this manuscript and also for editing an early draft of this article.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study was supported by the Bill and Melinda Gates Foundation (Grant # OPP1083531).

Notes on contributors

Nikhil Shah

The first version of the paper was written by Nikhil Shah and Sanjeev Sridharan. Nikhil Shah, Sharon Mathew, and Amanda Pereira worked on the literature review. April Nakaima, Sharon Matthew, and Amanda Pereira worked on additional drafts of the paper and also contributed additional sections to the paper. All authors have reviewed and approved this version of the paper.

References

  • Kruk ME, Gage AD, Arsenault C, et al. High-quality health systems in the sustainable development goals era: time for a revolution. Lancet Glob Health. 2018;6:e1196–12.
  • Hill PS. Understanding global health governance as a complex adaptive system. Glob Public Health. 2011;6:593–605.
  • Reynolds M, Sarriot E, Swanson RC, et al. Navigating systems ideas for health practice: towards a common learning device. J Eval Clin Pract. 2018;24:619–628.
  • Andrews M, Pritchett L, Woolcock M. Escaping capability traps through problem driven iterative adaptation (PDIA). World Dev. 2013;51:234–244 [Internet].
  • Patton MQ. Evaluation science. Am J Eval. 2018;39:183–200.
  • Yonehara A, Saito O, Hayashi K, et al. The role of evaluation in achieving the SDGs. Sustain Sci. 2017;12: 969–973.
  • Minary L, Trompette J, Kivits J, et al. Which design to evaluate complex interventions? Toward a methodological framework through a systematic review. BMC Med Res Methodol. 2019;19:1–9.
  • Cambon L, Terral P, Alla F. From intervention to interventional system: towards greater theorization in population health intervention research. BMC Public Health. 2019;19:1–7.
  • Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:979–983 [Internet].
  • Lam CY, Shulha LM. Insights on using developmental evaluation for innovating: a case study on the cocreation of an innovative program. Am J Eval. 2015;36:358–374.
  • Harper LM, Dickson R. Using developmental evaluation principles to build capacity for knowledge mobilisation in health and social care. Evaluation. 2019;25:330–348.
  • Pawson R, Tilley N. Realistic evaluation. London: Sage Publications; 1997.
  • Adams A, Sedalia S, McNab S, et al. Lessons learned in using realist evaluation to assess Maternal and Newborn health programming in rural Bangladesh. Health Policy Plan. 2016;31:267–275.
  • Kinney MV, Boldosser-Boesch A, McCallon B. Quality, equity, and dignity for women and babies. Lancet. 2016;388:2066–2068. . [Internet]
  • Kennedy HP, Cheyney M, Dahlen HG, et al. Asking different questions: a call to action for research to improve the quality of care for every woman, every child. Birth. 2018;45:222–231.
  • Srivastava A, Singh D, Montagu D, et al. Putting women at the center: a review of Indian policy to address person-centered care in maternal and newborn health, family planning and abortion. BMC Public Health. 2017;18:1–10.
  • Sudhinaraset M, Afulani P, Diamond-Smith N, et al. Advancing a conceptual model to improve maternal health quality: the person-centered care framework for reproductive health equity. Gates Open Res. 2017;1:1–15.
  • Jain AK, Hardee K. Revising the FP quality of care framework in the context of rights-based family planning. Stud Fam Plann. 2018;49:171–179.
  • United Nations Millennium Development Goals. UN [Internet]; 2000 [cited 2019 Jul 15]. Available from: https://www.who.int/topics/millennium_development_goals/about/en/
  • Alkema L, Chou D, Hogan D, et al. Global, regional, and national levels and trends in maternal mortality between 1990 and 2015, with scenario-based projections to 2030: a systematic analysis by the UN maternal mortality estimation inter-agency group. Lancet. 2016;387:462–474.
  • Alemayehu M, Mekonnen W. The prevalence of skilled birth attendant utilization and its correlates in North West Ethiopia. Biomed Res Int. 2015;1–8. DOI:https://doi.org/10.1155/2015/436938
  • WHO – UN Sustainable Development Summit. WHO [Internet]. 2015 [cited 2019 Jul 15]. Available from: https://www.who.int/mediacentre/events/meetings/2015/un-sustainable-development-summit/en/
  • Koblinsky M, Moyer CA, Calvert C, et al. Quality maternity care for every woman, everywhere: a call to action. Lancet. 2016;388:2307–2320.
  • Public health and welfare: concepts, methodologies, tools, and applications. IGI Global. 2017;268.
  • Institute of Medicine: crossing the Quality Chasm: a New Health System for the 21st Century. Washington, DC: The National Academies Press; 2001.
  • Donabedian A. Evaluating the quality of medical care. The Milbank Memorial Fund Quarterly. 1966;44: 166–206.
  • Hulton L, Matthews Z, Stones RW. A framework for the evaluation of quality of care in maternity services. Stud Family Plann. 2000;31:35–46.
  • Raven JH, Tolhurst RJ, Tang S, et al. What is quality in maternal and neonatal health care? Midwifery. 2012;28:e676–683.
  • Austin A, Langer A, Salam RA, et al. Approaches to improve the quality of maternal and newborn health care: an overview of the evidence. Reprod Health. 2014;11:S1.
  • Chaturvedi S, De Costa A, Raven J. Does the Janani Suraksha Yojana cash transfer programme to promote facility births in India ensure skilled birth attendance? A qualitative study of intrapartum care in Madhya Pradesh. Glob Health Action. 2018;8:1-13.
  • Coulter A. Patient engagement—what works? J Ambul Care Manage. 2012;35:80–89.
  • Hibbard JH, Jessica G. What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs. Health Aff. 2013;32:207–214.
  • Coulter A, Roberts S, Dixon A Delivering better services for people with long-term conditions: building the house of care. King’s Fund [Internet]; 2013. [cited 2019 Jul 15]. Available from: https://www.kingsfund.org.uk/sites/default/files/field/field_publication_file/delivering-better-services-for-people-with-long-term-conditions.pdf
  • Ahmad N, Ellins J, Krelle H et al. Person-centred care: from ideas to action. Health Foundation [Internet]; 2014. [cited 2019 Jul 15]. Available from: https://www.health.org.uk/sites/default/files/PersonCentredCareFromIdeasToAction.pdf
  • James J Patient engagement. Health Aff.[Internet]; 2013.[cited 2019 Jul 15]. Available from: https://www.healthaffairs.org/do/10377/hpb20130214.898775/full/
  • Epstein RM, Street RLJ Jr. The values and value of patient-centered care. Ann Fam Med. 2011;9: 100–103.
  • Parsons JA, Lavery JV. Brokered dialogue: a new research method for controversial health and social issues. BMC Med Res Methodol. 2012;12:12.
  • Gilmore B, Vallières F, McAuliffe E, et al. The last one heard: the importance of an early-stage participatory evaluation for programme implementation. Implement Sci. 2014;9:1–12.
  • Kananura RM, Ekirapa-Kiracho E, Paina L, et al. Participatory monitoring and evaluation approaches that influence decision-making: lessons from a maternal and newborn study in Eastern Uganda. Health Res Policy Syst. 2017;15:55-68.
  • Kerber KJ, Graft-Johnson JED, Bhutta ZA, et al. Continuum of care for maternal, newborn, and child health: from slogan to service delivery. Lancet. 2007;370:1358–1369.
  • Kruk ME, Kujawski S, Moyer CA, et al. Next generation maternal health: external shocks and health-system innovations. Lancet. 2016;388:2296–2306.
  • Wehrmeister FC, Restrepo-Mendez MC, Franca GV, et al. Summary indices for monitoring universal coverage in maternal and child health care. Bull World Health Organ. 2016;94:903–912.
  • Boerma T, Ronsmans C, Melesse DY, et al. Global epidemiology of use of and disparities in caesarean sections. Lancet. 2018;392:1341–1348.
  • Singh K, Story WT, Moran AC. Assessing the continuum of care pathway for maternal health in South Asia and Sub-Saharan Africa. Matern Child Health J. 2016;20:281–289.
  • Kikuchi K, Okawa S, Zamawe COF, et al. Effectiveness of continuum of care – linking pre-pregnancy care and pregnancy care to improve neonatal and perinatal mortality: a systematic review and meta-analysis. PLoS One. 2016;11:1–13.
  • Meyer MA. Mapping the patient journey across the continuum: lessons learned from one patient’s experience. J Patient Exp. 2019;6:103–107.
  • Kyei NNA, Campbell OMR, Gabrysch S. The influence of distance and level of service provision on antenatal care use in rural Zambia. PLoS One. 2012;7:10.
  • Goswami ND, Schmitz MM, Sanchez T, et al. Understanding local spatial variation along the care continuum: the potential impact of transportation vulnerability on HIV linkage to care and viral suppression in high-poverty areas, Atlanta, Georgia. J Acquir Immune Defic Syndr. 2016;72:65–72.
  • Willie TC, Stockman JK, Keene DE, et al. Social networks and its impact on women’s awareness, interest, and uptake of HIV pre-exposure prophylaxis (PrEP). JAIDS J Acquir Immune Defic Syndr. 2019;80:386–393.
  • Hill M, Huff A, Chumbler N. Variation in networks and forms of support for care-seeking across the HIV care continuum in the Rural Southeastern USA. J Rural Health. 2018;34:71–79.
  • Ricca J, Kureshy N, Leban K, et al. Community-based intervention packages facilitated by NGOs demonstrate plausible evidence for child mortality impact. Health Policy Plan. 2013;29:204–216.
  • Schiffman J, Darmstadt GL, Agarwal S, et al. Community-based intervention packages for improving perinatal health in developing countries: a review of the evidence. Semin Perinatol. 2010;34:462–476.
  • Raman S, Srinivasan K, Kurpad A, et al. My mother … my sisters … and my friends: sources of maternal support in the perinatal period in urban India. Midwifery. 2014;30:130–137.
  • Robertson E, Grace S, Wallington T, et al. Antenatal risk factors for postpartum depression: a synthesis of recent literature. Gen Hosp Psychiatry. 2004;26:289–295.
  • Chowdhury ME, Botlero R, Koblinsky M, et al. Determinants of reduction in maternal mortality in Matlab, Bangladesh: a 30-year cohort study. Lancet. 2007;370:1320–1328.
  • Personalised care and support planning handbook. NHS [Internet]; 2015; [cited 2019 Jul 15]. Available from:https://www.england.nhs.uk/wp-content/uploads/2016/04/core-info-care-support-planning-1.pdf
  • Hudon C, Chouinard MC, Couture M, et al. Partners for the optimal organisation of the healthcare continuum for high users of health and social services: protocol of a developmental evaluation case study design. BMJ Open. 2014;4:1–8.
  • Filippi V, Ronsmans C, Campbell OM, et al. Maternal health in poor countries: the broader context and a call for action. Lancet. 2006;368:1535–1541.
  • Kozhimannil KB, Henning-Smith CE, Hardeman RR. Reducing maternal health disparities: the rural context. Am J Obstet Gynecol. 2017;216:193–194 [Internet].
  • World Health Organization.Monitoring emergency obstetric care: a handbook. WHO Press [Internet]; 2009. Available from https://www.who.int/reproductivehealth/publications/monitoring/9789241547734/en/
  • Kruk ME, Freedman LP. Assessing health system performance in developing countries: a review of the literature. Health Policy. 2008;85:263–276.
  • Irimu G, Ogero M, Mbevi G, et al. Approaching quality improvement at scale: a learning health system approach in Kenya. Arch Dis Child. 2018;103:1013–1019.
  • English M, Irimu G, Agweyu A, et al. Building learning health systems to accelerate research and improve outcomes of clinical care in low- and middle-income countries. PLoS Med. 2016;13:1–8.
  • May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016;11:1–12 [Internet].
  • Hawe P, Shiell A, Riley T, et al. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Commun Health. 2004;58:788–793.
  • Ebenso B, Manzano A, Uzochukwu B, et al. Dealing with context in logic model development: reflections from a realist evaluation of a community health worker programme in Nigeria. Eval Program Plann. 2019;73:97–110 [Internet].
  • Olivier De Sardan JP, Diarra A, Moha M. Travelling models and the challenge of pragmatic contexts and practical norms: the case of maternal health. Health Res Policy Syst. 2017;15:71-87.
  • Kraamwinkel N, Ekbrand H, Davia S, et al. The influence of maternal agency on severe child undernutrition in conflict-ridden Nigeria: modeling heterogeneous treatment effects with machine learning. PLoS One. 2019;14:1–16.
  • Pawson R, Greenhalgh T, Harvey G, et al. Realist synthesis: an introduction(Paper 2). ESRC Research Methods Programme, University of Manchester. 2004. Available from http://betterevaluation.org/sites/default/files/RMPmethods2.pdf
  • Kemp L. Adaptation and fidelity: a recipe analogy for achieving both in population scale implementation. Prev Sci. 2016;17:429–438 [Internet].
  • Cook RJ. Human rights and maternal health: exploring the effectiveness of the Alyne decision. J Law Med Ethics. 2013;41:103–123.
  • Trelles Centurion M, Crestani R, Dominguez L, et al. Surgery with limited resources in natural disasters: what is the minimum standard of care? Curr Trauma Reports. 2018;4: 89–95.
  • Batalden M, Batalden P, Margolis P, et al. Coproduction of healthcare service. BMJ Qual Saf. 2016;25:509–517.
  • Solnes Miltenburg A, Lambermon F, Hamelink C, et al. Maternity care and human rights: what do women think? BMC Int Health Hum Rights. 2016;16:1–10 [Internet].
  • Shakibazadeh E, Namadian M, Bohren MA, et al. Respectful care during childbirth in health facilities globally: a qualitative evidence synthesis. BJOG An Int J Obstet Gynaecol. 2018;125:932–942.
  • Miller S, Abalos E, Chamillard M, et al. Beyond too little, too late and too much, too soon: a pathway towards evidence-based, respectful maternity care worldwide. Lancet. 2016;388:2176–2192.
  • Watt G, O’Donnell C, Sridharan S. Building on Julian Tudor Hart’s example of anticipatory care. Prim Health Care Res Dev. 2011;12:1.
  • Spicer N, Berhanu D, Bhattacharya D, et al. The stars seem aligned: a qualitative study to understand the effects of context on scale-up of maternal and newborn health innovations in Ethiopia, India and Nigeria. Global Health. 2016;12:1–13 [Internet].
  • Smith SL. Political contexts and maternal health policy: insights from a comparison of south Indian states. Soc Sci Med. 2014;100:46–53 [Internet].
  • Eriksson L, Bergström A, Hoa DTP, et al. Sustainability of knowledge implementation in a low- and middle- income context: experiences from a facilitation project in Vietnam targeting maternal and neonatal health. PLoS One. 2017;12:1–16.
  • Sridharan S, Pereira A, Hay K, et al. Heterogeneities in utilization of antenatal care in Uttar Pradesh, India: the need to contextualize interventions to individual contexts. Glob Health Action. 2018;11:1517929 [Internet].
  • Belaid L, Ridde V. Contextual factors as a key to understanding the heterogeneity of effects of a maternal health policy in Burkina Faso? Health Policy Plan. 2015;30:309–321.
  • Witter S, Palmer N, Balabanova D, et al. Health system strengthening—reflections on its meaning, assessment, and our state of knowledge. Int J Health Plann Manage. 2019;34:e1980–9.
  • Pinzón-Flórez CE, Fernández-Niño JA, Ruiz-Rodríguez M, et al. Determinants of performance of health systems concerning maternal and child health: a global approach. PLoS One. 2015;10:1–27.
  • Sanneving L, Kulane A, Iyer A, et al. Health system capacity: maternal health policy implementation in the state of Gujarat, India. Glob Health Action. 2013;6:1.
  • Bucagu M, Kagubare JM, Basinga P, et al. Impact of health systems strengthening on coverage of maternal health services in Rwanda, 2000-2010: a systematic review. Reprod Health Matters. 2012;20:50–61 [Internet].
  • Singh NS, Huicho L, Afnan-Holmes H, et al. Countdown to 2015 country case studies: systematic tools to address the “black box” of health systems and policy assessment. BMC Public Health. 2016;16 [Internet]. DOI:https://doi.org/10.1186/s12889-016-3402-5.