Using the consolidated framework for implementation research (CFIR) to assess the implementation context of a quality improvement program to reduce missed opportunities for vaccination in Kano, Nigeria: a mixed methods study

ABSTRACT

Background: Although understanding implementation context is essential, there is a dearth of research on how to systematically explore it in quality improvement (QI) programs. Therefore, in this study, we used the consolidated framework for implementation research (CFIR) to guide a systematic evaluation of the implementation context of an ongoing QI program in order to generate rapid site-specific feedback that can be used to improve subsequent plan-do-study-act (PDSA) cycles.

Methods: Formative cross-case evaluation was conducted using convergent mixed methods design. The study was conducted in five primary health care (PHC) facilities (PHC 1, PHC 3, PHC 5, PHC 9 and PHC 10) implementing the QI program. Health workers in those facilities formed the study population. Quantitative data were collected using a self-administered, Likert-based rating tool, while qualitative data collection was guided by an interview guide. The interviews were transcribed verbatim, and thematic analysis was performed. Raw median score and factor scores were computed. Methodological integration occurred at the design, analysis and reporting stage.

Results: A total of 165 health workers were included in this study with a mean age of 33.43 years (standard deviation of 7.15). Majority were females and they all had post-secondary education. Health workers in two facilities; PHC 1 and PHC 5, reported higher score for the QI program across all five domains of CFIR. Implementation facilitators included intervention flexibility, relative advantage, self-efficacy among health workers, health workers confidence in the intervention, services integration. While implementation barriers included vaccine stock out, faulty cold chain infrastructure, lack of incentives, and socio-cultural beliefs.

Conclusion: This study demonstrated that theory-driven formative evaluation can be integrated in QI programs in a low resource setting. It buttressed the value in conducting such assessment as they can be used to generate rapid feedback on factors that influence implementation success which can then be addressed in subsequent cycles.

KEYWORDS:

• Immunization
• child health
• Nigeria
• mixed methods research
• consolidated framework for implementation research
• missed opportunities for vaccination
• quality improvement
• implementation research
• primary health care

Introduction

Missed opportunities for vaccination (MOV), which refers to any contact with a health facility by an unvaccinated or partially vaccinated child which does not result in the child receiving their recommended vaccines, is a poor indicator of the quality of care for children in health-care services.^1,2 This is because routine screening of immunization history followed by prompt provision of required vaccines are part of the quality of care standards for pediatric services.² In an effort to reduce MOV in primary health care (PHC) facilities in Nassarawa Local Government Area (LGA) of Kano State, Nigeria, health workers co-designed and implemented a collaborative facility-based quality improvement (QI) program. MOV was said to have occurred if a child aged 0–23 months who is eligible for immunization, makes contact with any of the PHC facilities and fails to receive all their recommended antigens. Kano, like many states in the geopolitical zone, has low immunization coverage level that is significantly below the Global Vaccine Action Plan (GVAP) target.³ As such, a QI program that seeks to improve immunization coverage by strengthening facility “in-reach” effort is an imperative.^1,4

After two plan-do-study-act (PDSA) cycles spanning a six weeks period, a decline in the proportion of MOV became apparent in two out of the five PHC facilities that were implementing the QI program in Kano, Nigeria. However, this is not surprising as several studies have reported that in QI interventions, progress toward attainment of the desired health outcome can be inconsistent across settings.^5,6 It has been suggested that such variations may be due in part to the implementation context as local contextual factors can affect implementation process.^7,8 These factors may be related to stakeholders such as health-care providers, users of health services, and health service managers among others.⁷ They can also be organizational, or policy related.⁷ Thus, investigating them while implementing a QI program can provide additional information that can be used to modify subsequent rapid cycles to improve progress toward the desired outcome across settings.

Recent advancements in implementation science have led to the proliferation of several frameworks and theories for studying implementation contexts.⁹–¹¹ But most of these theories are often missing one or more domains that are necessary for explaining the complex attributes of contexts.⁸ Therefore, a meta-framework known as the Consolidated Framework for Implementation Research (CFIR) was developed by fusing domains from these existing theories.⁸ This framework is comprehensive as it encompasses a wide array of domains that can affect implementation.⁸

In this study, we explored the implementation context of five PHC facilities that are implementing a QI program to reduce MOV through the lens of the CFIR framework. This was to understand the facilitators of, and barriers to implementation success. CFIR has five main domains, namely intervention characteristics, outer setting, inner setting, individual characteristics, and implementation process.⁸ There are 39 constructs distributed across these domains.⁸ The CFIR’s overarching domains make it suitable for assessing implementation context from a multi-level perspective.⁸ We defined each domain broadly. Intervention characteristics represented the features of the quality improvement intervention. Outer setting reflected the features of the external environment. Inner setting represented the features of the primary health-care system where the program is being conducted. Individual characteristics encompassed the features of the health worker, and implementation process referred to the strategies that were employed during the plan-do-study-act cycles.

There is a dearth of research on how to systematically explore the implementation context of an ongoing quality improvement program. This study examined implementation context through a theory-driven formative assessment that was embedded within the QI programme.⁸ This enabled quick identification of site-specific barriers that needs to be addressed and facilitators that should be sustained or promoted in subsequent cycles. The added advantage of conducting theory-driven assessments is that they can yield more holistic information for program improvement.¹² The objective of this study was to examine the contextual factors that affect the implementation of a quality improvement program to reduce missed opportunities for vaccination among children aged 0–23 months attending primary health-care facilities in Nassarawa Local Government Area of Kano State, Nigeria.

Methodology

Quality improvement program

Frontline health workers from five PHC facilities (coded as PHC 1, PHC 3, PHC 5, PHC 9 and PHC 10) co-designed a multifaceted and multimodal change package that are being implemented in a QI program. So far, two plan-do-study-act (PDSA) cycles have been implemented. The first PDSA cycles started on 3^rd January 2019 with the following change ideas: placing immunization reminder cards on the patient card of all persons attending the PHC with a child (or children) to prompt health workers to screen the child’s (or children’s) immunization history, distribution of MOV job aids in the facilities, modification of clinic schedule to enable daily immunization, and over-the-phone persuasion of husbands that refuse vaccination. This cycle lasted for 4 weeks. Then, another cycle commenced on 31 January 2019 which retained all the change ideas from the first PDSA but added the distribution of additional vaccine cold boxes to service delivery points to promote integration of immunization services. Initial analysis at 2 weeks after commencement of the second PDSA cycle indicated that progress toward reduction in the proportion of MOV among children aged 0–23 months was apparent in PHC 1 and PHC 5.

Study design and research paradigm

A formative cross-case evaluation was conducted after commencement of the second PDSA using convergent mixed methods design.¹³ Mixed methods design, as opposed to single designs (qualitative or quantitative) can aid better understanding of complex implementation context within the PHC facilities.¹³ Leveraging on the advantage of this design, phenomenon that would otherwise not have been identified using a single means were explored and identified.¹⁴ This design allowed for the collection of different but complementary data on factors that influenced the implementation of the QI program. By combining the two data types, we gained complementarity of perspectives and used this to reconstruct the meaning of each CFIR domain for the study. Since CFIR, which is a pre-existing theoretical framework was used, a subtle realism paradigm was adopted from an epistemological perspective.^8,15

Study setting

The study was conducted in five PHC facilities (PHC 1, PHC 3, PHC 5, PHC 9 and PHC 10) that are currently implementing a collaborative QI program to reduce MOV. These facilities are located in Nassarawa Local Government Area (LGA). This LGA is one of the metropolitan LGAs in Kano State.¹⁶ It is located in Kano Central senatorial district. It has an area of about 35 km² with a population of 596,669 according to the 2006 National Population and Housing Census.^17,18 The current projected population of the local government based on an annual growth rate of 3.3% is 880, 922.

Study population

Health workers (regardless of their cadre) that work in the five PHC facilities were included. To be eligible, the health worker must be aged 18 years and above and have been working in the facility for at least 4 weeks. This is to ensure that they have participated in at least one full PDSA cycle.

Sampling and sample size

For the quantitative procedure, the entire population of eligible health workers in the PHC facilities were enrolled in the study. These included staffs of all service delivery points including outpatient department, maternity, immunization among others. For the qualitative procedure, heads of units were purposively selected as key informants.

Data collection tool

Qualitative data collection tool: A semi-structured interview guide was developed to elicit response from key informants. This interview guide explored the perspectives of health workers that led the QI program in their facilities. The interview was flexible and allowed probing questions. The guide was pre-tested with health workers in Kano Municipal for clarity and appropriateness of questions.

Quantitative data collection tool: The CFIR was used to guide the de novo development of a semi-structured Likert-based QI implementation rating tool that was used in this study.⁸ It was a 5-point Likert scale with responses ranging from strongly disagree to strongly agree. The tool had two sections; section 1 collected background information while section 2 collected health workers’ rating of the QI program. The second section was structured based on the five domains of CFIR, namely intervention characteristics, inner setting, outer setting, characteristics of individuals, and implementation process.⁸ For each of the domains, items were developed. Intervention characteristics had eight items. Outer setting had four items. Inner setting had 12 items. Characteristics of individuals had four items. And implementation process had six items. The tool was iteratively revised for fluency, clarity, and adequacy by the researchers. It was then tested with health workers in PHC facilities in Kano Municipal for the appropriateness of sentences, structure, and order of questions. The health workers’ feedback was that the tool was clear and simple to use.

Data collection

Qualitative data collection: Interviews were used to obtain data from key informants. These key informants were heads of units and they served in the facility QI teams. The interviews were conducted face-to-face in a private and quiet room within the facility. The interviewer and respondent sat opposite each other. Each interview session lasted about 45 min to 1 hour. Interviews were conducted in the afternoon to avoid interrupting service delivery. All interviews were recorded using a portable digital audio recorder. In each of the facilities, saturation was attained early, usually after the third interview, and additional one or two interviews were conducted to confirm it. Reflection notes were kept and updated after each interview. An initial thematic analysis and coding was performed using summaries to note emerging themes. A total of 24 interviews were conducted across all five PHC facilities. Each of the 24 interviews were transcribed verbatim. All transcripts were reviewed for accuracy. In cases where a respondent made a sentence in Hausa language, this sentence was translated to English and back translated to the original language to ensure that its meaning was not lost in translation. All transcripts were reread multiple times.

Quantitative data collection: Data was collected using semi-structured self-administered QI implementation rating tool on mobile tablets. This was guided by an assistant. Data were collected in the afternoon after most patients had left to ensure minimal distraction and avoid any significant disruption of health service provision. To ensure that every health worker was included, the assistant returned to each facility the following day. Research Electronic Data Capture (REDCap) was used for data collection and management.¹⁹

Both qualitative and quantitative data were collected between 6^th – 15^th February 2019.

Data analysis

Qualitative data analysis

Template analysis approach was used.²⁰ After reading each transcript thoroughly, thematic content analysis was used to identify all factors that affected the implementation of the QI program inductively. The codes that were generated from the transcripts were compared with the initial codes that were generated during data collection, and this informed some slight refinement of the themes. All the themes identified were placed in a codebook. In another codebook, the CFIR domains were specified. Then, the codes that were inductively developed from the transcripts were deductively mapped to the CFIR domains. These CFIR codes were analytical as they required the researcher to interpret data from interviews and apply them accordingly. Overlapping was avoided by re-reading. Quotes were used to support each theme. Also, illustrative quotes that reflected the experiences of health workers for each domain per facility were presented using a joint display table.

Quantitative data analysis

Firstly, the reliability coefficient of the items in each domain was calculated. Since the items are Likert scales, ordinal alpha was used.²¹ A polychoric correlation matrix was fitted and then used to compute the ordinal alpha.²¹ Ordinal alpha was used because it estimates reliability coefficients more precisely than Cronbach’s alpha for ordinal variables.²² Raw median scores with their corresponding interquartile range (First quartile – third quartile) for each item within the domains was calculated per facility. Similarly, the median scores with their corresponding interquartile range (first quartile – third quartile) for each domain was also calculated per facility. Additionally, factor scores based on regression coefficients for each CFIR domain were computed in exploratory factor analysis. Bartlett test of sphericity and Kaiser-Meyer-Olkin measure of sampling adequacy were calculated for all items in each domain. Then, using the polychoric correlation matrix, factor analysis was performed. To decide on which factors to retain, we used factors with eigenvalues greater than 1 or point of inflection on scree plot or number of items loading on a particular factor. To ensure that the items are consistent with each other and factors are uncorrelated, orthogonal varimax rotation was used. The factors scores for each domain per facility were presented using a joint display table. Analysis was performed in R-Studio which is an integrated development environment for R using the package psych and STATA 14.2 College Station, Texas.

Integration of mixed methods

In this study, methodological integration occurred at three stages; design, analysis, and reporting. In the design stage, both qualitative and quantitative data were collected within the same time frame in February 2019. Although in parallel, both types of data were collected from same group of health workers. In the analysis stage, integration occurred through connecting. This is because the interview respondents were a subpopulation of the participants that participated in the survey. In the reporting stage, integration occurred through the use of joint display. This involved using a table to organize and present both quantitative and qualitative data together.

Ethics approval

Ethical clearance for this study was obtained from Stellenbosch University Health Research Ethics Committee (with reference number: S18/02/044), Kano State Ministry of Health (with reference number: MOH/Off/797/T.I/374) and Aminu Kano Teaching Hospital (with reference number: NHREC/21/08/2008/AKTH/EC/2296). An information sheet was read to respondents and written informed consent was obtained. It was clearly explained to the study participants that they could decline to respond to any of the questions or exit the study at any time. To ensure anonymity, identifiers were not collected.

Results

Characteristics of health workers

A total of 165 health workers were included in the study. As shown in Table 1, the mean age of the health workers was 33.43 with standard deviation of 7.15. Over half were aged 20–34 years. Majority of health worker were females, and all had post-secondary qualification. Twenty-three percent were community health workers.

Table 1. Background characteristics of health workers in primary health-care facilities that implemented quality improvement program in Nassarawa local government area of Kano State.

Variable	Mean (SD)	Min; Max
Age	33.43 (7.15)	20; 52
Variables	Frequency	Percentage
Age group (in years)
20–34	92.00	55.76
35–52	73.00	44.24
Sex
Male	41.00	25.00
Female	124.00	75.00
Level of education
No formal education	0.00	0.00
Primary education only	0.00	0.00
Secondary education	0.00	0.00
Post secondary education	165.00	100.00
Cadre of health worker
Community health worker	39.00	23.64
Environmental health assistant	32.00	19.39
Auxillary nurse	1.00	0.61
Registered nurse	0.00	0.00
Pharmacy technician	10.00	6.06
Pharmacist	2.00	1.21
Medical Doctor	0.00	0.00
Others	81.00	49.09

Qualitative findings

Intervention characteristics

Flexibility and ease of implementation

Across the facilities, most of the health workers expressed that the quality improvement program was easy to implement and flexible. We found that this evoked a desire for broader participation and several participants recommended that the program be scaled up to other health facilities with the local government. Some of the reasons why they consider it easy to implement was because of the teamwork that was involved:

“It is very easy. It is not only one person that conduct the activity. We joined our hands together, all the unit heads in the clinic.” – PCH 3

“To be specific the activities is very flexible because we met and deliberated on methods or initiatives that we will apply” – PHC 3

Others even felt that what is required is persuasion and expressed that it is not difficult:

“There is nothing difficult it is just to talk and convince the clients about the importance of this initiative.” – PHC 5

Relative advantage of QI

In most of the facilities, health workers expressed strong positive perception about the advantage of the quality improvement program. They felt that it was a less time-consuming intervention since unimmunized children are detected in the health facility.

“I am sure this initiative is going to work.” – PHC 5

“We used to go out and look for unimmunized and defaulters, but this QI Initiative bring out the best ways to detect them at your own doorstep without wasting time.” – PHC 5

Some of the participants felt that the QI intervention is better than other interventions because it focuses on all the children that are visiting the health facility.

“It is better because now it involves everybody whether that person has come for immunization or not.” – PHC 10

Inner setting

Supportive supervision

The participants across facilities did not only acknowledge the importance of supervisory visits but affirmed that they received such visits while implementing the quality improvement program. Participants expressed satisfaction with the supervisory plan that was put in place for the quality improvement program as it enabled quick feedback. Supervisory visits were conducted by different stakeholders that are higher-ranking officials within the health systems. These include local government and zonal primary health-care management board officials.

“We receive supervision from local government, they use to come and supervised us to check how we conduct our duties.” – PHC 3

“the HOD had to come here for supervision to see how effective the intervention is.” – PHC 9

Vaccine cold chain

A participant in one of the facilities expressed an important gap in their inability to maintain vaccine cold chain which can affect the availability of antigens.

“We have solar refrigerator, but it is now faulty.” – PHC 5

However, as an adhoc measure to sustain availability of vaccines, which was informed by a shared perception of the importance of the quality improvement program, the facility instituted a plan to collect antigens on a daily basis from the cold chain office.

“In the morning I will request for the type of vaccines to be used on that day from the office of the CCO. After close of business also I arrange and send the remaining vaccines that were not used on that day to the same office.” – PHC 5

Vaccine stock out

Generally, it was found that the LGA team supported facilities to ensure availability of vaccines, but this level of support was not uniform. The issue generated a lot of interest among participants as the emphatically stated that the success of the quality improvement program depended on adequate stock level of all antigens.

“We get support from LGA, they supply us with vaccines in the event we have shortages because we immunized everybody.” – PHC 3

“we also have enough working materials and enough vaccines.” – PHC 5

In one of the facilities, BCG stock out experienced:

“last week BCG was not available, and we complained that BCG was not available, so we had to refer clients to another facility where BCG is available” – PHC 9

Concerns were expressed that if such referral was to a facility that did not offer daily immunization like them, there are chances that they child would still be missed.

Leadership engagement within facility

We found a strong readiness for implementation in one of the facilities. This was informed by the level of within-facility leadership interest and engagement that was expressed. It seems that there was a pre-existing internal bonding between health workers and the head of the facility, and this social capital positively influenced the QI program implementation as well.

“our facility in charge is a hardworking officer it is because of her hard work, that we under her supervision also cooperate with her in order to achieve this success.” – PHC 5

Outer setting

Lack of incentives

Despite being integrated into their routine work at the facility, participants expressed that some health workers were expecting some form of incentives. Some felt it was additional work especially those in out-patient departments.

“Initially these staffs were thinking they will be paid” – PHC 9

This posed some challenges at the initial stage as the program was resisted in some facilities.

Acceptability of vaccines among caregivers

Vaccine acceptance also generated a lot of discussion among participants. It was found that the influence of husbands were an important impediment. Although children were making contact with health workers at service delivery points like family planning, or accompanying caregiver, attempts to offer immunization are often met with strong resistance.

“When we try to immunize her child, she will complain that her husband doesn’t know” – PHC 3

“Maybe she just came for family planning or she came to visit someone that delivered, and she came along with her child, if I try to convince her to accept immunization, she will say that the father didn’t know.” – PHC 10

Sometimes, attempts to offer immunization services among children who are visiting health facilities for other reasons evoke strong emotional response which often discourages the health workers from further persuasion.

“She has to tell her husband and some cry. If you want to immunize her children, she will start crying that she doesn’t want to, and you cannot force her.” – PHC 9

Individual characteristics

Self-efficacy in personnel

There is a common understanding among participants about the need for more commitment. One of the routine immunization focal person expressed how she has modified her behavior to be able to properly execute the QI intervention in an effort to achieve the desired goals for her facility.

“I come to work early because patients don’t like to wait. I bring out my data tools early and talk to patients in a nice way even if they come late. I advise them to come early. As a leader in my unit who has been trained, I have to be kind to patient because we are looking for as many children as possible to immunize.” – PHC 5

Health workers belief about the QI program

Among the participants, the value placed on the quality improvement program was quite positive. Some of the health workers attested that the immunization program in their facility has improved because of the quality improvement program. Some of the areas of improvement that they highlighted were reduction in number of immunization defaulters and daily immunization service provision.

“Our facility has improved a lot from this QI initiative” – PHC 3

“After the first, second and third week everything changed completely. I even asked myself that is it possible for things to change within such minimal period of time.” – PHC 3

“It has benefited this facility because now our defaulters have gone down. Secondly, clients are also enjoying this thing as immunization is now on daily basis” – PHC 10

Implementation process

Engagement

Participants Although only quality improvement team members from each facility attended the central training, they ensured that training was cascaded down to other health personnel in their primary health-care facilities.

“After we came back from training, we also organized a stepdown training for all the staff” – PHC 5

Conducting a step-down training within facilities promoted better cooperation from other staffs that were not traditionally involved with immunization:

“we got full cooperation of our staff and clients and every staff consider himself RI Focal Person because of the stepdown training we conducted and the weekly meeting that they attended.” PHC 5

Some facilities took advantage of this step-down training to include all staffs:

“from security guard, cleaners, casual and all staff be it professionals or non-professionals are engaged in this QI initiatives in our health facility. We also involved our TBAs that are living within the community, we train them about this program and also attend all our meetings. This TBAs used to enlighten and make women aware on the importance of immunization at any social gatherings.” – PHC 1

Weekly facility meetings

One key factor that promoted smooth and consistent execution of the quality improvement program was the weekly facility meetings that were supported as part of the implementation strategies. This meeting served as an avenue to discuss challenges and performance.

“Another thing is we used to meet at every Thursday of the week to discuss different modalities among ourselves. This really helped us to achieve” – PHC 3

“We had to meet every week with all the heads of the units and health personnel. We sit every Thursday of the week and discuss the challenges and the achievements.” – PHC 9

In addition, the meetings were used as an avenue to conduct refresher trainings so that all health workers know more about vaccines.

“The first one we trained all our staff to have knowledge of vaccines, so every member of our staff have that things in his mind and immediately he sees the patient, first thing he will ask after greetings, what are the immunization status of your children,” – PHC 3

“every Thursday of the week all staff are invited to this meeting starting from security, casual and permanent staff of this facility. We then asked ourselves what we understand by missing opportunity child and every member must answer this question that a missing opportunity child is a person that comes to the hospital and do everything he could and walkout from the hospital without being immunized. At this meeting all of us will present what he did in the previous week under review.” – PHC 1

Integration of immunization services

One key area of where integration of immunization services was strengthened during this quality improvement program, based on the discussion with participants, was in maternity and labor room. However, it was still noted that health workers are reluctant to open BCG vial for few children.

“So immediately after ANC closed from morning shift they will transfer the vaccines directly to maternity to avoid missing opportunity.” – PHC 9

“We keep two vaccines and immediately a woman give birth here we give the OPV and Hepatitis B vaccine to the child. But if it happens the woman delivers on Thursday, or in the night we usually allow her to go home with the permission of the matron in charge so that she can come back the following morning for BCG vaccine” – PHC 5

“We to take the BCG and Hepatitis B vaccine to the maternity because we used to close by 4.00pm here in the OPD. So if they have any delivery after that time they give that immunization” – PHC 10

In fact, participants expressed that children that were delivered at home were linked to immunization services during post-natal visits:

“In the event a woman delivers at home or somewhere and come for postnatal visit and come along with the child we enquire about the child’s immunization status.” – PHC 5

Screening home-based records

Across all facilities, reminder tags were implemented across multiple service delivery points. These tags were attached to all hospital cards to prompt health workers to screen the immunization history of all children.

“We have tag which is asking about child’s status” – PHC 3

“If a patient relative comes to the maternity along with a child we also ask the mother about the child’s immunization status. If we found out that the child is not immunized, we quickly immunized him/her.” – PHC 5

It was confirmed that the use of a reminder card was necessary because health workers in service delivery points other than immunization were not paying attention to a child immunization status or making active effort to ensure that a child is vaccinated.

Before our staff don’t care whether your child is immunized or not. The concern is only to prescribe. Now, they must ask, and they have a reminder” – PHC 9

Daily immunization

Across all facilities, immunization is now provided on a daily basis. In addition to reducing missed opportunities for vaccination, participants said daily immunization has reduced workload in the immunization clinic. “Our workload has decreases”. With daily immunization, caregivers do not have to wait till specific days to bring their children to the hospital.

“Before we only conduct immunization on Thursday and Friday but now, we do immunization from Monday to Friday. On Saturdays and Sundays, even though we close early, maternity keeps some vaccines in case someone delivers so that the new child does not leave the facility without receiving their immunization.” – PHC 3

“We now conduct immunization daily from Monday to Friday and our assignment is beyond hospital we also immunization outreach inside the community to detect defaulters.” – PHC 5

“Before we are providing immunization only on Thursday, but due to implementation of this QI, we now offer immunization everyday” – PHC 10

Community defaulter tracking

We found that facilities also extended activities to community by employing social mobilization strategies to screen children within their catchment area and provide vaccination through volunteer community mobilizers (VCM).

“We also implement same in our community through our VCM to search for defaulters.” – PHC 5

Using bed nets as incentives for caregivers

We found that one of the facilities was leveraging on other public health programs to improve completed immunization. They gave long-lasting insecticide treatment nets to children who completed all the immunization in the schedule.

“We got support of mosquito nets which we distributed to any child that completed his immunization” – PHC 1.

Quantitative findings

The ordinal alpha for the items in each of the five domains; intervention complexity, outer setting, inner setting, individual characteristics, and implementation process, were 0.67, 0.49, 0.69, 0.57, and 0.77, respectively. Table 2 shows the raw scores for each of the items per facility. As shown in Table 3, PHC 1 had score of 4 (IQR:4–5) for intervention characteristics while other facilities had a score of 4 (IQR:4–4) for same domain. Similarly, PHC 1 had a score of 4 (IQR:4–4.5) for out setting and implementation process while other facilities had a score of 4 (IQR:4–4) for the domains. Other scores per facility per domain are shown on the table.

Table 2. Raw scores for each item in the CFIR domain for each facility that implemented quality improvement in Nassarawa LGA, Kano.

	PHC 3	PHC 9	PHC 5	PHC 10	PHC 1
CFIR Domain and items	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)
Intervention characteristics
The quality improvement strategies (interventions) were locally developed by stakeholders from this health facility.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
Health workers from this facility were fully involved in choosing the change ideas to be tested.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
I trusted that this quality improvement initiative would reduce missed opportunities for vaccination and improve the performance of our immunization service.	4 (4–4.5)	4 (4–4)	5 (4–5)	4 (4–4)	4 (4–5)
This quality improvement approach is better than other types of interventions for reducing missed opportunities for vaccination and improving performance of immunization systems.	4 (4–4)	4 (4–5)	4 (4–5)	4 (4–4)	4 (4–5)
I consider quality improvement approach to be very flexible.	5 (4–5)	4 (4–4)	4 (4–5)	4 (4–4)	5 (4–5)
I consider this quality improvement initiatives to be very simple to implement.	4 (4–5)	5 (4–5)	4 (4–5)	5 (4–5)	5 (4–5)
I am very happy with the way the change ideas were delivered in cycles.	4 (4–5)	4 (4–5)	4 (4–5)	4 (4–4)	4.5 (4–5)
I consider quality improvement approach to be an inexpensive strategy for reducing missed opportunities for vaccination in primary health care setting.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
Outer setting
I consider missed opportunities for vaccination to be a very important problem in our health facility.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
This primary healthcare facility is a key immunization service delivery center in this community.	4 (4–4)	4 (4–4)	4 (4–5)	4 (4–4)	4 (4–5)
I was interested in using quality improvement approach because I heard it has been successful in other places.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
Our facility will be rewarded by the local government and state ministry of health if we reduce missed opportunities for vaccination and improve the performance of our immunization clinic.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
Inner setting
There are enough healthcare workers in this facility.	4 (4–5)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
There is a health worker dedicated to managing the immunization clinic.	4 (4–5)	4 (4–4)	4 (4–5)	4 (4–5)	4 (4–5)
Immunization services are provided daily in this facility.	5 (4–5)	4 (4–5)	5 (4–5)	5 (4–5)	5 (5–5)
There was a social media group to keep everyone informed about meetings and key activities regarding quality improvement.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
It was easy to relate with all the members of the quality improvement team.	4 (4–4.5)	4 (4–4)	4 (4–5)	4 (4–4)	4 (4–4)
I will receive a financial reward for using quality improvement to reduce missed opportunities for vaccination and improving immunization services in this clinic.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
I will get a promotion for quality improvement to reduce missed opportunities for vaccination and improve immunization services in this clinic.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
It is easy to implement new ideas like quality improvement in this primary healthcare facility.	4 (4–4)	4 (4–4)	4 (4–5)	4 (4–4)	4 (4–5)
Mothers and caregivers consider this quality improvement initiatives to be very valuable and important.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
There was a good support system in place to seek more information about the quality improvement initiatives.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
The quality improvement team are very committed.	5 (4–5)	5 (5–5)	5 (5–5)	5 (5–5)	5 (4–5)
Their was easy access to information about the quality improvement initiatives that were implemented.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
Characteristics of individuals
I was confident in myself during the implementation of the quality improvement initiatives in my facility.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
I was familiar with the quality improvement strategies before implementation began.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
I consider myself an important stakeholder for reducing missed opportunities for vaccination in this health facility.	4 (4–4)	4 (4–5)	4 (4–5)	4 (4–4)	4 (4–4)
I was highly motivated and competent to implement quality improvement initiatives this health facility.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
Process
There was adequate planning and preparation before implementation of the quality improvement ideas commenced in this health facility.	4 (6)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
There was adequate involvement of mothers and caregivers as well as health workers in this facility in planning and implementation of the quality improvement initiatives.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
There was involvement and participation from local government health officials.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
There was a quality improvement team charged with the monitoring the change ideas.	4 (4–4)	4 (4–5)	4 (4–5)	4 (4–5)	4 (4–5)
There was an external person who provided support and direction for this quality improvement initiative.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
There was a good feedback mechanism for displaying progress and sharing experiences.	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)

Table 3. Raw implementation scores for each CFIR domain in facilities that implemented quality improvement in Nassarawa LGA, Kano.

	PHC 3	PHC 9	PHC 5	PHC 10	PHC 1
CFIR Domain	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)
Intervention characteristics	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–5)
Outer setting	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4.5)
Inner setting	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)
Characteristics of individuals	4 (4–4)	4 (4–4)	4 (4–4.5)	4 (4–4)	4 (4–4)
Process	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4)	4 (4–4.5)

*PHC = Primary Health Care *IQR = Interquartile Range *CFIR = Consolidated Framework for Implementation Research

As shown in Table 4, in PHC 1, the factor score for intervention characteristics was 5.40 with a standard deviation of 0.41. The illustrative quote from one of the participants in that facility suggests that the intervention is comparatively better than other interventions because it reminds health workers of their responsibilities. PHC 1 and PHC 5 had factor scores of 4.03 and 3.83, respectively, for implementation process domain. The illustrative quotes indicate engagement of broad engagement of facility staffs including those that are based in the community. Score and illustrative quotes for other facilities can be found on the table.

Table 4. A cross-case joint display showing illustrative quotes and implementation factor scores based on varimax rotation for each CFIR domain in primary health-care facilities that implemented quality improvement in Nassarawa LGA, Kano.

	PHC 3	PHC 9	PHC 5	PHC 10	PHC 1
CFIR Domains	Mean (SD) & Quotes	Mean (SD) & Quotes	Mean (SD) & Quotes	Mean (SD) & Quotes	Mean (SD) & Quotes
Intervention characteristics	5.23 (0.36) “To be specific the activities is very flexible because we met and deliberated on methods or initiatives that we will apply”	5.07 (0.29) “it is flexible”	5.09 (0.36) “There is nothing difficult it is just to talk and convince the clients about the importance of this initiative.”	5.09 (0.28) “It is not complex, it is flexible”	5.40 (0.41) “The reason why this initiative is better than others is because it reminds health workers of their responsibilities. They complain about defaulters and now there is an initiative to help reduce it. This is a very good programme.”
Outer setting	4.18 (0.25) “When we try to immunize her child, she will complain that her husband doesn’t know”	4.17 (0.23) “initially these staffs were thinking they will be paid”	4.26 (0.28) “ … she said she was not interested, her husband does not like immunization at all. But by the grace of God I was able to convince her after a very long time with a patient”	4.17 (0.24) “Maybe she just came for family planning or she came to visit someone that delivered, and she came along with her child, if I try to convince her to accept immunization, she will say that the father didn’t know.”	4.38 (0.33) Sincerely, some women do not like to immunize their children. For these types of women, we take them aside and counsel them on the dangers of not vaccinating a child. They’d say it’s the husband that don’t want it. We then collect the husband’s number and explain to them. And they usually agree.”
Inner setting	4.30 (0.22) “We receive supervision from local government, they use to come and supervised us to check how we conduct our duties.”	4.36 (0.19) “the HOD had to come here for supervision to see how effective the intervention is.”	4.42 (0.28) “We receive supervision from the senior officials, and they used to come from time to time.”	4.37 (0.21) “There are supportive supervisions from the LGA to see how we are doing it.”	4.45 (0.33) “we receive supervision from the (LIO) Nassarawa LGA he attends our meeting every week and also, they provide us with refreshment during the meeting.”
Characteristics of individuals	2.80 (0.31) “Our facility has improved a lot from this QI initiative”	2.89 (0.37) “I go round the whole facility to maternity to ANC, from ANC to OPD to see how far, are people complying”	2.97 (0.42) “I come to work early because patients don’t like to wait. I bring out my data tools and talk to patients in a nice way even if they come late. I advise them to come early too. As a leader in my unit who has been trained, I have to be kind to patient because we are looking for as many children as possible to immunize.”	2.79 (0.33) “It has benefited this facility because now our defaulters have gone down. Secondly, clients are also enjoying this thing as immunization is now on daily basis”	2.79 (0.23) “I got to understand that this work involves every health personnel whether you in pharmacy, lab, family planning unit or RI unit.”
Process	3.64 (0.12) Before we only conduct immunization on Thursday and Friday but now, we do immunization from Monday to Friday. On Saturdays and Sundays, even though we close early, maternity keeps some vaccines in case someone delivers so that the new child does not leave the facility without receiving their immunization.”	3.75 (0.11) “We had to meet every week with all the heads of the units and health personnel. We sit every Thursday of the week and discuss the challenges and the achievements.”	3.82 (0.26) “We also implement same in our community through our VCM to search for defaulters.”	3.75 (0.11) “Before we are providing immunization only on Thursday, but due to implementation of this QI, we now offer immunization everyday”	4.03 (0.43) “from security guard, cleaners, casual and all staff be it professionals or non-professionals are engaged in this QI initiatives in our health facility. We also involved our TBAs that are living within the community, we train them about this program and also attend all our meetings. This TBAs used to enlighten and make women aware on the importance of immunization at any social gatherings.”

Discussion

In this study, we used convergent mixed methods approach to investigate the implementation context of a collaborative QI program in five PHC facilities in Nassarawa LGA, Kano, Nigeria, to identify implementation facilitators and barriers. Using CFIR enabled a more comprehensive formative assessment that led to the identification of actionable findings that can be used to adjust subsequent PDSA cycles.

Formative assessment in QI programme to reduce MOV

The value of integrating formative assessment in health systems intervention has been emphasized by researchers.²³ This study demonstrated how such assessments can be conducted in health-care QI programs using qualitative and quantitative data. In addition, it also led to the identification of site-specific factors that influence implementation. This can inform rapid adaption of implementation strategies and the refinement of change ideas. Hence, the study advanced the argument for embedding formative assessment in health systems program implementation.^23,24

Using formative assessment to evaluate contextual influencers is critical in informing a better understanding of the intervention across setting.²³ In this study, it was found that health workers in PHC1 gave bed nets to caregivers that completed immunization. Similarly, health workers in PHC5 intensified their community defaulter tracking during the QI program. These tactics are important modifications that occurred within specific sites and they might have contributed to the overall progress toward the reduction of MOV among children.

Embedding formative assessment in this QI program provided deeper insights into the barriers that might have affected implementation within sites and impeded progress toward reduction in MOV. For example, vaccine stockout were reported in some of facilities. This is not surprising as stock out events are a common occurrence in low- and middle-income settings.²⁵ Failure to ensure constant supply of all antigens in the PHC facilities during the QI program would sustain the occurrence of MOV regardless of the kind of change ideas being implemented. Similarly, faulty cold chain refrigerators and negative social-cultural beliefs about immunization were reported in some sites. Thus, the persistence of MOV likely reflects the presence of these barriers in the implementation context and not necessarily the non-effectiveness of the change ideas.

Some positive factors such as self-efficacy, confidence in the QI intervention, reported intervention flexibility, and service integration were identified as facilitators of implementation. Some of these factors were more apparent in some implementation context than others and could potentially have influenced the variation in progress toward the reduction of MOV across facilities.

Implications for the QI program

Although most of the health workers described the QI program as flexible and easy to implement and expressed that it had better relative advantage compared to other interventions, the score for intervention characteristics was not uniform across all sites. Based on the Likert scale responses, PHC 1 had the highest score in the intervention characteristics domain compared to other facilities. The high score suggests that QI team members probably engaged the rest of the staffs more robustly in the pre-implementation phase to inform such positive perception. The health workers in this facility saw the intervention as an advantage to improve their work as it can reduce the number of immunization defaulters. For example, if a child who has defaulted immunization attends the out-patient department (OPD) for other childhood illnesses, they would be identified and immunized. The intervention characteristics domain gauges the perception of implementers and adopters regarding an intervention. Before commencing the next PDSA cycle, QI teams in the other PHC facilities should rigorously re-engage all health workers in their facilities in deciding the change ideas to implement.

Inner setting explored factors within the PHC system itself. One important facilitator within this domain that was consistent across all PHC facilities was supportive supervision by senior officials from the local government or primary health-care management board zonal office. Supervision provides an opportunity for onsite feedback during the implementation process. In PHC 5, the implementation climate, shaped by leadership engagement was considered a facilitator. The level of respect for the facility in-charge, garnered strong commitment from the rest of the staffs. Barriers that were identified within this domain include vaccine stockout and inability to sustain cold chain due to faulty infrastructure. Although the facility head in PHC 5 instituted an ad hoc measure to support daily collection of antigens from another site, such measures might not be sustainable in other sites without the availability of funds. Therefore, in the next PDSA cycle, site-specific mechanism should be put in place to facilitate the collection of antigens from the local government expanded program on immunization (EPI) office. For sustainability, the local government EPI manager need to strengthen the system to prevent vaccine stockout, especially for Bacillus Calmette–Guerin (BCG). Overall, PHC 1 and PHC 5 still had higher scores in this domain compared to other facilities.

The outer setting broadly encompasses all external influences.⁸ In this study, we found barriers that are related to policies (lack of incentives) and the socio-cultural beliefs of caregivers. The QI program did not include incentives as part of the change ideas. Caregivers resist immunization if their husband did not give prior approval. Although health workers usually counsel caregivers on the benefits of vaccines and initiate phone conversation to persuade their husbands, to comprehensively address this problem, community-wide behavior change interventions are necessary.

Health worker behavior can influence the implementation of interventions.⁸ In this study, we identified self-efficacy and beliefs about the QI as important facilitators. When an individual believes in their capabilities to implement an intervention, they modify their behavior to achieve the desired goal. This was apparent among health workers in some of the PHC facilities. In PHC 5, one of the health workers said that she modified her work resumption time in order to provide immunization services to more children. Furthermore, health worker beliefs and perception about an intervention is important.⁸ The value placed on an intervention, which is largely informed by background working knowledge of that intervention influences adoption.⁸

Implementation process explored the activities that were implemented during the PDSA cycles. One key facilitator within this construct that was consistent across the PHC facilities was the integration of immunization services in the maternity section of the facilities which included antenatal and family planning clinic, or labor room. Service integration is considered an important investment area for immunization programs and is a key component of GVAP.²⁶ Other activities in the QI program were implemented across all facilities.

Limitations and strengths

Although all health workers in the five PHC facilities were included, we were limited by sample size. As such, our findings are not generalizable. However, as a formative evaluation, the intention is to provide facility-specific feedback that can be used to enhance the QI program. Since the rating tool collected self-reported data, social desirability bias is a potential limitation. The convergent mixed methods design enabled the integration of quantitative and qualitative data which improved the validity of our findings. This mixed methods design allowed us to gain additional insights into the implementation context of the QI program across facilities.

Conclusion

This study demonstrated that theory-driven formative evaluation can be integrated in a QI program in a low resource setting. It buttresses the value of conducting such assessment as they can be used to generate rapid feedback on context-specific factors which can then be addressed in subsequent PDSA cycles. CFIR proved to be a useful theoretical framework as it facilitated the systematic analysis of multidimensional factors.

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.

Author contribution

AAA conceptualized the study, conducted interviews, analysis, interpretation of findings, and wrote the first draft of the manuscript. CSW, OAU, MAG contributed to statistical analysis and interpretation. CSW, OAU, MAG reviewed, supervised and approved. All authors approved the final manuscript.

Acknowledgments

The authors would like to thank the management team of Aminu Kano Teaching Hospital, Kano, Nigeria for providing administrative support during the project. We would also like to thank the Kano State Primary Health Care Management Board (KSPHCMB).

Additional information

Funding

The research reported in this publication was supported by the South African Medical Research Council with funds received from the National Research Foundation of South Africa through its competitive programme for rated researchers. This work is based on research supported wholly/in part by the National Research Foundation of South Africa (Grant Number: 106035). OAU receives support from National Institute of Health’s Official Development Assistance (ODA) funding. The views expressed in this publication are solely those of the authors.