Methods used for immunization coverage assessment in Canada, a Canadian Immunization Research Network (CIRN) study

ABSTRACT

Accurate and complete immunization data are necessary to assess vaccine coverage, safety and effectiveness. Across Canada, different methods and data sources are used to assess vaccine coverage, but these have not been systematically described. Our primary objective was to examine and describe the methods used to determine immunization coverage in Canada. The secondary objective was to compare routine infant and childhood coverage estimates derived from the Canadian 2013 Childhood National Immunization Coverage Survey (cNICS) with estimates collected from provinces and territories (P/Ts). We collected information from key informants regarding their provincial, territorial or federal methods for assessing immunization coverage. We also collected P/T coverage estimates for select antigens and birth cohorts to determine absolute differences between these and estimates from cNICS. Twenty-six individuals across 16 public health organizations participated between April and August 2015. Coverage surveys are conducted regularly for toddlers in Quebec and in one health authority in British Columbia. Across P/Ts, different methodologies for measuring coverage are used (e.g., valid doses, grace periods). Most P/Ts, except Ontario, measure up-to-date (UTD) coverage and 4 P/Ts also assess on-time coverage. The degree of concordance between P/T and cNICS coverage estimates varied by jurisdiction, antigen and age group. In addition to differences in the data sources and processes used for coverage assessment, there are also differences between Canadian P/Ts in the methods used for calculating immunization coverage. Comparisons between P/T and cNICS estimates leave remaining questions about the proportion of children fully vaccinated in Canada.

KEYWORDS:

• Canada
• immunization coverage
• immunization registries
• immunization registers
• vaccine-preventable diseases

Introduction

Immunization coverage is essential for immunization program monitoring and evaluation. Such data are used to determine if coverage targets are achieved, to identify communities and sub-populations which may be in need of supplemental immunization activities, and to assess the success of targeted campaigns. High quality coverage data are an essential input into effective decision-making for immunization programs at the local, provincial and national levels. In all United Nations countries, including Canada, coverage data are necessary to support enhanced surveillance initiatives for the documentation and certification of elimination status for diseases such as measles, rubella and polio.^1-3

In Canada, health service delivery including immunization is the responsibility of provincial and territorial governments. The National Advisory Committee on Immunization (NACI) provides guidance on vaccines and immunization schedules, but each province and territory (P/T) has its own publicly-funded immunization schedule and its own system(s) for assessing immunization coverage.⁴ There is considerable variation between the data sources and processes used for immunization coverage assessment by P/Ts. At the federal level, routine immunization surveys are regularly conducted by the Public Health Agency of Canada (PHAC) which for the first time in 2013, involved an expanded sampling frame to allow for the calculation of P/T coverage estimates. Previous work by our group has outlined the variability in immunization information systems (IISs), infrastructure and capacities at the P/T level in Canada.⁴ Our group hypothesized that similar heterogeneity in methods for the calculation of immunization coverage also exists.

This study aimed to document the methods used by Canadian P/Ts and by the childhood National Immunization Coverage Survey (cNICS) of PHAC. Our secondary objective was to compare routine infant and childhood coverage estimates derived from the 2013 cNICS with estimates derived from P/Ts to examine concordance between the 2 sources. Information collected about the challenges and processes used for coverage assessment among First Nations children living on-reserves in Canada is summarized in a separate manuscript.⁴

Results

Study sample

Twenty-six individuals across 16 public health organizations participated in the interviews and represented 12 of Canada's 13 P/Ts [British Columbia (BC, n = 1 individuals interviewed), Alberta (AB, n = 2), Saskatchewan (SK, n = 3), Manitoba (MB, n = 1), Ontario (ON, n = 5), Quebec (QC, n = 1), New Brunswick (NB, n = 1), Nova Scotia (NS, n = 2), Prince Edward Island (PEI, n = 1), Newfoundland and Labrador (NL, n = 2)], Nunavut (NU, n = 1), Northwest Territories (NWT, n = 1), one federal organization [Public Health Agency of Canada (PHAC) (1)], and 2 First Nations organizations [First Nations Health Authority (FNHA) of BC (1), First Nations and Inuit Health Branch (FNIHB) (3)]. Participants included vaccine program managers, epidemiologists, immunization nurses, communicable disease control specialists, nurse consultants, public health information system specialists and a medical health officer.

Eight provinces provided coverage estimates or referral to a publicly-available coverage report: BC, AB, SK, MB, ON, QC, NL, and PEI. The jurisdictions that did not provide coverage data were the participants from Canada's northern territories (NWT and NU) and 2 eastern provinces (NS, NB).

Coverage surveys

Coverage surveys are conducted bi-annually in QC (at 15 and 24 months of age) and conducted periodically in one regional health authority (Vancouver Coastal Health) in BC among 2-year-olds, to address known gaps in immunization information. At the time of data collection, QC was using several unlinked local immunization systems exist, Systèmes d'information sur la clientèle et les services des Center Locaux de Services Communautaires (I-CLSC), which had limited capacity to assess coverage and were recently replaced with Panorama, a centralized IIS used by other Canadian jurisdictions. In BC, the Vancouver Coastal Health Authority (VCHA), representing approximately 20% of the provincial population, uses a different electronic IIS than the rest of the province and it does not include all immunizations administered by physicians, who administer the majority of immunizations for infants and young children. The coverage surveys are conducted by telephone in Vancouver and by mail in QC, using population-based sampling frames based on the provincial health insurance database (QC) or Ministry of Health Client Registry (VCHA). Two northern regions that represent less than 1% of the population of the province are excluded from the QC survey.

Coverage measurement methodology

The remaining participating P/Ts assess coverage using the IIS or another immunization record review process. Three jurisdictions (ON, NS and NB) have no existing mechanism to assess immunization coverage at the age of 2 y, a national and internationally recognized milestone age for coverage assessment,^5,6 instead each of these jurisdictions rely upon assessment at school-entry (Table 1). Depending on the age or grade assessed, there may be differences in data collection. For example, in BC the IIS is used to assess coverage only in 7-year-olds (with the exception of VCHA), while Grade 6 and 9 coverage is assessed at the provincial level through a combination of registry data and aggregate reporting by the regional health authorities.

Table 1. Methods used for immunization coverage assessment, by P/T.

	NWT	BC	AB	SK	MB	ON	QC		NS	NL	PEI	NB
Measures up-to-date coverage	^✓	^✓^1	^✓	^✓	^✓	x	^✓		^✓	^✓	^✓	^✓
Measures on-time coverage	^✓	^✓	x	^✓	^✓	x	^✓^2		x	x	x	x
Measures coverage using other method(s)	x	^✓^3	^✓	x	x	^✓^4	x		x	x	x	x
Coverage definition includes valid doses only	valid	valid	valid	valid	valid	valid	valid and all^5		valid	valid	all	valid
Grace period^6 considered for valid dose assessment	n/a	x	^✓	^✓	^✓	x	x		x	x	n/a	^✓
Denominator data source for coverage assessment	HI and statistics bureau	Mixed^7	HI	Mixed^8	HI	SL	Mixed ^9		SL	SL	Mixed^10	SL
Age milestones for assessing coverage	2 yr.	2 , 7 yr.	2 yr.	2, 7 and 17 yr.	1, 2,7, 11 and 17 yr.	7, 12,13 and 17 yr.	15 and 24 mo.		x	2 yr.	2 yr.	School entry (4-5 yrs)
Grade milestones for assessing coverage	x	Grade 6 and 9	Grade 1 and 5	x	x	x	Grade 4 and 9		Grade 7	Grade 4, 6 and 9	Grade 1, Grade 6 and 9	Grade 7 and 9
Exceptions to coverage measurement	x	Influenza, Hepatitis A	x	x	x	Influenza Rotavirus^11		x	childhood vaccines	Influenza,	Influenza	Influenza, childhood vaccines
Frequency of coverage measurement	bi-annually	annually for all age groups	annually	quarterly (age 2 and 7), annually for other ages	annually	annually	bi-annually, annually^12		annually	annually	annually	annually

✓ denotes yes; x denotes no; n/a denotes not applicable; HI denotes health insurance; SL denotes school registration lists

¹ British Columbia assesses up-to-date coverage for all recommended vaccines for the 2 and 7 year old milestones, and assessment of vaccines scheduled for administration in Grade 6 and 9.

² Quebec assesses on-time coverage using a coverage survey for 15 and 24 month olds.

³ British Columbia has used IDEA score periodically to examine timeliness of immunization among 2 year olds.¹¹

⁴ Ontario has historically used a “complete-for-age’ definition. The exception is the method used for measles-mumps-rubella vaccine, which identifies the proportion of students who have received one or two doses of MMR vaccine by antigen.

⁵ Quebec assesses coverage for 15 and 24 months olds using a coverage survey, reporting two different estimates: one based on valid doses only and the other based on all doses. Coverage calculated from the provincial registry (i.e., measles) considers only valid doses.

⁶ Saskatchewan and New Brunswick apply a four day grace period to valid dose assessment. In Manitoba, a seven day grace period may be applied in certain scenarios. In Alberta, inactivated vaccines given 4 days before the minimum age or 4 days before the minimum interval has elapsed require consultation with the local Medical Officer of Health to determine if the dose is accepted and/or for further scheduling advice.

⁷ Denominator source in British Columbia varies across the health authorities where some regions rely on registry data (as derived from public health records when a person is born in the province or presents themselves to public health after moving from another province) or Ministry of Education enrollment estimates, school class lists or combination of both data sources including information collected on consent forms.

⁸ The denominator source in Saskatchewan is based on the numbers derived from public health records when a person is born in the province or presents themselves to public health after moving from another province).

⁹ Quebec does not have a population-based immunization registry because only those who received at least one dose of vaccine are included in the provincial data. The denominator data source for coverage assessed among Grade 4 and 9 students is the number of students attending school each year. The exception is a measles vaccine registry which includes the Quebec population born on and after 1970, based on the following information sources: health insurance database, Ministry of Education student database. In addition, a coverage survey is used to assess toddler immunization coverage (15 and 24 months old). The sampling frame for the coverage survey is the provincial health insurance database.

¹⁰ PEI uses the discharge abstract data or perinatal database to identify infants born in PEI (all babies are born in the hospital with very few exceptions) and uses data from the Department of Education for school age indicators.

¹¹ The first birth cohort eligible for publicly-funded rotavirus vaccine has not yet reached the aged of routine coverage assessment in Ontario (age 7 years). When this occurs in the 2019-20 school year, rotavirus coverage will be assessed through routine processes. Other processes (i.e. doses distributed) are being used in the interim.

¹² Quebec assesses coverage on a bi-annual basis for the 15 and 24 month olds. Coverage is assessed in grade 4 and 9 for Hepatitis B and HPV on an annual basis.

Most P/Ts measure up-to-date (UTD) coverage (Appendix), with some exceptions (Table 1). For example, ON has historically used a complete-for-age definition due to limitations of the previous IIS, which was replaced over 2013 to 2016 with a new system (Panorama). Complete-for-age coverage represents the proportion of students who are not yet 'overdue' for a particular immunization based on decision rules developed by the Ministry of Health and Long-Term Care.⁷ ON plans to assess up-to-date coverage going forward. In NB, immunization records of children enrolled in day care and entering school are reviewed by Public Health Nurses to determine if they are in compliance with all vaccine-preventable diseases cited in the Public Health Act,⁸ according to their routine schedule and age. Overall compliance (i.e. the percentage of children meeting the immunization requirements for school entry or daycare attendance) is reported and is not reported by antigen/vaccine or age. However, UTD coverage is assessed for certain school-based immunization programs. Finally, AB assesses coverage using a time-to-immunization (survival analysis) method to calculate the probability that a child received the vaccine of interest by age 2. For a particular birth cohort, individuals are followed until the earliest of: leaving AB, death, or December 31^st of the year when the cohort turns 2 y of age. This method removes potential biases associated with migration or duplicate records.⁹ UTD coverage assessment is used for school-age milestones.

With regard to the period of assessment for coverage among school-aged children, 2 jurisdictions (NL and SK) count the doses received by the child's birthday, while 4 (AB, MB, NS, PEI) count the doses received by the end of the academic year. BC counts the doses received by the child's 7^th birthday for doses received by the end of June corresponding to the academic year for that birth cohort, and for grade 6 and 9 cohorts. Four of the participating P/Ts indicated they apply a grace period to valid doses assessment (Appendix) when they measure UTD coverage; PEI does not invalidate doses.

Four P/Ts assess on-time coverage (Appendix; Table 1) and these P/Ts assess various age milestones (NWT: 2 y olds, BC: 2, 3, 6, 12 and 18 months, SK: 2 and 7 y olds, QC: 15–24 months). On-time coverage is assessed across most vaccine programs in these P/Ts. The on-time coverage definition in BC and QC is based on the number of valid doses and is calculated based on a one-month period from the recommended date of vaccine administration. NWT reports a “fully vaccinated” on-time coverage measurement that is based on series completion for 6 vaccine agents, comprising 12 antigens.¹⁰ SK assesses on-time coverage using a one-month period for young age groups (e.g., 2, 4 months) and 2 months for older age groups (e.g., 6 and 18 months). BC has also used the Immunization Delivery Effectiveness Assessment (IDEA) score to assess the timeliness of administration of each vaccination with reference to recommended age intervals. The score is a calculated value that incorporates the degree of delay in vaccine administration.¹¹ Both on-time coverage and the IDEA score are periodically measured in BC.

Coverage assessment in populations of special interest

Most P/Ts (11/12) include children born outside their jurisdiction in coverage assessments provided the child registers for healthcare services. However, previous immunizations may be under-reported because parent-held records, which may be unavailable or incomplete, must be manually entered into the systems of the new P/T. Nine P/Ts reported including home-schooled children and those attending private schools in routine coverage assessments. No P/T reports specific coverage estimates for these groups. No P/T described routine coverage assessment for vaccine programs specific to children with high-risk medical conditions.

Comparison of coverage estimates by different data sources

Figure 1 compares the coverage estimates for measles reported by P/Ts with those derived from cNICS for 2, 7 and 17 year-olds. The same birth cohort was used within each comparison. Seven P/Ts provided coverage data that could be directly compared with cNICS measles coverage at 2 y of age. In 4 jurisdictions, the provincial estimate was comparable to the 95% Confidence Interval (CI) around the relevant cNICS estimate. In 2 additional jurisdictions, the P/T estimate was within 1% of the upper confidence level (UCL) of the CI. Seven jurisdictions provided data for comparison at age 7. MB, and ON provided estimates comparable to cNICS-derived estimates; AB estimate was 0.4% higher than the UCL of the CNICS estimate. The remaining provinces had estimates that were higher than the UCL from cNICS (range: 3.0–8.1%). For measles coverage at 17 y of age, 4 jurisdictions provided data that could be directly compared and in all cases estimates differed from the cNICS upper and lower confidence limits for the respective provincial coverage estimates. With the exception of Manitoba, P/T-derived estimates were higher than the UCL from cNICS (range: 4.9–7.5%).

Figure 1. Comparison of immunization coverage estimates reported by select Canadian Provinces and as estimated by the 2013 childhood National Immunization Coverage Survey (cNICS): Measles

Notes:

1. Two-year old immunization coverage data for the Regional Health Authority of Vancouver Coastal Health are not included within the provincial estimate because coverage for this region is assessed periodically using survey methods.

2. Coverage estimates for Saskatchewan are based on children who were registered in the Saskatchewan Immunization management System (SIMS) and with provincial health coverage at the time of assessment.

3. Manitoba coverage estimates can be found online: http://www.gov.mb.ca/health/publichealth/surveillance/mims/docs/2012.pdf.

4. The reference birth cohort used by Quebec is children born between Oct 1, 2011 and Dec 31, 2011 and evaluated at 24 months of age in 2014.

5. The birth year cohort for the assessment of 2-year-old measles coverage in PEI are those born within the province during the 2011/12 fiscal year.

6. Immunization coverage for measles at 7 years-of-age in British Columbia is derived from Panorama and the immunization registry in use in Vancouver Coastal Health (PARIS). In three regional health authorities, Ministry of Education data is used for denominator estimates.

7. The exception to Ontario's 'complete for age' coverage methodology is the ability to report on coverage for measles, mumps and rubella by number of doses. Two doses measles coverage estimates are presented for the birth cohorts of 1995 and 2005 who were 7 and 17 years of age in the 2012-13 school year, respectively.

8. The coverage definition used by PEI is the receipt of 2 doses of measles-containing vaccine by the 6^th birthday.

Figure 2 compares the coverage estimates for pertussis reported by P/Ts with those derived from cNICS for 2, 7 and 17 year-olds. Seven P/Ts provided coverage data that could be directly compared with cNICS pertussis coverage at 2 y of age. In 4 jurisdictions, the provincial estimate was comparable to the 95% CI around the relevant cNICS estimate. NL, PEI and QC's estimates were higher than the UCL from cNICS (range 7.9–13.4%). Six jurisdictions provided data for comparison for pertussis coverage at age 7. Three provinces provided estimates comparable to cNICS-derived estimates and 3 provinces had estimates that were higher than the UCL from cNICS (range: 9.0–13.5%). For pertussis coverage at age 17, 4 jurisdictions provided data suitable for comparison. The estimate from MB was comparable to cNICS. For the remaining jurisdictions, provincial coverage was higher than the UCL from cNICS ranging from a difference of 3.2% (SK) to 48.3% (NL).

Figure 2. Comparison of immunization coverage estimates reported by select Canadian Provinces and as estimated by the 2013 childhood National Immunization Coverage Survey (cNICS): Pertussis

Notes:

1. Two-year old immunization coverage data for the Regional Health Authority of Vancouver Coastal Health are not included within the provincial estimate because coverage for this region is assessed periodically using survey methods.

2. Manitoba coverage estimates can be found online: http://www.gov.mb.ca/health/publichealth/surveillance/mims/docs/2012.pdf

3. Quebec assessed coverage for children born between Oct 1 and Dec 31, 2011 using a coverage survey. The coverage definition used was receipt of at least 4 valid doses, with validity of doses assessed in relation to a minimum age at first dose and minimum intervals between doses.

4. The 2011-2012 fiscal year birth cohort was used in PEI and this only includes those born in PEI.

5. Immunization coverage for measles at 7 years-of-age in British Columbia is derived from Panorama and the immunization registry in use in Vancouver Coastal Health (PARIS). In three regional health authorities, Ministry of Education data is used for denominator estimates. Coverage is defined as 4th or 5th dose of diphtheria/acellular pertussis/tetanus and 3rd or 4th dose of polio after the fourth birthday and by the 7th birthday.

6. Ontario has historically calculated 'complete for age' coverage, which represents the proportion of students who are not yet 'overdue' for a particular immunization. For the cohort assessed, the pre-school booster was considered valid if given on or after the age of three years and nine months.

7. PEI assesses coverage by agent. Coverage among 7 year olds reflects administration of the Tdap-IPV pre-school booster administered between the ages of 4 and 6 years.

8. Ontario has historically calculated 'complete for age' coverage, which represents the proportion of students who are not yet 'overdue' for a particular immunization. For the cohort assessed, the pre-school booster was considered valid if given on or after the age of three years and nine months.

In general, concordance between P/T reported coverage and cNICS was highest for Western provinces and Ontario and lowest for Quebec and Eastern provinces, and also higher for younger than older age groups.

Discussion

We have demonstrated that substantial differences exist in the methods used to assess childhood immunization coverage across Canada's P/Ts. For example, at the time of data collection coverage at 2 y of age was assessed by surveys in QC and one large regional health authority in BC. Coverage was not routinely assessed in 3 P/Ts, and the remaining P/Ts used IISs or some type of aggregate reporting mechanism for this age group. There were also differences in the approach to coverage calculation with variability in the period of assessment and the use of grace periods. Minimum ages and intervals were applied to dose assessment when calculating coverage in all but one jurisdiction and are not incorporated into the coverage definitions of cNICS. The clinical significance (i.e., the magnitude of impact) of these methodologic differences on coverage estimates is not known. The concordance between coverage estimates between P/T sources and cNICS varied and when discrepancies were found, cNICS estimates were generally lower. Absolute differences between the sources tended to increase with a requirement for increasing number of doses and increasing age at assessment, as illustrated by comparisons conducted for pertussis coverage.

Immunization coverage standards in Canada have been developed by the Canadian Immunization Registry and Coverage Network and were first released in 2005¹² and revised in 2015⁵. The 2005 recommendations specified frequency and timing of assessment, age cohorts (coverage assessment at the milestone ages of 2, 7 and 17 y of age), data collection, and methodology.¹² The 2015 recommendations included expanded guidance on how to operationalize these standards including the recommended number of doses for UTD assessment and how to incorporate late starters.⁵ We did not aim to comprehensively assess the extent to which the 2015 standards are followed as these were made publicly available on the PHAC website after our data collection from key informants was completed.

This is the first study that has comprehensively compared P/Ts coverage estimates derived from cNICS with estimates independently generated by P/Ts, although the final 2013 cNICS report includes a similar comparison for measles coverage estimates at age of 2 y.¹³ Prior to 2013, cNICS reported only national estimates as the sample size was too small to report P/T estimates. Although cNICS uses a representative sampling frame, the overall participation rate in the survey was only 61%, which is comparable to other recent Statistics Canada surveys such as the Canadian Community Health Survey (CCHS). The CCHS response rate was 66% in 2013–2014.^14,15 The representativeness of the cNICS sample may be one contributor to the differing coverage estimates observed between cNICS and P/Ts. Previous Canadian research has found that respondents to other immunization surveys are more likely to be UTD, as compared with non-responders.¹⁶ However, if parents of under-immunized children are less likely to participate in the coverage survey, one would expect the cNICS estimates to be higher than those from the P/Ts, which was generally not the case.

Several other reasons may explain why the estimates obtained from cNICS are lower than the P/T estimates. The cNICS relies on availability of the child's immunization record and HCP validation could only be completed for one-third of respondents due to the requirement of parental/guardian signature, rather than verbal consent as is used in some other national coverage surveys.¹⁷ Parent-held records may be incomplete if immunizations were not recorded or captured correctly.¹³ This is particularly true for pertussis coverage at 17 y of age, as the 6 doses required to be considered as UTD includes a dose of Tdap often given at school, which may not be recorded in a parent-held record. Canadian qualitative research has explored the challenges expressed by parents in maintaining up-to-date immunization records.¹⁸ Importantly, differences in coverage estimates between the 2 sources will have been influenced by different methodologies and definitions used to assess coverage.

Strengths and limitations

This study interviewed a multidisciplinary group of key informants engaged in immunization information collection, reporting and coverage assessment at the federal, provincial and territorial level in Canada. All P/Ts, with the exception of the Yukon territory (YT), participated and publicly available documents indicate that YT has recently transitioned to Panorama, which has been described within this manuscript.

Although we asked for coverage estimates from all participating P/Ts, only 8 provided estimates to enable comparisons with cNICS. As we recognized that immunization programs are complicated and involve many parties, we allowed respondents to solicit feedback from their colleagues and we interviewed multiple individuals from the same jurisdiction, where possible. Follow-up with respondents occurred to validate the study data. Despite these mechanisms, respondents may have been unaware of all the immunization-related processes and activities that exist within their jurisdiction which could lead to inaccurate reporting.

A further limitation is that we did not include those working at a sub-provincial or regional level with regards to immunization coverage assessment activities. Individuals working at the regional level may or may not have the capacity for local analysis but may be more familiar with local data quality, as well as other issues and initiatives that influence coverage data. Inclusion of regional public health professionals engaged in local immunization surveillance activities would provide a valuable perspective in a future extension of this work. A final limitation to note is that Canadian P/Ts currently using the Panorama IIS were, over the period of information collection in 2015, in various stages of adoption and implementation of the system, and the information represented within the manuscript may not reflect current operational processes with Panorama in full use.

Conclusions

There is considerable variability between Canadian P/Ts in the methods used for coverage assessment. P/T coverage reports should provide sufficient detail about the methods of data collection and analysis to allow for inter-jurisdictional comparisons and should also acknowledge limitations. As work continues in Canada on the implementation of electronic immunization information systems, further refining national guidance on the methodology for immunization coverage assessment should continue to facilitate a more standardized approach across jurisdictions. The study's participants who are members of Canada's Immunization Registry and Coverage Network, a Federal-Provincial-Territorial working group reporting to the Canadian Immunization Committee, are uniquely positioned to lead this work.

Methods

Sampling and recruitment

We used a mixed methods approach to collect information from key informants who are subject matter experts regarding their jurisdiction's immunization coverage surveillance infrastructure. At least one key informant per federal, provincial or territorial organization was identified by contacting members (full and ex-officio) and the secretariat of the Canadian Immunization Registry and Coverage Network (CIRC). This network includes representation from all 13 P/Ts, Correctional Services Canada, Department of National Defense, and the First Nations and Inuit Health Branch (FNIHB) of Health Canada, with secretariat support provided by PHAC.¹⁹ We also invited BC's First Nations Health Authority to participate. We excluded representatives from the Department of National Defense and Correctional Services Canada because we were primarily interested in childhood and adolescent immunizations. All respondents provided informed consent. Ethics approval was granted by Public Health Ontario Research Ethics Board.

Data collection

Questionnaire

We developed a 56-item questionnaire to collect information about immunization delivery and the systems and methods used for assessing immunization coverage. A variety of resources were used to develop the questionnaire.^5,20,21 Findings related to immunization delivery and information systems are described elsewhere.⁴ We asked participants questions regarding the methods used for routine immunization coverage assessment including definitions for up-to-date and on-time coverage (Appendix). Due to the scope of the content covered in the questionnaire, key informants were asked to consult with colleagues to complete the questionnaire, at their discretion.

Interview

Following completion of the questionnaire, a semi-structured interview that included questions about perceived strengths and limitations of the systems and methods used for immunization coverage assessment was administered to each participant between April and August 2015. All interviews were conducted in English by one member of the research team (S.Q.), audio-recorded, and transcribed verbatim.

Coverage template

We developed a template to collect P/T coverage estimates from key informants. Estimates were collected by antigen and age (2, 7 and 17 y of age) specifying particular birth year cohorts to enable intra-jurisdictional comparisons with corresponding estimates derived from the 2013 cNICS. Coverage definitions, based on CIRC guidelines for coverage assessment, were included in the template.⁵ Completing the coverage template was optional and we did not request coverage estimates from FNIHB or British Columbia's First Nations Health Authority (FNHA).

2013 Childhood National Immunization Coverage Survey

cNICS is a national survey conducted approximately every 2 y to estimate national coverage for all routine childhood immunizations recommended by NACI and publicly-funded by P/Ts.¹³ The survey has been in place since 1994 with changes in methodology over time. In 2013, cNICS used Canadian Child Tax Benefit claimants, estimated to include 96% of Canadian children, as the sampling frame. Similar to previous iterations of cNICS and other federal surveys, children living on First Nations reserves were excluded. Households with a child aged 2, 7, or 17 y of age and girls between 12 and 14 y of age as of March 2013 were included in the survey; only one eligible child in a household was selected. The child's parent or guardian was asked to provide immunization data by telephone interview using the child's parent-held immunization record. Parental recall was accepted only for influenza vaccine in 2-year-old children and for adolescent vaccines provided primarily by school-based programs (HPV assessed at 12–14 y and 17 years, and Hepatitis B assessed at 17 years). Immunization data were validated with a review of immunization records from healthcare providers (HCPs) for approximately one-third of participants where both parental consent and HCP participation were obtained. A child is considered up-to-date for a specific antigen if the full number of required vaccine doses is received by the age milestone (e.g., 2 doses of measles-containing vaccine before the 7th birthday for MMR vaccine). Validation rules for dose assessment (e.g., minimum ages and minimum intervals) are not applied.

Data analysis

Frequencies and proportions were calculated for the questionnaire responses. Two researchers used content analysis to analyze the interview data using an approach described previously.⁴ Drafts of the data tables and results were reviewed by key informants to ensure the responses were accurate. The absolute difference in coverage estimates was calculated between estimates provided by P/Ts and the upper or lower bound of the 95% confidence interval (CI) of estimates derived from cNICS, with the P/T generated value used as the reference.

Abbreviations

AB	=	Alberta
ACIP	=	Advisory Committee on Immunization Practices, Centers for Disease Control and Prevention, Atlanta, Georgia
BC	=	British Columbia
CCHS	=	Canadian Community Health Survey
Cnics	=	Childhood National Immunization Coverage Survey
CI	=	confidence interval
CIRC	=	Canadian Immunization Registry and Coverage Network
CIRN	=	Canadian Immunization Research Network
EMR	=	Electronic Medical Record
FNHA	=	First Nations Health Authority
FNIHB	=	First Nations Inuit Health Branch
HCP	=	healthcare provider
I-CLSC	=	Center Locaux de Services Communautaires
IDEA	=	Immunization Delivery Effectiveness Assessment
IIS	=	immunization information system
IMIT	=	information management and information technology
MB	=	Manitoba
NACI	=	National Advisory Committee on Immunization
NB	=	New Brunswick
NL	=	Newfoundland and Labrador
NS	=	Nova Scotia
NU	=	Nunavut
NWT	=	Northwest Territories
ON	=	Ontario
PEI	=	Prince Edward Island
PHAC	=	Public Health Agency of Canada
P/Ts	=	provinces and territories
QC	=	Quebec
SK	=	Saskatchewan
UCL	=	upper confidence limit
UTD	=	up-to-date
VCHA	=	Vancouver Coastal Health Authority

Disclosure of potential conflicts of interest

SMM has received research grants from GSK, Sanofi, Merck and Pfizer for unrelated vaccine research.

Acknowledgments

Thank you to Jason Cabaj for his assistance in the early stages of development of the data collection tools used in this study and to Ariane Renaud for her assistance with manuscript preparation. We are also grateful to the study participants for their participation and important contributions to this study.

Funding

This study was supported by an operating grant from the Public Health Agency of Canada and the Canadian Institutes of Health Research through the Canadian Immunization Research Network.

Appendix

Table

Term	Definitions^5	Example
Up-to-date coverage	Up-to-date immunization coverage refers to assessing coverage in a specific age-cohort in relation to how many doses of vaccine have been received by a particular milestone age. The number of doses is informed by NACI recommendations and the relevant P/T immunization schedule.	≥ 4 doses of pertussis-containing vaccine by the 2^nd birthday (numerator definition)
On-time coverage	On-time immunization coverage refers to the proportion of children in a particular age cohort who have received all valid doses required to be on-time and up-to-date by specific milestone ages (typically by the first or second birthday). The assessment is made examining doses for multiple vaccine series, as per the routine schedule. Thus, even if one dose of a particular antigen is late, the child will not be considered on-time. In practice, a leeway period is often used for calculating on-time coverage (i.e., a 1 month leeway period after the recommended date for vaccine administration).	All doses for all recommended vaccines (as per the P/T schedule) administered within 1 month of recommended interval by the 2^nd birthday (numerator definition)
Minimum interval	The minimum interval of time recommended between 2 vaccine doses to allow for the body to mount an appropriate immune response before the subsequent dose is received. Minimum intervals are typically shorter than the recommended spacing of vaccines outlined within routine immunization schedules. Minimum intervals are informed by NACI or other (e.g., ACIP) expert body recommendations.	1 month (defined as 28 days) between doses of live virus vaccines
Valid/invalid dose	A valid dose is one that is delivered in accordance with decision support rules outlining the minimum age recommended for vaccine administration and/or the minimum time interval between 2 doses of vaccine. Doses that are administered too early (before the minimum age and/or before the minimum interval has elapsed) are considered invalid and are not typically ‘counted’ in the dose assessment used for coverage assessment.	A child who receives MMR vaccine at 12 months of age and again 14 d later would be regarded to have received only 1 valid dose of MMR. The second dose was administered before the minimum interval and is considered invalid.
Grace period	When used, grace periods provide an allowable period before dose eligibility such that an administered dose can still be considered valid.	If a 4 day grace period is used, doses of MMR administered up to 4 d before a client turns 1 y of age, would be considered valid doses for the purposes of coverage assessment (assuming 12 months of age is used as the minimum age)

Note: To generate a coverage estimate, the numerator (of children with the requisite number of valid doses) is divided by the eligible population (denominator). For coverage assessment, the numerator is divided by the denominator, which CIRC defines as all children within a defined birth cohort who are current residents in the P/T during the time period of interest (not only those who present for immunization).⁵