Abstract
Background: A new measurement process based upon a well-defined mathematical model was applied to evaluate the quality of human papillomavirus (HPV) vaccination centers in 3 of 12 Local Health Units (ASLs) within the Lazio Region of Italy. The quality aspects considered for evaluation were communicational efficiency, organizational efficiency and comfort.
Results: The overall maximum achievable value was 86.10%, while the HPV vaccination quality scores for ASL1, ASL2 and ASL3 were 73.07%, 71.08%, and 67.21%, respectively.
Conclusions: With this new approach it is possible to represent the probabilistic reasoning of a stakeholder who evaluates the quality of a healthcare provider. All ASLs had margins for improvements and optimal quality results can be assessed in terms of better performance conditions, confirming the relationship between the resulting quality scores and HPV vaccination coverage.
Methods: The measurement process was structured into three steps and involved four stakeholder categories: doctors, nurses, parents and vaccinated women. In Step 1, questionnaires were administered to collect different stakeholders’ points of view (i.e., subjective data) that were elaborated to obtain the best and worst performance conditions when delivering a healthcare service. Step 2 of the process involved the gathering of performance data during the service delivery (i.e., objective data collection). Step 3 of the process involved the elaboration of all data: subjective data from step 1 are used to define a “standard” to test objective data from step 2. This entire process led to the creation of a set of scorecards. Benchmarking is presented as a result of the probabilistic meaning of the evaluated scores.
Introduction
Cervical cancer is the third most common cancer in women worldwide and the seventh overall, with an estimated 500 000 new cases and 275 000 deaths in 2008.Citation1 In Italy, approximately 2800 new cases are diagnosed, and over 1000 women are estimated to die every year.Citation2,Citation3
The World Health Organization recognizes cervical cancer as the first type of carcinoma completely due to an infection: virtually all cervical cancer cases (99%) are linked to a genital infection with Human Papillomavirus (HPV).Citation4 The persistent infection with a carcinogenic type of HPV represents the necessary condition for the evolution to cervical cancer.Citation5
In Italy, beginning in March 2008, a campaign of active and free offer of both bivalent and quadrivalent vaccines against carcinogenic HPV types was launched. The campaign was aimed toward girls 11 to 12 y of age, starting with those born in 1997.Citation6,Citation7 The goal of the HPV immunization program was to achieve coverage greater than 95% with three doses of the vaccine within the year 2011. Despite numerous campaigns and recommendations to the population by institutional agencies,Citation8 as of December 31, 2011, the HPV vaccination coverage with three doses of the vaccine in the cohort of girls born in 1997 was only 65%.Citation9
To achieve the health goals for cervical cancer reduction, the quality of services requires improvement. It is necessary to reinforce all actions known to improve vaccination coverage, such as better communication to the target population and efficient offer of the vaccine,Citation10 as required by the Italian National Vaccine Prevention Plan 2012–2014.Citation11
Quality of vaccination services is part of the broader context of the quality of health services and is currently considered an essential feature by patients-users, staff working in this area and institutions.
Many methods for analyzing and improving quality have been used in the health care arena, including models that utilize users perspectives, such as the “SERVQUAL” methodCitation12 and “Discrete Choice Modelling.”Citation13,Citation14 However, the assessment of the quality of care should not exclude the direct and active involvement of both those who participate in the delivery of health services and those who directly use these services.
Within a health care organization, the desire to overcome the partiality of the doctor’s “absolutistic” point of view or the patient’s “individualistic” one stimulated the search for a new methodology to assess service quality and to deliver consistent results. This is useful for the comparison of different providers.
Since 2005, starting with the quality measurement needs of a hospital,Citation15 the “Measures of Quality in Health Care Services” research project (i.e., a multidisciplinary research team composed of medical doctors, engineers, and mathematicians) created a model of quality events related to service delivery that is based on the theory of probability according to Bruno de Finetti.Citation16 From this theory, a consistent and rigorous quality measurement process was further developed and applied to different settings.Citation15,Citation17-Citation19 Beginning in 2010, the focus of the research shifted from the hospital to local health services, particularly vaccination centers.Citation20 Generally, when evaluating vaccination activities a key performance indicator (KPI) approach is used, which is based on the assessment of facilities, personnel and all available resources.Citation21 The administration of vaccines, the patient invitation and registration, the achievement of vaccination coverage within the established time limits, the notification for vaccine-related diseases and the surveillance of adverse reactions have been used as main quality indicators.
The “Quality in Vaccination: Theory and Research” (QuaVaTAR) group, composed of medical doctors, engineers, epidemiologists and mathematicians, was designed to implement a new approach to the quality measurement and benchmarking of HPV vaccination services. This approach is based on a well-defined mathematical modelCitation15 and was tested for its applicability for the evaluation of the vaccination services in the Lazio Region of Italy.
In this pilot study, three of 12 Local Health Units (ASLs) within the Lazio Region of Italy were involved based on their willingness to participate: ASL1, located in the center of Rome; ASL2, located on the edge of the city; ASL3, located outside of the city. HPV vaccination services were evaluated on the basis of their communicational efficiency (CE), organizational efficiency (OE) and comfort (Co).
Results
Step 1 of the measurement process (subjective measurement)
During the first phase of the study, i.e., step 1 of the measurement process, from April to July 2010, 264 questionnaires investigating the probability (i.e., opinion) that a stakeholder (a vaccinated woman, a parent, a doctor or a nurse) assert communicatively efficient (event E1), organizationally efficient (event E2) and comfortable (event E3) the HPV vaccination services, assuming different performance conditions (in the following indicated by H), were obtained. The socio-demographic characteristics of the different stakeholder groups showed that 120 of 170 parents had a high school or university education level, and the mean age was 42.7 y old (±5). Among vaccinated women, 23 out of 63 had a high education level, and the mean age was 16.5 y old (+/− 4.2). The weighted sum of the opinions of all the stakeholder categories allows the obtainment of the “virtual” Expert point of view.
show the output of step 1 of the measurement process. For example, for communicational efficiency (CE), which is represented as the means and time of communicating information related to vaccination, the maximum probability of event E1, from an expert’s point of view, was 87.05%, assuming true the occurrence of performance condition H1 i.e., information is provided through a brochure with the assistance of a person in the waiting room. Additionally, the minimum probability of event E1 was 7.16%, assuming true the occurrence of performance condition H8 i.e., information is not provided (). For organizational efficiency (OE), which is represented by the time spent by a user in a waiting room and the time spent by a user for vaccination, from an expert’s point of view the maximum value was 82.91% and the minimum value was 22.15% (). For Comfort (Co), represented by the opening times and the availability of magazines in the waiting room, we obtained 88.52% and 52.99% ().
Table 1A. Stakeholders opinions related to the communicational efficiency of the HPV vaccination service delivery
Table 1B. Stakeholders opinions related to the organizational efficiency of the HPV vaccination service delivery
Table 1C. Stakeholders opinions related to the comfort of the HPV vaccination service delivery
Step 2 of the measurement process (objective measurement)
In step 2 of the measurement process, 115 observations related to HPV vaccine delivery in ASL1, ASL2 and ASL3 were collected. CE, OE and Co performance conditions considered in step 1 were investigated. Using the CE aspect as an example, shows that the communicative performances (from H1 to H8) differed among the ASLs. ASL1 and ASL2 always (100%) provided information orally by a person during the vaccination phase (H7), while ASL3 provided the information in various ways: 38.50% as defined in event proposition H3 (information is provided through a brochure), 23.00% as defined in H6 (information is provided orally by a person in the waiting room) and 38.50% as defined in H7. and show the results for OE and Co in the same way as CE in .
Table 2A. Performance related to the communicative efficiency of the HPV vaccination service delivery
Table 2B. Performance related to the organizational efficiency of the HPV vaccination service delivery
Table 2C. Performance related to the comfort of the HPV vaccine service delivery
Step 3 of the measurement process (final results)
In step 3 of the measurement process, the probabilities of the main quality events under evaluation were obtained using the algorithm described in the mathematical appendix in Supplemental Material. shows the probabilities of E1, E2, and E3, that is, the quality scores regarding the HPV vaccination services, in comparison with the resulting minimum and maximum values, for each aspect, i.e. CE, OE, and Co, in each ASL under evaluation. The overall HPV vaccination quality scores were 73.07%, 71.08%, and 67.21% for ASL1, ASL2, and ASL3, respectively, and the overall minimum and maximum reference values were 20.82% and 86.10%.
Table 3. Main events under evaluation and their probabilities of being true from an expert’s point of view
Discussion
This pilot study was performed to test the applicability of the measurement process of quality for HPV vaccination centers by a multidisciplinary team: the QuaVaTAR group.
Step 1 of the process was performed to collect “subjective data,” i.e., opinions, by means of a survey (questionnaire). The results of the survey were used to measure how stakeholders perceived quality of the service of a “standard provider” (i.e., a provider compliant with the best practice/recommendations or guidelines). The adjective “subjective” should not be interpreted as meaning “without objectivity.” Rather, this term refers to the fact that a subject (i.e., a stakeholder) can express its opinion (i.e., a probability) about a fact (i.e., an event) using all available data and experience (i.e., using maximum objectiveness). Different performance conditions defined by relevant KPIs were investigated to elicit the best and worst service delivery conditions for each stakeholder’s point of view. To obtain an expert point of view, the opinions expressed by each stakeholder were combined by means of numerical weights that represent the relevance recognized in a function of the stakeholder’s role in the quality evaluation of the service delivery. For example focusing on HPV vaccination CE, from the expert point of view the best way to perform communication is to provide information through a brochure with the aid of a person in the waiting room (). This was the so-called “HPV vaccination CE target” for all the ASLs. In contrast, and as expected, the worst way to perform communication is obtained when information is not provided at all.
In this pilot study, a remarkable consensus was obtained among the respondents for the KPIs identified for CE: in fact, the difference between the achievable maximum and minimum values ranged from 87.50% to 7.16%. The choice of indicators for OE were also appropriate. Indicators for Co, instead, most likely did not represent elements of the service delivery considered particularly important for respondents, and the difference between the achievable maximum and minimum values was small (88.52% and 52.99%).
The fact that the best and worst service delivery conditions do not correspond to the theoretical maximum and minimum probability values (100% and 0%, respectively) is not a limitation of the model. Other performance conditions may be better (e.g., a higher value for HPV vaccination CE could be obtained if information is implemented by means of emails in response to customers’ questions) or worse (e.g., the value 0% could be related to the communication of incorrect information) than those investigated. However, not all performance conditions that could be effectively observed by the working group or implemented by the ASLs (assuming budgeting or organizational limitations) were investigated.
Differences in the perception of the stakeholders of the service quality are accepted. For example, analyzing our results in detail if information was not provided, doctors perceived this CE condition as worse in respect to nurses, parents, and vaccinated women ().
It is important to note that there are no prescriptions for the minimum number of questionnaires. In fact, according to the subjective probability theory, any available information can be used to assess (prior) probabilities of events of interest. Thus, even if only a small number of questionnaires are available, they are useful in providing an initial evaluation of the probabilities of events of interest. In the event that further information becomes available, prior evaluations can be updated to obtain new (posterior) probabilities.
Step 2 of the process was performed to collect “objective data,” i.e., observations on those KPIs used in step 1 that describe the performances of the ASLs under evaluation.
Some KPIs can be “constant,” such as those related to the structure of the organization, and only one observation is needed to assess their values until some organizational change occurs. Other KPIs can be “variable” and more observations per service delivery are needed, even if there are no prescriptions for the minimum number as described for questionnaires. For example, regarding HPV vaccination CE (), all information is provided orally by a person during the vaccination phase (100% of the HPV vaccination delivery) in ASL1 and ASL2, as determined by the local organization. In ASL3, however, three conditions with differing occurrence were observed; this variability was due to the absence of a specific commitment in the means of communication.
Step 3 of the process was performed to compute both the HPV vaccination service quality scores (i.e., the probabilities of E1, E2, and E3) and the “overall” quality scores of the ASLs (). At this point, it was necessary for the working group to quantify the relevance of the quality aspects under investigation (i.e., CE, OE, and Co) by means of some numerical weights. In this study, the working group decided to assign a higher relevance to CE, taking into account that the HPV vaccination targets a young-adult population, for which it is established that accurate information is one of the most effective ways to increase vaccination coverage.Citation22,Citation23
clearly shows that all ASLs perform differently: for example, ASL1 and ASL2 have the same HPV vaccination CE score (71.57%), while the HPV vaccination OE score is higher for ASL2 (69.12%) than for ASL1 (67.79%) and the HPV vaccination Co score is higher for ASL1 (88.52%, equal to the maximum value) than it is for ASL2 (75.23%). In relation to the maximum overall quality score (86.10%), all ASLs have margins for improvements. In regard to CE (), ASL1 and ASL2 both performed 100% of the HPV vaccination delivery by providing oral information from a person during the vaccination phase. In the opinion of the Expert, the value obtained (71.57%) for this condition () is not the maximum achievable. Therefore, ASL1 and ASL2 could improve the HPV vaccination quality by providing information through a brochure with the aid of a person in the waiting room. More improvements can be achieved by taking into account similar considerations for OE and Co. It is important to emphasize that by analyzing the results of this pilot study, a correlation between quality scores and levels of vaccination coverage of the different ASLs has been detected. In fact, in ASL1 and 2, where the highest quality scores were obtained, the levels of immunization coverage for three doses of the HPV vaccine are higher than those observed in ASL3.Citation24 Further work could identify a target value accepted by the Health Authorities. The strength of this approach lies in the possibility to define a common set of quality events (e.g., the fact that a patient or a doctor asserts efficient the HPV service delivered by a vaccination center) for a set of providers of the same type (e.g., vaccination centers). Thus, even if the performance of a service delivery can be modeled by different KPIs due to many different factors (e.g., clinical governance, budget limits, etc.), quality evaluation of a single provider and benchmarking among providers can be performed through the probability assessment of such aforementioned events. In summary, our approach explicitly represents the probabilistic reasoning of a stakeholder (e.g., patient, doctor, etc.) who evaluates the quality of a healthcare provider. This approach can be used to predict the quality of the delivery of a service, and it is possible to build different models, assuming different stakeholders, weights and indicators.
The weaknesses of such an approach is in the startup effort and complexity necessary to define the resulting model, as it requires a bottom up analysis in order to represent the service delivery process and the evaluation from different stakeholders’ points of view. The subsequent implementation phases, i.e., the observation of real providers during the delivery of services and the resulting quality assessments, are only expensive in terms of time and costs proportional to the duration of the evaluation period, but not complex to perform.
Materials and Methods
A three step process was defined and used. It is mainly based on the algorithm described in the mathematical appendix in Supplemental Material, and more extensively in a recent PhD Thesis.Citation25
In our algorithm, the problem of evaluating the quality of a health care service was handled in a Bayesian inferential setting using a general frameworkCitation17 by which it is possible to handle different kinds of information (i.e., managing uncertainty and vagueness together) related to the events of interest. In this context, it is not necessary to perform any sensitivity analysis, statistical tests or confidence interval evaluation.
In this particular application we were interested in the probability evaluation of three main events: E1, E2, E3, respectively defined by the propositions “You assert communicatively efficient the HPV vaccination service,” “You assert organizationally efficient the HPV vaccination service” and “You assert comfortable the HPV vaccination service.” The term “You” in each proposition is interpreted as an HPV vaccination service stakeholder (real as a patient, a doctor, a nurse, or a parent, or virtual as an “expert,” i.e., a combination of the real stakeholders). Thus, in our model, what we refer to as “scores” are precise probabilities of well-defined events related to the quality of a service delivery. It is therefore not necessary to assess confidence intervals or apply any statistical tests because the model is deliberately sensitive.
Step 1 of the measurement process (subjective measurement)
Between April and July of 2010, three ASLs in the Lazio region of Italy were selected, based on their willingness to participate, to collect stakeholder’s opinions, i.e., “subjective data,” on quality aspects of HPV vaccine delivery. The working group decided to represent communicational efficiency (CE) as the means and time of communicating information related to vaccination (with oral or written support before or during vaccination, or not provided at all), organizational efficiency (OE) by the time spent by a user in a waiting room and the time spent by a user for vaccination. Comfort (Co) was represented by the opening times and the availability of magazines in the waiting room. An “ad hoc” questionnaire was administered to selected stakeholders: parents of HPV immunized subjects, adolescent, and young women (12–25 y old)Citation26 vaccinated against HPV, medical doctors and nurses working in the ASLs. There were three sections in the questionnaire: (1) an anonymous demographic section with questions on age, sex, education, marital status and occupation; (2) an informative section on the functioning of a generic vaccination service as presented by institutional guidelines; (3) a judgment section containing 8 questions on CE, 16 questions on OE, and 8 questions on Co. Stakeholders were asked to express opinions in terms of probabilities, i.e., values ranging from 0% to 100%, of the events E1, E2, and E3 related to the HPV vaccine delivery assuming different performance conditions (H). The aforementioned questions were prepared based on the general model described by Coletti et al.Citation17 and were related to three main conditional events defined as follows:
E1|H: i.e., you assert communicatively efficient the HPV vaccination service, assuming the performance condition H;
E2|H: i.e., you assert organizationally efficient the HPV vaccination service, assuming the performance condition H;
E3|H: i.e., you assert comfortable the HPV vaccination service, assuming the performance condition H.
The term “you” in each proposition can be interpreted as an HPV vaccination service stakeholder and H can be defined by ranges or values of selected KPIs which describe the service delivery.
For example, one question was “At what level between 0 and 100% do you assert communicative efficient the HPV vaccination service, assuming that an operator of the staff gives information on the vaccination by means of a leaflet in the waiting room?” Another question was “At what level between 0 and 100% do you assert organizational efficient the HPV vaccination service, assuming that the waiting time is 15 minutes and the vaccination time is 10 minutes?” A single answer from a stakeholder, i.e., an opinion, can be used to assess directly the conditional probability of the investigated conditional event. If, as in our case, there are more answers related to the same questions, these can be integrated for the assessment. In this application the working group proposed to assign a specific relevance (weight) to the opinions of each stakeholder category, with the chosen weights 0.25, 0.20, 0.30, and 0.25 for parents, vaccinated women, doctors and nurses, respectively. Summing the stakeholders’ opinions and taking into account the aforementioned weights, the group derived the opinion of the so-called Expert, i.e., a virtual HPV vaccine service stakeholder.
Step 2 of the measurement process (objective measurement)
During the same period of the survey (April–July 2010), KPIs related to the performance conditions that were investigated in the judgment section of the questionnaire were observed in the ASLs during the service delivery and recorded in a paper form by the operators of the QuaVaTAR group. These data, i.e., “objective data,” included waiting times, length of vaccinations, means of communication and comfort features.
Step 3 of the measurement process (activities and criteria)
At the end of the evaluation period, objective and subjective data were entered into a Microsoft Excel spreadsheet that implemented the algorithm described in the mathematical appendix in Supplemental Material. The QuaVaTAR group assigned relevance to the quality aspects under investigation by means of some numerical weights: 0.5, 0.3, and 0.2 for CE, OE, and Co, respectively. With all of these available data, the algorithm was allowed to compute the HPV vaccination service quality scores and finally the overall quality scores of each of the ASLs.
Conclusions
A new approach to quality measurement and benchmarking of health care services was investigated. This approach was based on a well-defined mathematical algorithm composed of three main steps and based on a well-defined probabilistic inference framework. Step 1 involved the elicitation of the best and worst performance conditions related to service delivery by means of a survey that considered different service stakeholders’ points of view (i.e., subjective data collection). Step 2 involved the observation and collection of data regarding the performance during the service delivery by some providers under investigation (i.e., objective data collection). Step 3 involved the elaboration of all gathered data using subjective data from step 1 to define a standard to test the objective data from step 2.
To realize the QuaVaTAR pilot study, this new approach was applied to measure the quality of HPV vaccination services in the Lazio Region of Italy by evaluating three related quality aspects and four stakeholders’ perspectives. It was possible to represent the probabilistic reasoning of stakeholders who evaluate the quality of a HPV vaccination service. This approach could also be used to predict the quality of service delivery from an integrated (i.e., Expert) point of view. All ASLs have margins for improvements and optimal quality results can be assessed in terms of better performance conditions, confirming the relationship between the resulting quality scores and HPV vaccination coverage.
The QuaVaTAR pilot study was recognized as exemplar research by the Lazio Region Government with an explicit citation in the Regional Vaccination Plan for the period 2012–2014Citation27and future work is planned to extend the study to all 12 ASLs of the Lazio Region in collaboration with Regional Health Authorities.
| Abbreviations: | ||
| ASL | = | local health unit |
| CE | = | communicational efficiency |
| Co | = | comfort |
| E1 | = | …, En, quality events related to service delivery |
| H1 | = | …, Hn, performance conditions related to service delivery |
| HPV | = | human papillomavirus |
| KPI | = | key performance indicator |
| OE | = | organizational efficiency |
| QuaVaTAR, Quality in Vaccination | = | Theory and Research |
Additional material
Download Zip (676.3 KB)Disclosure of Potential Conflicts of Interest
The authors declare no financial and personal relationships with other people or organizations that could inappropriately influence their work.
Acknowledgments
The authors are grateful to the following people and organizations:
A.Fa.R.–Fatebenefratelli Association for Biomedical Research for the granting of the study and for project management.
University of Rome “Tor Vergata,” Department of Biomedicine and Prevention—Specialization School for Hygiene and Preventive Medicine, for the allocation of human resources, the enrollment of three vaccination centers of the Lazio region of Italy (ASL-RMB, ASL-RMF, ASL-RMH) and for public relations activity with the Agency for Public Health, Lazio Region, Rome, Italy (ASP-Laziosanità).
University of Rome “Sapienza,” Department of Basic and Applied Sciences in Engineering (SBAI), School of Doctorate in Models and Methods for Technology and Society, for the mathematical modeling.
Dr Piero Borgia, former Scientific Director of Agency for Public Health (ASP-Laziosanità), Lazio Region, Rome (Italy) for suggestions and helpful discussions.
Troy Nachtigall, for stylistic suggestions in revising the English version.
The QuaVaTAR Group
Bagnato B, Bartolaccini T, Benigni M, Borgia P, D’Anna C, Di Marzio L, Fraioli A, Laudati F, Mangia ML, Marchetti C, Papa R, Pozzato S, Rabbiosi S, Rossi S, Seminara L, Serino L, Sinopoli MT, and Sorbara D.
Authorship
Maurici M, Paulon L, Franco E, and Campolongo A, conceived and designed the study.
Maurici M, Paulon L, Franco E, Campolongo A, Meleleo C, Perrelli F, Sgricia S, and Ferrante M, discussed and approved the design of the study.
Maurici M, Paulon L, Franco E, Campolongo A, Meleleo C, and Sgricia S, contributed to the acquisition of data.
Maurici M, Paulon L, Franco E, Campolongo A, Meleleo C, Carlino C, Giordani A, Perrelli F, Sgricia S, and Ferrante M, contributed to the analysis and interpretation of data.
Maurici M, Paulon L, Franco E, Meleleo C, Campolongo A, Carlino C, and Giordani A drafted the article.
Maurici M, Paulon L, and Franco E, revised the article critically for important intellectual content.
Maurici M, Paulon L, Franco E, Campolongo A, Meleleo C, Carlino C, Giordani A, Perrelli F, Sgricia S, and Ferrante M, gave final approval of the version to be submitted.
Supplemental Materials
Supplemental materials may be found here: www.landesbioscience.com/journals/vaccines/article/26600
References
- Ferlay J, Shin H-R, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. Int J Cancer 2010; 127:2893 - 917; http://dx.doi.org/10.1002/ijc.25516; PMID: 21351269
- Ricciardi A, Largeron N, Giorgi Rossi P, Raffaele M, Cohet C, Federici A, Palazzo F. Incidence of invasive cervical cancer and direct costs associated with its management in Italy. Tumori 2009; 95:146 - 52; PMID: 19579858
- Capocaccia R, Martina L, Inghelmann R, Crocetti E, De Lisi V, Falcini F, Guzzinati S, Rosso S, Tagliabue G, Tumino R, et al. A method to estimate mortality trends when death certificates are imprecisely coded: an application to cervical cancer in Italy. Int J Cancer 2009; 124:1200 - 5; http://dx.doi.org/10.1002/ijc.24033; PMID: 19058196
- World Health Organization. Immunization, Vaccination and Biologicals: Human Papillomavirus (HPV). Last updated: 3 September 2010. http://www.who.int/immunization/topics/hpv/en/
- Rodríguez AC, Schiffman M, Herrero R, Wacholder S, Hildesheim A, Castle PE, Solomon D, Burk R, Proyecto Epidemiológico Guanacaste Group. Rapid clearance of human papillomavirus and implications for clinical focus on persistent infections. J Natl Cancer Inst 2008; 100:513 - 7; http://dx.doi.org/10.1093/jnci/djn044; PMID: 18364507
- Ministero della salute. Vaccinazione contro il Papillomavirus. 13 December 2007. http://www.salute.gov.it/speciali/piSpecialiNuova.jsp?id=75
- Intesa tra il Governo, le Regioni e le Province autonome di Trento e di Bolzano concernente “Strategia per l’offerta attiva del vaccino contro l’infezione da HPV in Italia”. 20 December 2007. http://www.statoregioni.it/Documenti/DOC_016696_264%20csr.pdf
- Ministero della Salute. Campagna di comunicazione 2008. Vaccinazione contro il Papillomavirus. http://www.salute.gov.it/servizio/galleria.jsp?lang=italiano&id=559&dad=s&men=campagn 08&label=emergenze.
- Epicentro. Stato di avanzamento della campagna vaccinale per l’HPV: dati di copertura vaccinale al 31/12/2011 http://www.epicentro.iss.it/problemi/hpv/pdf/Aggiornamento_datiHPV_31_12_2011_validato.pdf
- Guide to Community Preventive Services. Increasing appropriate vaccination. www.thecommunityguide.org/vaccines/index.html. Last updated: 07/22/2013
- Ministero della Salute. Piano Nazionale Prevenzione Vaccinale (PNPV) 2012-2014. http://www.salute.gov.it/imgs/C_17_pubblicazioni_1721_allegato.pdf
- Parasuraman A, Valarie A. Zeithaml, Leonard L. Berry. A Conceptual Model of Service. Quality and Its Implications for Future Research. J Mark 1985; 49:41 - 50; http://dx.doi.org/10.2307/1251430
- Ryan M. Discrete choice experiments in health care. BMJ 2004; 328:360 - 1; http://dx.doi.org/10.1136/bmj.328.7436.360; PMID: 14962852
- Ryan M, Skåtun D. Modelling non-demanders in choice experiments. Health Econ 2004; 13:397 - 402; http://dx.doi.org/10.1002/hec.821; PMID: 15067675
- Coletti G, Paulon L, Scozzafava R, Vantaggi B. Measuring the Quality of Health-Care Services: A Likelihood-Based Fuzzy Modeling Approach Symbolic and Quantitative Approaches to Reasoning with Uncertainty Lecture Notes in Computer Science 2007; 4724: 853-864.
- De Finetti B. Teoria della probabilità. Einaudi, Torino, 1970 (English version: Theory of Probability, New York: Wiley, 1974-5)
- Coletti G, Scozzafava R. Probabilistic Logic in a Coherent Setting. Springer, Series: Trends in Logic, Vol. 15. ISBN 978-1-4020-0917-4. Year 2002.
- Coletti G, Gervasi O, Tasso S, Vantaggi B. Generalized Bayesian inference in a fuzzy context: From theory to a virtual reality application. Comput Stat Data Anal 2012; 56:Issue 4 967 - 80; http://dx.doi.org/10.1016/j.csda.2011.06.020
- Maurici M, Campolongo A, Pozzato S, Giordani A, Bartolaccini T, Ferrante M, et al. Utilizzo di un approccio probabilistico per misurare l’efficienza di un provider sanitario. In: Prevenzione e Sanità Pubblica: modelli organizzativi e buone pratiche. Napoli, 15-17 Ottobre 2009, Vol. Suppl: N. 5, P. 527-528.
- Maurici M, Meleleo C, Campolongo A, D’Anna C, Mangia ML, Sgricia S, Serino L, Paulon L, Franco E, Ferrante M. [Application of the QuavaTAR model to vaccination services in Latium, Italy]. Ig Sanita Pubbl 2010; 66:793 - 801; PMID: 21358776
- Key Performance Indicator Guidelines 2012 based on National Service Plan 2012. Health Service Executive. Ireland 3rd December 2012. http://www.hse.ie/eng/services/Publications/corporate/performancereports/kpiguidelines2012.pdf
- Panatto D, Amicizia D, Trucchi C, Casabona F, Lai PL, Bonanni P, Boccalini S, Bechini A, Tiscione E, Zotti CM, et al. Sexual behaviour and risk factors for the acquisition of human papillomavirus infections in young people in Italy: suggestions for future vaccination policies. BMC Public Health 2012; 12:623; http://dx.doi.org/10.1186/1471-2458-12-623; PMID: 22871132
- Carlino C, Aniuskevich A, Morciano L, Franco E, Zaratti L. [Acceptability of vaccinations in adults]. Ig Sanita Pubbl 2012; 68:885 - 94; PMID: 23370000
- Chini F, Fano V, Pezzotti P, Bontempi K, Perrelli F, Giorgi Rossi P, Guasticchi G. Copertura della vaccinazione anti-HPV nelle adolescenti della regione Lazio. Congress AIE, Bari. October 30.2012.
- Paulon L. “Mathematics in Health Care with Applications”, PhD thesis (chapter 2), Department of Basic and Applied Sciences in Engineering, “Sapienza” University of Rome, Italy, 2012.
- Epicentro. Malattie Infettive. Infezioni da HPV. http://www.epicentro.iss.it/problemi/hpv/hpv.asp
- Regione Lazio: Piano Regionale Prevenzione Vaccinale 2012 - 2014. D C A n. U00192 del 5/11/ 2012.