Cost-effectiveness calculators on health and social services planning and evaluation: an explorative interview study of key informants

ABSTRACT

The aim of this study was to examine the views of key experts on developing and using cost-effectiveness calculators to plan and evaluate health and wellbeing promotion interventions in health and social services. Data for this qualitative interview study were collected from 14 Finnish experts in health and wellbeing coordination, health and social service management and research and health economics in spring 2021. A semi-structured interview method with thematic analysis was used. The experts said that there is a need for cost-effectiveness evaluation tools that support local evidence-based decision-making. This would enable organizations to plan and allocate scarce resources for interventions that promote equitable and effective health and wellbeing. However, practical tools and calculators that enable users to make decisions based on the best available evidence are not widely used. Local decision-makers, researchers and service providers all need to be involved in agreeing goals and selecting the right target groups and measures. They also need to make decisions about the best available data sources and how to use calculators to define and evaluate outcomes. Cost-effectiveness calculators are needed for local evidence-based decision-making, so that municipalities can allocate scarce resources to effective services that increase the wellbeing and equality of residents. This requires key stakeholders to work together to plan, develop and evaluate comprehensive, easy-to-use cost-effectiveness calculators.

KEYWORDS:

• Calculator
• cost-effectiveness
• health promotion
• key informants
• municipality
• wellbeing

Background

Economic evaluations of health and social care services depend on culture, institutional contexts and values (Torbica, Tarricone, and Drummond 2018). Economic tools that support decisions, such as cost-effectiveness calculators, are an integral part of attempts to moderate anticipated increases in public expenditure. They can also help to reduce social inequalities. Scandinavian countries make extensive use of cost-effectiveness evaluations that are strongly focused on social equality (Hale and Giese 2017; Merkur, Sassi, and Mcdaid 2013). These prospective evaluations compare the resources required for a particular intervention against the expected outcomes (Brenas and Shaban-Nejad 2020; CDC 2021). For example, cost-effectiveness calculators can provide mathematical estimates of annual savings achieved by an intervention (Siyan et al. 2015; Hult et al. 2021).

Preventive health and wellbeing interventions combine activities or strategies that seek to increase positive outcomes by influencing behavior (Clarke et al. 2019). They can target whole populations or certain risk groups at both individual and community levels (Qing et al. 2015; Giannakopoulos et al. 2021; Hansell, Jr Giacobbi, and Dana 2021). Evaluating the suitability and effectiveness of interventions at the planning and implementation stages is essential for their continuous development (WHO 2022). However, preventive interventions that provide sustainable and positive outcomes are often limited, especially when they target wider communities (Goldgruber and Ahrens 2010; Nickel and von Dem Knesebeck 2020). Therefore, tools to assess the effectiveness and cost-effectiveness of these interventions need to be reliable and objective when national funding is limited (Stenborg et al. 2021), so that resources are allocated to effective and reasonable measures. Evaluation tools also need to assess non-monetary outcomes, as understanding the broader outcomes, timing, target groups and context is essential for decision-making (Clarke et al. 2019; Ghislaine A.p.g.van et al. 2015).

Measuring the cost-effectiveness of health and well-being promotion is difficult, however, due to limited resources, and prioritizing and choosing which actions to evaluate is challenging. For example, policy problems are often difficult to solve and not clearly defined, and operationalized, for comprehensive cost-effectiveness evaluation (Cheung et al. 2016). Despite this, carrying out economic evaluations with cost-effectiveness calculators could respond to the needs of stakeholders, such as policymakers, researchers and practitioners. According to Health in All Policies, health and wellbeing promotion requires intersectoral work, policy coherence, stakeholder participation, high-quality information and data and health equity. In other words, no-one is left behind (Ramirez-Rubio et al. 2019).

However, there are numerous barriers to adopting cost-effectiveness tools, including uncertainty about the quality of evidence, their limited applicability for settings and a lack of tools. Barriers related to users include limited experience of producing economic evaluations, negative attitudes towards scientific evidence, insufficient skills to interpret evidence, lack of management support and difficulties applying evidence to local contexts (Marieke E van, Severens, and Novak 2005; Drummond 2004; Velasco et al. 2008). Motivational factors that foster the use of economic evaluation tools include the users’ attitudes, skills in applying the tools and social support in the workplace (Cheung et al. 2016). Potential users also need to assess whether the cost can be justified, the relevant outcomes for the organization and if the tool provides sufficient scientific support for decision-making and integrates all the relevant information (Cheung et al. 2016). However, there has been limited research on how cost-effectiveness calculators can be used to plan and evaluate preventive interventions, as part of decision-making. More knowledge would enable us to develop and use calculators that respond to objectives to control costs and increase equality in health and well-being at local, regional and national levels.

Aims

The aim of this study was to examine the views of key informants on the development and use of cost-effectiveness calculators to plan and evaluate interventions that promote health and well-being in health and social services.

Methods

Study design

We used an explorative qualitative study design, with small, semi-structured groups for key Finnish informants in Spring 2021. We reported the study following the COREQ checklist (Tong et al. 2007).

Research environment

Three-quarters of Finnish health and social services are offered by public providers and funded by taxes, but the private social sector has been growing fast (Finnish Association of Private Care Providers, 2021). Some provision is funded by health insurance policies. Municipalities are responsible for organizing services, including those that promote health and well-being. They work with other municipalities or purchase services from private companies or non-governmental organizations. There is ongoing social welfare and health care reform in Finland. This aims to maintain, and develop, equal and adequate access to services and help municipalities to organize and provide services by creating larger units. The reforms will establish 22 health and social services counties that will provide services, instead of the current 300 municipalities, which have many statutory duties (Hakari 2021). However, the municipalities will still be responsible for health and wellbeing promotion.

Participants and setting

We used the key informant technique (Marshall 1996) to purposively recruit participants by sending an invitation to 44 experts in health and wellbeing coordination, health and social service management and research. Invitations were sent by e-mail through the Finnish Healthy Cities Network and the authors’ networks; however, the interviewing authors did not have the previous contact with the participants. Of them, 16 experts agreed, but because of illness, 14 could participate. These experts were considered potential developers or users of cost-effectiveness calculators. We carried out four group interviews with experts with different backgrounds and expertise (Table 1). The mean age of the participants was 51 years, ranging from 41 to 75 years, with equal numbers of males and females, and they represented most of the country, apart from the northernmost part of Finland.

Table 1. Background characteristics of the participants.

	M (Sd)	n (%)
Age, years	51 (9.7)
Gender
Male		7 (50.0)
Female		7 (50.0)
Education
Master’s degree		7 (50.0)
Doctoral degree		7 (50.0)
Position
Developmental and welfare leaders and coordinators		8 (57.1)
Researchers and research leaders		6 (42.9)
Workplace
Hospital districts or counties		5 (35.7)
Universities		4 (28.6)
Finnish Institute for Health and Welfare		3 (21.4)
Municipalities		2 (14.3)

Data collection

A semi-structured interview method was used to collect the data (Kallio et al. 2016). The interview guide was formulated based on a literature review (Hult et al. 2021) and tested with the first interview group. The final guide comprised five themes (Table 2). We also sent the participants a list of cost-effectiveness calculators we had identified before the interview [Supplementary table 1] to launch an initial discussion about calculators and ensure that all participants shared the understanding of them. These calculators targeted workplace health and safety (n = 10), public health and health behavior (n = 8) and social and regional well-being and safety (n = 10) interventions.

Table 2. The semi-structured interview guide.

Initial discussion	What do you think about cost-effectiveness calculators (Supplementary table 1)?Have you been using, or developing, such calculators?
Interview themes	1. What parties/levels need such calculators? (regional/national, public/private, other parties/levels) 2. What effects and target groups should the calculators target? 3. Who should develop and fund calculators? 4. What kind of information could be used in the calculators? (Existing information, new information, other information, what?) 5. What features would affect your use of the calculator? (Usability, accessibility, reliability, other features)
End discussion	Is there anything else you would like to say regarding cost-effectiveness calculators?

Four small group interviews were held, two with three participants and two with four, to enable reflective and cumulative discussion on the topic. These were conducted online with the Zoom video conferencing tool to ensure everyone had the same chance to take part and lasted from 78‒98 minutes, providing us with a total of four hours and 53 minutes of audio-recorded material. Two of the researchers (MH, OH, MK) acted as moderators in each interview and made field notes.

Analysis

We used inductive thematic analysis (Vaismoradi, Turunen, and Bondas 2013) to analyze the data. First, we transcribed the data verbatim, which produced 114 pages of text in 12-point Calibri with 1.5 line spacing. Then we read the data to get an overview of the entire content and organized the expressions to create main themes, categories and sub-categories. Two researchers (MH, HK) analyzed the data using NVivo12 software and the research team finalized the analysis.

Results

The data emerged three main themes: need for calculators, selection and usability of calculators, and the development, use and funding of calculators (Table 3).

Table 3. The views of key informants on the development and use of cost-effectiveness calculators to plan and evaluate interventions that promote health and well-being in health and social services.

Main theme (n = 3)	Category (n = 9)	Sub-category (n = 30)
Need for calculators	The value base for health, well-being and safety promotion	Identifying needs
		Prioritization
	Tool and support for knowledge-based management and decision-making	Strengthening preventive work in municipalities
		Demonstrating the effects of well-being promotion work
		Assessment of the suitability of the measures and comparison of alternatives
	The basis for cost-effectiveness evaluation	The starting point is a change in human well-being
		Effectiveness of services and interventions
		Social impact
		Need for prospective evaluation
Selection and usability of calculators	Outcomes estimated by the calculators	Cost-effectiveness
		Experienced health and well-being
	Calculator features	Comprehensibility and practicality of the calculator
		Technical usability
		Transparency of calculator background assumptions
	Challenges related to calculator selection and use	Scarce resources of municipalities
		There is no national guidance
Development, use and funding of calculators	Roles of different actors	National development and financial responsibility
		Intra-regional and inter-regional differences must be taken into account
		Cooperation between different departments of municipalities in the management of well-being and health
		Networking with the NGOs (third sector) and companies
		Inclusion of the individual
	Data sources and data collection for calculator development	Research data
		Statistics and registers
		Basic data collected by service providers
		Regional and municipal knowledge bases
		Information collected by intelligent technology
	Challenges of data collection and utilization	Relevance and fragmentation of data
		Data protection legislation as a limiting factor
		Accumulation of data alongside normal operation
		Groups for which little data is available

Need for calculators

The experts said that calculators were needed to determine the value base for health and well-being promotion, by identifying the needs and priorities for interventions. Identifying needs referred to recognizing polarization among citizens and selecting the groups who had the greatest need for interventions. The experts concluded that it was difficult to acquire ethically sustainable knowledge for promotion purposes with experimental research design. They said that calculators could promote equal services and prioritize key groups. They discussed different groups, such as older people, children and youths, and whether interventions should prioritize frequent service users.

‘Calculators could be used to change the services towards preventive. It is difficult if there is nothing to show the effectiveness of interventions’. (Group 1)

Calculators could support evidence-based management and decision-making that strengthen preventive work in municipalities. They could assess the costs of the transition from care to prevention and the effects of early interventions. Calculators were also needed to boost, and show, the vitality of local and regional initiatives. Experts said calculators could make the effects of promoting well-being understandable and transparent for the public and other authorities. Moreover, calculators could help to assess the suitability of measures, compare optional interventions and select the best approaches. They could compare national and international data and motivate municipal stakeholders to make their procedures more effective.

‘Whenever decisions impacting well-being are taken and prepared, this ex-ante evaluation for decisions should be used. That makes the decision-making transparent’. (Group 2)

Calculators could also be used for cost-effectiveness evaluations. Experts said that the starting point for health and well-being promotion was supporting local people in the best possible way. However, they also highlighted the need to prospectively evaluate the effectiveness of services and interventions and said that calculators could be used to quantify aims and assess ineffective interventions that could be discontinued. Calculators could be also used to strengthen, and demonstrate, the effects and effectiveness of services. They were also needed to assess the impact of interventions on society, influence political decision-making and address long-term effectivity.

‘Cost-effectiveness is also essential in the sense that our decision-makers understand the euros. Understandably, describing well-being work is much more challenging than showing euros; in that sense, we would also need a tool in our small municipality’. (Group 1)

Selection and usability of calculators

Experts said that calculators should provide two equal outcomes: cost-effectiveness and health and well-being. Cost-effectiveness referred to costs, benefits and efficiency, while health and well-being outcomes referred to more comprehensive changes in the population, including people’s experiences of health and quality of life. Participants also said that the calculators’ features affected its usability. It was important that calculators were practical and easy to understand and use. For example, they needed to help the user interpret and present results, as well as being free of charge and easy to find.

‘For me, cost-effectiveness means that the well-being of individual changes’. (Group 1)

The calculators needed to provide easily transferable data that could be further analyzed and visualized. Open-source calculators were beneficial because others could continue their development. The questions posed by calculators needed to be accurate and based on transparent background assumptions, so that the user could see the logic and reliability of the results that were produced. The experts said that it was important that calculators were based on scientific evidence.

‘Open-source models would be good, there would be no copyright for any project, funder or university. We could have one unified base bank, and not that everyone does their own thing and then only with hard money you can update it’. (Group 4)

There were challenges related to the selection and use of calculators. The experts stated that municipalities had scarce resources and possibilities to use calculators, and this highlighted the need to incorporate them into the daily management of the organizations. They pointed to the lack of national guidance on using calculators and emphasised the need for centralized funding for their development and recommendations for their use. The experts would like to see all municipalities using the same calculators, rather than different ones.

Development, use and funding of calculators

The roles of the different stakeholders in the development, use and funding of calculators was highlighted, and these were national, regional, municipal, non-governmental organizations and individuals. The experts emphasised the Finnish Government’s central development and funding role and their responsibility to steer local stakeholders towards cost-effective services. They highlighted the need for centralized calculator development that was integrated into governmental funding systems. Potential developers were the Finnish Institute for Health and Welfare, universities, other research institutions and pension insurers.

‘One challenge, after all, is that we have all kinds of tools. Still, if each municipality chooses its own, we don’t get systematic information or choose quite different indicators to collect. There should be someone who recommends a calculator’. (Group 3)

Experts mentioned potential regional actors and highlighted the need for taking intra-regional and inter-regional differences into account when developing calculators. They underlined the differences in different geographic morbidity rates and how services were organized and the need for adequately resourced regional governance. Regional networking was also needed by hospital districts and research and development projects.

‘The regions and the municipalities are different, and within the municipalities, there are very polarized areas’. (Group 4)

Stakeholders who developed calculators needed to collaborate with different municipal departments. The experts underlined the wide-ranging environments they worked in and the need for municipal health and well-being promotion to engage other sectors. They pointed to the significant differences in the populations of different municipalities, and within big cities, and the importance of choosing the most suitable calculator for the area.

Networking with non-governmental organizations was considered beneficial in developing calculators, as organizations could share data collected by service providers. The experts said that it was important that all the companies adopted efficient investment and prevention approaches and that calculators could help them do that. In addition, they emphasised the importance of raising an individual’s awareness and responsibility for their own well-being. The experts also said that those members of the public who used smart technology in their daily lives could also provide information on health and well-being promotion.

‘The involvement of the third sector in lifestyle guidance and substance abuse treatment is essential’. (Group 3)

The experts said there were several possibilities when it came to the data sources and collection. Research data could be collected with experimental or quasi-experimental designs and systematic reviews, but the study context was important. Useful data included the Finnish Institute for Health and Welfare registers, the national School Health Promotion Study and Statistics Finland. Participants also considered that data from health centers, trade unions and other authorities could be useful for developing calculators. Regional and local databases were also mentioned, such as municipality reports and inquiries. However, the experts believed that the fragmented existing data made developing calculators challenging.

‘There is expert knowledge, which can sometimes be enough, and experiential knowledge, which can also sometimes be enough. However, the crucial question is how strong evidence of effectiveness we want to be behind decision-making’. (Group 3)

The experts identified some challenges for collecting and using data for developing calculators. Some data were irrelevant or unsuitable, due to slow updates, different recording policies by service providers and because they were only collected for administration purposes. Another challenge was data protection, which limited the information that different stakeholders could share. Experts also highlighted that separate data collection for calculators was seldom possible in daily practice. For example, municipal data should be collected during normal activity. They also mentioned the challenges of gathering data within groups where little data was available, including working-aged people and particularly those who used many services.

‘Is the information available alongside the regular function? Even if there are excellent and elegant cost-effectiveness assessment methods, the data collection can be too laborious’. (Group 1)

Discussion

Our results showed that cost-effectiveness calculators could be a necessary and practical part of planning and executing health and well-being promotion, but intersectoral and multidisciplinary cooperation by all key stakeholders was needed to develop feasible, targeted calculators. Cost-effectiveness calculators were not widely used at municipal, regional and national levels of decision-making. However, all the experts in this study highlighted the importance of evidence-based policy, prioritizing municipal interventions and the strategic use of scarce resources. They also stated that reliable and timely calculators relied on background data that was up-to-date, relevant and accurate.

This study emphasised the value-basis of the use and development of cost-effectiveness calculators. Interestingly, the experts stated that the value basis was not just related to monetary values (Ghislaine A.p.g.van et al. 2015; Clarke et al. 2019). It also referred to the how the calculators were used and targeted, how the results were interpreted and selecting a target group for preventive services or interventions. In addition, the experts said that setting profitability margins was a value-based decision. Therefore, intersectoral approaches and common goals were essential among municipalities, as rigid systems, value conflicts or competing objectives did not produce longer-term societal benefits (Williams and Bryan 2007). The experts called for a holistic and humane perspective of wellbeing to be adopted but stated that cost-effectiveness evaluations often provided intangible outcomes that were difficult to value, measure and quantify in terms of money (Leck, Upton, and Evans 2016).

The calculators to be selected stemmed from local policies. The experts agreed that calculators could not be selected for every need, due to scarce municipal resources, and that only a few could be used. Previous studies also showed (Ghislaine A.p.g.van et al. 2015; Clarke et al. 2019) that the ability to choose and apply the right, and most suitable, calculators was important. However, there is a need for national and centralized guidance and funding for calculators. It does not make sense to develop a calculator for all local needs, as its results would be too vague and untargeted (Clarke et al. 2019). This main challenge of selecting and developing calculators is that they must be sufficiently sensitive to the context and customized for local needs.

The interviewed experts wanted calculators with a simple and comprehensive designs and user-friendly web-based interfaces (Cheung et al. 2016; Götschi et al. 2020). The calculators should provide visual information that is easy to understand by users without any professional background in economics or health promotion. In order to be acceptable, they need to be strongly evidence-based and be valid from a societal, economic, health system and environmental point of view (Clarke et al. 2019).

Implications for practice

Our results indicate there are two key challenges with regard to developing and using cost-effectiveness calculators. The first is how to conduct economic evaluation of holistic and multi-component services using calculators that are based on reliable data. Acceptability and trust can be eroded if calculators use fragmented, partly elusive and slowly updated data (Williams and Bryan 2007). Data sources are abundant, but they will only be useful for cost-effectiveness calculators if they are reported in accessible, relevant and meaningful formats by parties involved in the development process (Leck, Upton, and Evans 2016). Decision-makers must be involved in the whole development process (Williams and Bryan 2007), along with researchers and health economists.

The second challenge is how the scientific evidence provided by calculator can make the difficult transition from theory to practice (Marieke E van, Severens, and Novak 2005). Another challenge is that the objectives of decision-makers are often too complex to be turned into unambiguous and measurable indicators. Local decision-makers would probably need a trained analyst to help with calculators (Williams and Bryan 2007). Health and social care planning and evaluation require a long-term commitment and the experts highlighted the capacity to continue preventive measures if the administration of the municipal councils changed following elections (Visram et al. 2020).

Strengths and limitations

This study had some limitations. The number of participants was small, but they were experts in their field and the data began to saturate. Also, the concepts of cost-effectiveness calculators have not been established. Although our experts had limited experience of these calculators (two of the participants knew HEAT but had not used it), they were actively aware of the local need for them, and they described their expectations based on their practical and research experience. Of course, the concrete use of calculators would have brought experiential insights into our study. Face-to-face interviews may have been better with regards to confidentiality, greater information and group dynamics, but this was not possible because of COVID-19 restrictions. We also wanted to encourage experts from all over the country to take part. Finally, our results reflect the current status of health and social policy in Finland and the expectations that long-awaited health and social care reforms will finally materialize. The results may not be transferable to other contexts.

Conclusions

The experts we interviewed said that there is a need for cost-effectiveness evaluation tools that support local evidence-based decision-making. This would enable organizations to plan and allocate scarce resources for interventions that promote equitable and effective health and wellbeing. However, practical tools and calculators that enable users to make decisions based on the best available evidence are not widely used. Local decision-makers, researchers and service providers all need to be involved in agreeing goals and selecting the right target groups and measures. They also need to make decisions about the best available data sources and how to use calculators to define and evaluate outcomes. The selection of calculators should be nationally, and centrally controlled, as limited municipal resources would only allow the use of a few calculators.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was funded by the Finnish Prime Minister’s Office (grant number VN/14626/2019).