Improving a web-based employability intervention for work-disabled employees: results of a pilot economic evaluation

Abstract

Purpose The purpose of this study is to improve web-based employability interventions for employees with work-related health problems for both intervention content and study design by means of a pilot economic evaluation. Methods Uptake rate analysis for the intervention elements, cost effectiveness, cost utility and subgroup analyses were conducted to identify potential content-related intervention improvements. Differences in work ability and quality-adjusted life years and overall contribution of resource items to the total costs were assessed. These were used to guide study design improvements. Results Sixty-three participants were a-select allocated to either the intervention (n = 29) or the control (n = 34) group. Uptake regarding the intervention elements ranged between 3% and 70%. Cost-effectiveness and cost-utility analyses resulted in negative effects although higher total costs. Incremental effects were marginal (work ability −0.51; QALY −0.01). Conclusions The web-based tool to enhance employability among work disabled employees requires improvements regarding targeting and intensity; outcome measures selected and collection of cost data. With respect to the studies of disability and rehabilitation, the findings and methods presented in this pilot economic evaluation could guide the assessment of future assistive “e-health” technologies.

IMPLICATIONS FOR REHABILITATION

• The methods presented in this pilot economic evaluation have large potentials to guide the assessment of future assistive e-health technologies addressing work-disabilities.

• The findings show that the web-based tool requires content related improvements with respect to targeting and intensity to enhance employability among work disabled employees.

• The findings show that the web-based tool would benefit from improvements related to the study design by more adequately selecting and collecting both outcome measures and cost data.

• The burden attributable to large-scale studies and implementation issues were prevented as the outcomes of the pilot economic evaluation did not support the implementation of the web-based tool.

Keywords:

• E-health
• healthcare
• prevention & control
• productivity
• return-to-work
• work disability
• work functioning

Introduction

Since 2000 research has established that sickness absence results in considerable negative effects on work for employees, employers, and society as a whole. The absence from work due to sickness absence constitutes a large economic burden, especially for those on long-term sickness absence.[1,2] Employees with health complaints, chronic illness or disability often worry about the sustainability of their employment. Due to their health complaints, problems can arise at work, primarily with respect to either obtaining work or remaining at work. Franche et al. and Amick et al. in 2002 and 2004, respectively, provided evidence regarding the readiness for return to work and predictors for successful work functioning.[3,4] Both studies found significantly improved self-efficacy, or self-perceived work ability, among supportive organizations that acknowledge the importance of psychosocial management [4] and the interplay among care, workplace, and insurance.[3] Additionally, the importance of improving an employee’s self-efficacy to maintain successful work-role functioning and improve work ability has been established.[3–6] More recently, Hoefsmit et al.[7] found that in order to improve work ability, cooperation between employe and employer is essential. Likewise, employees with higher levels of self-efficacy are better able to stay at work despite their health problems and may even return to work faster after reporting sick.[7] Furthermore, a recently published study showed that return-to-work self-efficacy was a significant predictor of reintegration to work among very long-term sick-listed employees.[8] Thus, improving self-efficacy may empower an employee to handle difficulties in relation to his or her health problems.[9]

In recent years, the Internet is being used more often to locate information on a broad range of topics, including work- and health-related issues. Previous research has investigated the reach, compliance, and perceived effectiveness of interactive websites or e-therapies aimed at empowering work-disabled employees. However, these interventions occurred either prior to the disability assessment undertaken by insurance physicians [10,11] or focused solely on information provision and helping patients to formulate questions to ask their physicians.[12,13] Because of the increasing number of Internet users, a growing number of interactive interventions are delivered online.

To increase self-perceived work ability (i.e., self-efficacy) among work-disabled employees who experience problems in relation to their employability, a web-based employability intervention was developed. The web-based intervention consisted out of four different intervention components: (i) a knowledge website, (ii) personal advice, (iii) a feedback session, and (iv) a forum.[14]

Previously, the effectiveness of a web-based employability intervention was evaluated among unemployed persons. Unfortunately, among an unemployed study population, no overall effects were found and only one intervention component appeared to be effective on the degree of self-control, empowerment, or return to work.[15] Before assessing the impact of a web-based intervention in another study population, i.e., employees at work hampered by health complaints, it was assumed very valuable to first critically assess both the study design and the content of the web-based employability intervention before conducting a full and extended assessment of the web-based tool. Smaller pilot trials can inform about the feasibility to conduct (cost-) effectiveness trials, which can be costly and time-consuming.[16] This saves valuable resources being unnecessarily spent on large scale studies that may not be feasible.

The purpose of this study is therefore to detect how to improve web-based employability interventions for work-disabled employees for both intervention content and study design by means of a pilot economic evaluation. The aim of this pilot economic evaluation was two folded: (i) To evaluate whether, based on the results of the pilot economic evaluation, the content of the web-based employability intervention requires improvements to achieve widespread implementation. Insight in the uptake of the various intervention elements and the effectiveness in diverse employee groups are therefore required. (ii) To evaluate whether, based on the results of the pilot economic evaluation of the web-based intervention, the study design requires improvements to guide future research.

Materials and methods

The data collection for this pilot economic evaluation was conducted in the Netherlands between June 2013 and March 2014 with a follow-up period of 3 months. The Medical Ethics Committee (METC) was consulted and declared no ethical approval was needed. The research was conducted conforming The Code of Ethics of the World Medical Association (Declaration of Helsinki).

The web-based employability intervention was compared with regular trade union support among participants with work-related health problems. Members of the trade union could call or use the instant messaging service via the website to ask questions about work, health, and disabilities. They were further attended to a union consultant who answered the questions or referred the caller to a disciplined expert. Participants were approached via the participating employees trade union in the public sector. Members of this union predominately worked for privatized government agencies or in the care and welfare sector. Eligible participants were adults aged between 18 and 65 years. Participants either had concerns regarding their ability to continue their work responsibilities, were at a crossroad of calling in sick for a longer period of time, or were absent for more than 6 weeks and were feeling unable to return to work in the near future. Those who were absent from work for more than 2 years were excluded from the study. Advertisements in the trade unions’ newsletters targeted the participant group and based on individual’s own intrinsic motivation could be recruited into the research. After the completion of the first questionnaire, participants were randomly divided into either the control group or the intervention group. Participants and researchers were not blinded for either the treatment allocation or intervention uptake. In the control group, a reward of €10 was provided to those who completed both questionnaires. In the intervention group, participant received free usage of the web-based employability intervention for three months.

Intervention and control condition

Participants from the control group could contact the participating trade union for regular support (e.g., information on website, telephone contact, etc.) regarding their work- and health related concerns. The control group did not have the access to the web-based employability intervention.

The web-based employability intervention focused on improving the self-efficacy of work-disabled employees by encouraging and motivating them to take control of their own work context and enhance their work ability. The web-based employability intervention focused on teaching employee skills to return to work and/or stay at work (i.e., to optimize reintegration). In particular, the training allowed the participants to ask job and health-related questions, as well as guiding communication with relevant stakeholders (e.g., supervisors, colleagues, occupational physicians, medical practitioners, labour experts, job coaches). Specifically, the intervention aimed to “empower” employees to access regular, curative, and occupational health care services during their illness, recovery and work reintegration. The four core elements of the intervention were: (i) a knowledge based web-site of current information focusing on work ability in relation to health; (ii) an assessment of the work ability of respondents (more information on the Work Ability Index-test can be derived from the Technical Appendix), followed by a subsequent telephone interview providing respondents with feedback on their results; (iii) personal and technical expert advice via phone and email; and (iv) an online interactive forum on work and health for sharing worker’s experiences, questions, and answers with content experts. Using the intervention elements was voluntary. However, a subscription to the knowledge site was needed (http://www.wijzermetwelder.nl/).

Aim I: evaluating the content of the web-based employability intervention

The first aim of this pilot economic evaluation was to consider whom to include and how to target suitable participants for the web-based employability intervention in the future. The intervention aimed to provide an easily accessible method for interested people to ask questions about disability, health and work. Therefore, the intervention was offered to interested members of the included trade union who experience problems in relation to their employability, regardless of their type of problems or disabilities. A number of intervention features were studied. First, the uptake rates for each of the four core elements of the intervention were examined. Additionally, the economic evaluation was conducted for both the complete group of workers and several subgroups. A cost-effectiveness analysis (CEA) and a cost-utility analysis (CUA) were conducted by accounting for all costs, irrespective of who pays or bears them. Subgroup analyses were conducted based on the age (under and above 50 years), sex (males versus females), and working hours (more than 20 h per week). The results of the economic evaluation were expected to provide information on how to better target the intervention and support recommendations regarding compliance and uptake.

Aim II: evaluating the study design of the web-based employability intervention

The second aim of this study was to improve the study design of web-based employability interventions by using the results of the pilot economic evaluation to carefully select outcomes and cost-categories that matter both to the employee and to the society as whole.

Therefore, first, a critical assessment was made of the differences in work ability and quality adjusted life years (QALY) among both groups. Work ability was measured via “The Work Ability Index” (WAI), by considering both the physical and mental demands of the work. The index helps to identify those that need support in these areas.[17] A point improvement on the WAI denoted a substantial improvement and hence, demonstrated a gained effect (i.e. better work ability). Differences in Quality Adjusted Life Years (QALYs) were calculated using the EuroQol-5D-5 L.[18,19] A crosswalk to the 3-level value sets was used to calculate the utility weights based on the EuroQol scores using Dutch tariffs.[20,21] Utilities at baseline and after 3 months follow-up were used to compute QALYs, by means of the area-under-the-curve method.[22] More information on the WAI and the EuroQol-5D-5 L can be found in the Technical Appendix.

Second, in order to weigh the contribution of certain cost-categories (healthcare costs, patient and family costs, and costs in other sectors) to the calculation of the total costs, the utilization frequency of these cost-categories, and the attributed unit costs were evaluated. Questionnaires measuring healthcare resource use were provided to the respondents at baseline and 3 months after the intervention started. Healthcare costs (including primary, secondary, alternative care, hospitalization, and medication), costs to patient and family (including travel- and parking, and informal care costs), and productivity costs (including costs related to absenteeism and presenteeism) were estimated and indexed for the reference year 2013.[23–26] Costs of the web-based employability intervention included those related to implementing and operating the intervention.

Statistical analysis

Missing data per respondent were replaced by mean imputation using the mean of the series of the appropriate outcome. The analyses required complete cases responses for both baseline and 3 month follow-up measures. Therefore, respondents who were lost to follow-up were excluded from the analyses. Baseline characteristics of both groups were compared using t-tests of variances, or by the non-parametric equivalent. That is, Kruskal–Wallis for analysis of continuous variables or χ² tests for categorical comparisons of proportions. The costs of both groups were compared by the non-parametric bootstrapping method using 95% confidence intervals in percentiles. By bootstrapping, samples of the same size as the original data were drawn with replacements from the observed data. An incremental approach was used to compare the employability intervention with regular trade union support. The incremental costs per “improved” respondent on the WAI scale and per QALY gained (i.e., the incremental cost-effectiveness ratios (ICER)) were graphically plotted on one of the four quadrants of the cost-effectiveness planes. For this, non-parametric bootstrap re-sampling techniques were, therefore, used. In the North West quadrant, if fewer effects were produced for additional costs, then the intervention was considered “dominated” by the control condition. In the South East quadrant, if the intervention generated superior effects (relative to the comparator condition) with fewer costs; the intervention was considered “dominating” the control condition. Detailed calculations relevant for the economic evaluation are found in the Technical Appendix.

Results

Participants

Between June 2013 and March 2014, 70 potential participants were eligible for inclusion, provided their informed consent, and were allocated to either the web-based employability intervention (n = 34) or the control (usual care) group (n = 36). In total, 63 participants (29 in the intervention group; 34 in the control group) completed both the baseline and the 3 month follow-up questionnaires and were included in the analyses. The demographic characteristics of the participants are shown in Table 1. At baseline, more females were allocated to the intervention group. A significant difference was found for mean health scores with participants in the intervention group rating their health lower than the control group. Aside from gender and self-rated health, no significant differences between the two groups with respect to their demographic characteristics were found.

Table 1. Baseline characteristics (N = 63).

	Intervention group (n = 29)	Control group (n = 34)	p Values
Age, mean (SD)	51 (8)	53 (7.9)	0.4(a)
Female, N (%)	22 (76)	18 (53)	0.06(b)
Education^a, N (%)			0.8(b)
Low	2 (7)	3 (9)
Intermediate	10 (34)	14 (41)
High	17 (59)	17 (50)
Working hours, mean (sd)	26.5 (8.9)	29.3 (8.5)	0.2(a)
Living situation, N (%)			0.4(b)
Living alone	5 (17)	8 (24)
Living alone with children	–	2 (6)
Living together with a partner	14 (48)	17 (50)
Living together with partner and children	10 (35)	7 (20)
Job demands, N (%)			0.8(b)
Physical	2 (7)	1 (3)
Psychological	14 (48)	18 (53)
Both physical and psychological	13 (45)	15 (44)
Function, N (%)			0.14(b)
Industrial sector	–	2 (6)
Agricultural sector	1 (3)	–
Administrative sector	3 (10)	10 (29)
Service sector	1 (3)	–
Healthcare sector	15 (52)	9 (27)
Research & development	2 (7)	3 (9)
Other	7 (24)	10 (29)
Social benefits, N (%)			0.6(b)
WIA/WGA/WAJONG	3 (10)	3 (9)
Other	2 (7)	1 (3)
None	24 (83)	30 (88)
Permanent employment, N (%)	27 (93)	33 (97)	0.4(b)
Absenteeism days, mean (sd)	5.47 (8.6)	3.21 (6.9)	0.15(c)
Self-rated health^b, mean (sd)	67 (12.3)	75 (17.2)	0.015(c)
Costs, mean in €(sd)
Intervention	56.51 (22)	–	–
Medication	350.86 (926)	197.86 (679)	0.465(c)
Healthcare	644.91 (858)	918.50 (3052)	0.187(c)
Patient and family	68.39 (120)	48.53 (61.93)	0.341(c)
Absenteeism	847.05 (1439)	543.87 (1195)	0.243(c)
Presenteeism	241.87 (296)	311.81 (283)	0.189(c)

a Low, preschool or primary school; intermediate, lower- or upper secondary; high, tertiary school, university or postgraduate.

b Self-reported health score ranging 0 (worst) to 100 (best).

(a) ANOVA; (b) Х test; (c) Kruskal–Wallis test.

The content of the web-based employability intervention

The uptake rates for each of the four core elements (knowledge website, personal advice, feedback session, and forum) of the intervention were examined and found low. Out of the 29 participants in the intervention group, 50% (n = 15) logged on to the knowledge site, approximately 70% (n = 20) of the respondents participated in a telephone interview after completing the WAI-test to discuss their results, 16 requested personal and technical expert advice, and only one person used the online interactive forum. Its clear compliance and uptake are low. In order to improve the uptake, the cost-effectiveness of the complete group and several subgroups could provide insights. First, the economic evaluation outcomes of the complete group of workers with health complaints are presented. Marginally lower scores on the WAI (−0.51) and higher total costs (€483.8) for the intervention group (when compared with the control group) resulted in a negative ICER (Δ total costs/Δ total effects) (see also Table 2). The majority of the incremental cost-effect pairs were located in the northwest quadrant indicating fewer effects, albeit higher costs, for the web-based employability intervention (Figure 1). The 3-month ICER resulted in a large negative incremental cost utility ratio surrounded with large uncertainty which can be seen in the wide distribution of ICERs (Figure 1). Eighty-two percent of the bootstrapped cost utility pairs were situated in the north-west quadrant, representing less effectiveness (i.e., less quality of life gains), and more costs for the intervention group compared with the control group.

Figure 1. Cost-effectiveness planes for (A) cost-effectiveness analysis and (B) cost-utility analysis.

Table 2. Mean bootstrapped cost and effect differences between subgroups of the intervention- and control group including incremental cost-effectiveness ratios, 95% CIs and cost-effectiveness plane distributions.

	Sample size		ΔC	ΔE		Distribution of CE plane^c
Analysis	Intervention	Control	Euro	WAI QALY	ICER (95% CI)	%NE	%SE (dominant)	%SW	%NW (inferior)
Total group
CEA^a	29	34	483.8	−0.51	−948.37 (−1164; 335)	5	4	7	84
CUA^b	29	34	483.8	−0.01	−45 045.13 (−51 078; 11 526)	6	3	9	82
CEA Subgroup
Aged under 50	12	12	345.94	−0.67	−518.91 (−496; −3133)	3	7	16	75
Aged above 50	17	22	9.74	0.03	280 (−88; 2318)	13	41	8	38
Females	22	18	58.01	0.05	1276.16 (5450; 10 667)	19	35	10	36
Males	7	16	1147.36	−1.37	−839.90 (−970; 151)	4	4	2	91
Working hours >20	21	28	261.54	−0.57	−457.69 (1354; 4588)	1	7	22	70

a In the CEA ΔE is the mean difference in work ability (WAI score), ΔC is the mean difference in societal costs.

b In the CUA ΔE is the mean difference in QALYs, ΔC is the mean difference in societal costs.

c NE refers to north-east quadrant of the cost-effectiveness plane, which indicates the intervention is more effective and more costly than the control. SE refers to the south-east quadrant of the cost-effectiveness plane, which indicates the intervention is more effective and less costly than the control. SW refers to south-west quadrant of the cost-effectiveness plane, which indicates the intervention is less effective and less costly than the control. NW refers to north-west quadrant of the cost-effectiveness plane, which indicates the intervention is less effective and more costly than the control.

Second, the results of the subgroup analyses are presented in the lower level in Table 2. Subgroup analyses can distinct certain groups more or less sensitive for the intervention. Hence, subgroup analyses can provide valuable information on targeting priorities. The CEA for the subgroup respondents aged above 50 years and the subgroup female genders resulted in a minimal increase in work ability (0.03 and 0.05 points improvement on the WAI, respectively), associated with higher costs compared with the respondents in the usual care condition. In order for the intervention to be effective, a point improvement on the WAI was required. However, the results of both subgroup analyses should be interpreted with caution as one cannot state that the web-based employability intervention was more effective although it had higher associated costs. Furthermore, based on the distribution of ICERs on the CE plane, these estimates are not robust. The subgroup of employees working more than 20 h per week resulted in lower work ability in the web-based employability intervention group compared with the control group. There were also higher total costs with a negative ICER of −457.69. It is important to note that the ICER does not fall within the confidence interval (Table 2). These results provide preliminary information on who to target the intervention on. Better targeting can consequently impact the effect of the intervention, and thereby impact the uptake and compliance. Content-wise adjustments derived by adequate targeting might improve the uptake for specific subgroups of employees and thus influence the results.

The study design of the web-based employability intervention

In order to guide a careful selection of appropriate outcomes, first, the differences in work ability and quality adjusted life years (QALY) were assessed among both groups. The mean incremental differences of −0.51 point improvement on work-ability and mean incremental differences of −0.01 QALY were unable to demonstrate an effect of the web-based intervention (Table 2).

Second, the contribution of certain cost-categories to the calculation of the total costs showed that participants who received the intervention consulted their physiotherapist and medical specialist more often than participants in the control condition. They used more alternative care and had longer stays in the hospital. The number of visits to the general practitioner and the medical specialist were almost equal in both groups. The number of hours of informal care was larger in the control group when compared with the intervention group. Table 3 presents the uncorrected costs of different healthcare utilization costs after 3 months. As costs were not distributed normally (Table 1), aggregated costs per category (healthcare costs, patient and family costs, costs in other sectors, and societal costs) were bootstrapped. The highest total societal costs were seen in the intervention group (€1707) versus (€1223) in the control group. The relative contribution of patient-and-family out-of-pocket costs was smallest; while productivity costs contributed for almost 50% of the total societal costs in both groups (Table 3).

Table 3. Cost prices, mean total costs (SD) and mean bootstrapped costs (95% CI) during the 3-month follow-up (n = 63).

		Mean costs (SD)		Mean bootstrapped costs (95% CI^a)
Cost type	Cost price per unit (€)^b	Intervention group	Control group	Intervention group	Control group
Healthcare costs		399.7 (386.1)	365.9 (566.99)	399 (270;543)	366 (194;574)
General practitioner	30.5	37.1 (63.3)	40.59 (66)	36 (17;62)	40 (22;65)
Physiotherapist	33.7	165.1 (225.7)	102.1 (180.72)	165 (91;250)	102 (49;165)
Occupational physician	38.1	16.5 (40)	26.39 (48.23)	17 (5;33)	26 (12;43)
Medical specialist	34.3	27 (40.32)	25 (47.21)	27 (14;43)	25 (11;42)
Psychologist	94.3	31.5 (80.2)	91 (216.04)	30 (4;64)	90 (27;169)
Hospital admission	549.7	19.3 (102.03)	0.27 (1.55)	19 (0;57)	0
Hospital care day parts	273.2	19.3(101.41)	8.43 (46.84)	19 (0; 57)	8 (0; 25)
Alternative care	67.1	75.6 (170)	21.1 (60.78)	75 (24; 144)	21 (4; 44)
Home care	71.6	8.5 (40.11)	54.47 (248.85)	9 (0; 24)	51 (0; 143)
Medication costs	Variable^c	280.3 (846.36)	147.9 (299.62)	281 (84; 627)	148 (69; 262)
Patient and family costs	Variable^d	43.5 (45.4)	46.9 (64.28)	43 (29; 61)	47 (28; 69)
Costs in other sectors		890.1 (1334.35)	662.2 (1028.5)	890 (457; 1411)	661 (364; 1040)
Absenteeism	Age & gender specific wages^e	724.4 (1330.65)	464.5 (1064.96)	723 (293; 1250)	462 (150; 853)
Presenteeism	Age & gender specific wages^e	165.7 (71.98)	197.7 (89.22)	166 (141; 192)	198 (168; 226)
Total social costs		1706.8 (1675.1)^f	1223 (1539.97)	1707 (1153; 2327)	1223 (758; 1767)

a The upper and lower confidence limits are the 2.5th and 97.5th percentile based on 10 000 bootstrap replications.

b Standard costs for the Netherlands, 2013 index.

c Dutch Pharmacotherapeutic Compass.

d Including informal care costs and travel and parking costs.

e Dutch guidelines for costing studies.

f Deviation from the sum due to the inclusion of per-participant intervention costs.

Conclusion

This research demonstrates a number of benefits for undertaking a pilot economic evaluation to provide recommendations regarding the content and the study design of the web-based intervention. Based on the results of this pilot, an attempt is made to bridge the gap between the application of research and a real-life setting. Furthermore, this pilot economic evaluation is unique with respect to the study field of disability and reintegration as the findings and methods presented here could guide the assessment of future assistive “e-health” technologies for work-disabled employees.

The content of the web-based employability intervention

The first aim of the study was to consider the uptake rates and critically assess the intervention by conducting a pilot trial based economic evaluation to provide suggestions to achieve widespread implementation. The findings of the pilot economic evaluation do not indicate a widespread implementation of the web-based employability intervention in practice before improvements related to the content and compliance to the intervention are conducted.

First, all four intervention elements need to be improved to increase compliance and uptake rates. In particular, the content of the online interactive forum should be further investigated, by conducting, for example a process evaluation, as uptake of this component was very low. Second, the results of this pilot economic evaluation showed that the web-based employability intervention was not associated with superior clinical or economic impact when compared with regular trade union support. The fact that the main analyses indicated that the intervention is being dominated by the control condition (i.e., more costs and fewer effects as the ICER is located in the inferior quadrant of the cost-effectiveness plane) could also be explained by low compliance and low uptake, hampering the correct interpretation of the impact of the intervention. Based on the economic evaluation results, the subgroup analyses demonstrated a tendency females and participants aged 50 or older might be more sensitive for an intervention effect. Streamlining the intervention content more towards the employee groups which appear more sensitive for an intervention effect, in other words, “better targeting” the study population towards the intervention components might strengthen the trustworthiness of the results, and off course the uptake rates.

Improving the study design of the web-based employability intervention

The second aim was to provide insight in how to improve the study design by selecting those outcomes and cost-categories that matter both to the employee and to the society. First, in this pilot economic evaluation, no significant improvements were found in effectiveness of the intervention when compared with the control condition. Based on the negligible mean incremental differences, and the inability of the outcome measures to detect an effect, it is unclear whether it is appropriate to assess the true effect of these types of interventions in terms of “quality of life” and “self-efficacy”. Furthermore, only the intervention itself was expected to yield differences in outcomes. However, in practice, it might appear so that participants use information other than via the (web-based employability) intervention or that the intervention might be used in combination with other resources (e.g., social networking, other websites, etc.) which might hamper the correct interpretation of the true effectiveness of the intervention. Second, given the time and effort required to collect cost-effectiveness data, this pilot indicates that the collection of patient-and-family costs might be omitted based on their relative contribution to the total costs. Productivity costs on the contrary appear to be essential in the economic evaluation of web-based employability.

Discussion

The study results should be placed in the light of other research conducted within the field of e-health and web-based interventions. A recently conducted review summarized the effectiveness (empowerment-related outcomes) of several web-based interventions.[9] They concluded that when small positive effects on empowerment were found, the duration and the intensity of the interventions were much higher than the web-based employability intervention described here.[9] For clinical problems and the work ability assessment of employees, a long-term time span might have been more appropriate to capture all consequences of the intervention under study. The limited follow-up period (3 months) of this pilot might also mask the visibility of a true effect (e.g. increase in work ability over a longer period of time). However, another study with a longer follow-up period (6 months) was also unable to detect improvements in work functioning.[27] The authors indicated that the reasons for poor implementation of the e-health intervention could be attributed to the fact that the intervention was not fully integrated into the organization before testing its effectiveness. These are similar finding to our pilot study which had low uptake rates.

In addition, although randomized-controlled trials (RCT) are considered the best method to examine the effectiveness of an intervention or treatment, this pilot questions the feasibility of conducting an RCT in a pragmatic setting. The use of web-based employability interventions within a non-disease specific population raises the issue of generalizability of the results versus the applicability of the results in a disease specific population. As mentioned before, no significant effects were found. On the contary, because of the inclusion of many types of work-related disabilities, the generalizability of the results of the RCT is upturned. Considering the challenges of using RCT methods to study the effectiveness of the intervention, a pilot cost-effectiveness study could be accompanied by observational and qualitative studies to explore whether and how to conduct RCTs.

Practical implications and recommendations

Based on the findings, the voluntary usage of all four elements is advised to be reconsidered. Furthermore, methods to stimulate ongoing involvement with web-based employability interventions are suggested to be found and appraised. However, difficulties were experienced when recruiting employees to participate in the study. This might hamper potential efforts to increase participation- and intervention uptake rates. As a result, future interventions may need to specifically consider the current workplace culture, company-specific policies regarding absenteeism and sick-leave, and the employee’s own perceived health problems. Careful evaluations incorporating both quantitative and qualitative methods will be needed to document how the web-based employability intervention can be operationalized and potentially adapted for use in other settings. Next, the added value of the intervention should be made clear as the effects might also depend on personal characteristics of the individual user (e.g., healthy or unhealthy behaviour) and on the usage of alternative information and support resources. Expert opinions or other forms of qualitative research on the appropriate size of quantitatively important distinction between baseline and follow-up might be helpful.

Implications for future research

Before conducting a full-scale study, several key elements need to be improved. At this time, the web-based employability intervention was not associated with an economically relevant impact over regular trade union support for a heterogeneous population with work- and disability-related concerns. However, based on the knowledge gained from this pilot economic evaluation, the administration of data collection as well as deciding upon cost categories and outcome measures can be used to direct future research. A web-based employability intervention should attempt to be easily accessible for interested people to ask questions about disability, health, and work. The web-based tool might also benefit from better “targeting” given the heterogeneous population and from revising and enlarging its intensity.

Declaration of interest

This work was financially supported by the Dutch Foundation Instituut Gak. No disclosures to report. The authors report that they have no conflicts of interest.

Acknowledgements

The authors would like to thank Marjet Veldhuis, Jenny Huijs, Catrien Funke, and Kerst Zwart for the data collection, data entry, and supplying the raw data. The authors would like to acknowledge Susan Peters, Senior Research Coordinator at Brisbane Hand and Upper Limb Research Institute, Australia, for assistance with English editing. The Medical Ethics Committee (METC) Noord-Holland was consulted and declared no ethical approval was needed (METC-registration M013-016).

Appendix

Technical appendix

Cost calculation

Questionnaires measuring the use of resources with a three month recall period were provided to the respondents at baseline and 3 months after commencement of the intervention to identify healthcare, patient-and-family and productivity costs (indexed to the reference year 2013).

Health care costs comprised visits for primary, secondary, alternative care, hospitalization, and medication. The Dutch Manual for Costing was used to value health care utilization. For prescribed medication, prices were calculated by first determining the standard daily dose as reported in the Dutch Pharmacotherapeutic Compass and then multiplied the price per dose by the number of days.

Patient-and-family costs included out-of-pocket costs (i.e., travel and parking costs and informal care costs). Travel and parking costs incurred while making use of health services are valued by computing the costs per kilometer to a health service, adding parking costs, for the health care resource used. The reference price for informal care was based on the hourly minimum wage of a housekeeper at 40 h per week from the age of 23 years.

Productivity costs (i.e., costs in other sectors) comprised losses in productivity from paid word (absenteeism) and costs due to reduced productivity while at work (presenteeism). Sick leave hours were derived from the total duration of sick leave days in the previous 3 months by converting them into work-hours equivalents based on the Dutch average of 1540 work hours per year. The human capital theory was used to value productivity losses by multiplying the sick-leave hours by the gender and age-specific productivity levels per paid employee.

Costs of the web-based employability intervention were calculated by using a bottom-up costing approach and included those related to implementing and operating the employability intervention (Appendix Table 1). In depth cost calculation, methods can be obtained by contacting the corresponding author.

Appendix Table 1. Costs (all in EURO) of the web-based employability intervention.

Intervention component	Units	Unit prices	Annual costs (n = 336)	Participant costs
(i) Operational costs website “Wijzer met Welder”
Maintenance ICT	52	45	2340	6.96
Actualization content	52	65	3380	10.06
Content support	260	45	11 700	34.82
Hosting, telephone, etc.	^a	NA	3500	10.42
Handling – administration	52	35	1820	5.42
Customer management	16	80	1280	3.81
Marge	^a	NA	1500	4.46
Total costs per participant per year			25 520	75.95
Total costs per participant – 3-month follow-up				18.99
(ii) Utilization costs WAI-test
Price arrangement (invoice) – license: WAI-test including questionnaires,  updates and protocol	NA	NA	1337	3.98
Total costs per participant per year				3.98
Total costs per participant – 3-month follow-up				1.00
(iii) Feedback coach
Facilitating	10 min	47.7	NA	7.95
Preparation	10 min	47.7	NA	7.95
Conversation	30 min	47.7	NA	23.85
Registration	1 min	47.7	NA	0.795
Total costs per participant per uptake				40.55
(iv)Personal advice
Mailing or advisory telephone line	15 min	47.7	NA	11.925
Registration	5 min	47.7	NA	3.975
Total costs per participant per uptake				15.9
(v)Forum
Moderator	5 h per week	95	24,700	73.51
Total costs per participant per uptake				73.51

NA, not applicable.

a Average estimate.

Outcome measures

Work ability was measured via “The Work Ability Index” (WAI), a questionnaire of seven items assessing the work ability of an individual employee by scoring from 2 (very bad) to 10 (very good). The WAI takes into consideration both the physical and mental demands of the work. The index helps to identify those that need support. A point improvement on the WAI was denoted a substantial improvement and was used to demonstrate the effect for the cost-effectiveness of the intervention. The validity and reliability of the WAI have been assessed in correlation analyses.[28] The WAI and all its items reliably predicted work disability, retirement, and mortality. More recently, the validity and the test–retest reliability of WAI have been studied.[29,30]

Differences in Quality Adjusted Life Years (QALYs) gained during the 3-month follow-up were assessed in the cost utility analysis. A QALY combines survival and utilities into one single measure. Utilities refer to the preference for any particular set of health outcomes and are indicated with a number between 0 (death) and 1 (perfect health). Utilities were calculated using EuroQol-5D-5 L by measuring health outcomes on five dimensions (mobility, self-care, daily activities, pain/discomfort, and depression/anxiety). Each dimension can be rated on five levels ranging from none to major complaints. A crosswalk to the three level value sets was used to calculate the utility weights based on EuroQol scores using Dutch tariff. Utilities at baseline and after 3 months follow-up were used to compute QALYs by means of the area-under-the-curve method. The new version of the EuroQol was tested for face and content validity, suggesting that the EuroQol-5D-5 L yielded good validation evidence and was acceptable to present to respondents.[18]