Opportunities and challenges in designing and evaluating complex multilevel, multi-stakeholder occupational health interventions in practice

ABSTRACT

Extant research suggests the effectiveness of Occupational Health Psychology (OHP) interventions depends on their design in the broader organisational context. While the field recognises that pre- and posttest evaluation do not sufficiently capture the complex dynamics around OHP interventions, complex multi-level OHP interventions are still scarce in the literature. As established intervention implementation frameworks suggest, it remains difficult to address this complexity in practice. The present position paper re-evaluates lessons learned from two complex European OHP intervention projects, by applying the Integrated Process Evaluation Framework (IPEF) and related theories to bridge the gap between the theoretically recognised complexity and practical challenges. The re-evaluations emphasise that programme-multilevel theories rooted in OHP-perspectives contribute to adequately hypothesising around systemic factors and mechanisms relevant to OHP interventions. Concretely, middle range theories that outline how an intervention’s mechanisms work within a specific context to produce certain outcomes are crucial. Additionally, strategically and actively involving key stakeholders at all levels of the system and across the different intervention phases improves the embedding of OHP interventions in organisations. We elaborate on these insights with seven concrete recommendations for complex OHP intervention research.

KEYWORDS:

• OHP complex intervention
• systemic approach
• programme-level theory
• stakeholder
• multilevel (N = 184 words)

Introduction

Well-designed and theory-based Occupational Health Psychology (OHP) interventions are important and useful tools to preserve workers’ health, well-being, and workplace functioning (Beehr, 2019; Burgess et al., 2020). Despite their importance, publications on such interventions that incorporate the systemic complexity, involve all relevant stakeholders, and are rooted in theory remain relatively scarce. This is understandable as the perceived need to quantify and pinpoint the mechanisms by which OHP interventions produce desired effects, favours strictly designed and reductionist approaches in both OHP research and practice (Dooris et al., 2017; Gasparatos et al., 2019). However, such rigid designs cannot account for the complexity of the organisational context in which interventions are implemented (Picco et al., 2022). Consequently, researchers call for centralising the organisational context by implementing a system view that acknowledges potential antecedents, mechanisms, and stakeholders at different levels (De Angelis et al., 2020; Greenhalgh & Papoutsi, 2018; Nielsen & Noblet, 2018; Picco et al., 2022).

Emergent research addresses this call to some extent as it includes broader contextual factors in designing, implementing, and evaluating complex system-based, multilevel OHP interventions (Abildgaard et al., 2016; Heijkants et al., 2022; Mikkelsen et al., 2000; Moore et al., 2019; Nielsen et al., 2018). Here, “multi-level” refers to different interventions that in various combinations span: (a) the micro or individual worker-level, (b) the meso team or leadership level, or (c) the macro or organisational (and even national) level. In complex OHP interventions micro-, meso- as well as macro-level interventions can be implemented simultaneously (Nielsen et al., 2018), resulting in diverse effects or more synergy across the levels. Individual-level interventions tend to lead to short-term effects (Blume et al., 2019), while more systemic changes in organisation-level interventions take more time and resources before effects emerge (Bourbonnais et al., 2011). A literature review by Lamontagne et al. (1990) found that multilevel interventions targeting two or more of the individual, group, leader or organiasational levels have more synergetic effects than interventions targeting only one of these levels.

In line with increased attention to multilevel interventions, the importance of multi-stakeholder perspectives is increasingly recognised as well. Specifically, Nielsen (2017) proposes that complex OHP interventions are – or should be – typically directed at multiple target groups and involve different types of stakeholder groups that can make or break the impact of the OHP intervention over time (e.g. CEO-level, HR departments, employee or employer associations). By involving these different stakeholder groups, a researcher can create shared mental models in the implementation phase, making the transfer of knowledge among stakeholder groups easier. The proactive engagement of these various stakeholder groups seems crucial in designing and implementing effective complex OHP interventions in practice (Nielsen et al., 2010).

Appreciating the importance of multilevel, multi-stakeholder perspectives in OHP intervention research, this paper aims to discuss the main challenges and opportunities of including these in practice. Specifically, it re-evaluates two complex OHP intervention programmes from different countries in healthcare through a process evaluation framework (i.e. Integrated Process Evaluation Framework (IPEF)) (Nielsen et al., 2022) and implementation research theories (i.e. systems and programme multilevel theories). This re-evaluation enables the systematic identification of success factors, practical challenges, and possible pitfalls in implementing these complex interventions.

In this article, we will examine the lessons learned from two larger European intervention programmes that successfully implemented interventions on mental health and wellbeing in healthcare. Concretely, in re-evaluating these two larger intervention programmes, we will address the following questions:

1. How does the published complex OHP intervention programme include quantitative and qualitative evaluation approaches?

2. How does the intervention programme incorporate a multilevel programme-level theory and integrate multiple stakeholders’ perspectives?

3. What is the overlap between the intervention programme and different evaluation stages mentioned in earlier intervention frameworks?

4. What recommendations follow from the OHP intervention programme using earlier intervention frameworks and other relevant theories for new research in our field?

With the aforementioned approach, this paper aims to contribute to the literature on OHP interventions in three ways. First, by integrating lessons learned, the current paper will develop recommendations and concrete guidance for future OHP researchers and interventionists. Second, this paper examines the applicability of intervention evaluation frameworks and theories for moving the OHP intervention field forward and making possible systematic multi-level analyses in the evaluation of the design and implementation process. Third, this paper is intended to form the start of a discussion on handling the complexity surrounding OHP interventions among researchers. Specifically, this paper presents relevant frameworks like the Integrated Process Evaluation Framework (IPEF), Systems Theory, and Programme-Level Theory and offers step-by-step recommendations for future research regarding the design, implementation and examination of complex OHP interventions within specific contexts. We will first describe the relevant intervention evaluation frameworks in more detail.

The Integrated Process Evaluation Framework (IPEF), Systems Theory, and Programme-Level Theory as relevant intervention evaluation frameworks

The Integrated Process Evaluation Framework (IPEF) is based on a systematic review of 39 intervention studies and provides recommendations for designing, implementing, and evaluating OHP interventions throughout the preparation, design, intervention, implementation, and follow-up stages (Nielsen et al., 2022). It incorporates the IGLO (Individual, Group, Leader, Organizational) model (Nielsen et al., 2018) as a stakeholder management strategy and for evaluating OHP interventions at the relevant levels and among stakeholder groups (Havermans et al., 2016; Innstrand & Christensen, 2020; Nielsen et al., 2006; Nielsen et al., 2021; Nielsen & Christensen, 2021). Moreover, the IPEF encourages OHP researchers to utilise valid and reliable quantitative and qualitative measures to study effect and process evaluations across IGLO levels as well as over time.

Importantly, the IPEF includes transfer (i.e. to ensure application of the intervention in day-to-day work practice) components for stakeholders and provides a stepwise procedure (i.e. from the needs-analysis and prioritising in context) for designing OHP interventions in context. The IPEF emphasises the importance of thorough stakeholder mapping (i.e. which agents are relevant in which stages) that recognises organisations as complex adaptive systems (CAS) (Miller & Page, 2007). This is important, as organisations consist of unique interdependent change agents and stakeholders (e.g. employees, managers, directors, and other related stakeholders) that interact to pursue a common overarching and related subgoal(s) (e.g. improve production, population health, customer satisfaction) within a broader dynamic context (e.g. organisational networks, communities, political influences, rules and regulations, and societies) (Carley & Lee, 1998; Rusoja et al., 2018). This also implies that individual employees’ health and wellbeing shapes and is simultaneously shaped by various actors or system key stakeholders as well as macro-, meso- and micro-level factors around them (e.g. Beehr, 2019; Fleuren et al., 2020; Jackson et al., 2014). As IPEF integrates these temporal, multilevel, and multi-stakeholder perspectives around OHP interventions, it arguably represents the most comprehensive OHP intervention protocol to date.

Systems theory viewed from a lens of systemic multilevel perspectives. Relevant elaborations of the systemic multilevel perspective and organisations as complex adaptive systems are provided in system-based and multi-level implementation theories. First, according to Gillespie and Dietz (2009), systems theory depicts organisations as CAS that convert external inputs (e.g. characterised by medical resources, workers or inputs of OHP interventions) into outputs (e.g. products, services, secured intervention as new HR practice) via a “throughput” stage, which includes the organisation's entire operations and activities. To what extent OHP interventions are correctly implemented and secured in existing organisational policies and practices depends on successful throughput by the quality and supply of fitting an OHP intervention to the reported needs of the targeted staff group(s), their interdependent stakeholders, and broader network (e.g. clients, broader stakeholder network) (Burke, 2002). Accordingly, the systemic design perspective (Jones, 2018), suggests that OHP interventions should align with the organisational culture, strategy, structures, policies, and processes, and relevant system key stakeholders (e.g. external governance or employee unions). These parameters form the basis for acceptable behaviours and can instil or control OHP intervention target groups’ trust or openness to the planned OHP intervention (Gillespie & Dietz, 2009). The alignment of OHP intervention(s) to relevant parameters of the organisational system affects the readiness for change of included organisational members (Lundmark et al., 2021). Consequently, it is important to develop co-creative or participatory approaches that integrate social systems principles to better guide stakeholder design for complex systems (Jones, 2018).

Programme-level theory. In evaluating complex OHP interventions it is additionally relevant to consider the notion of a programme-level theory. A programme-level theory includes a description of how and under which conditions a complex OHP intervention produces (Funnell & Rogers, 2011; Moore et al., 2019). Such a theory is important as it explicates a complex interplay of the activities and synergies at, and between, all components in an OHP intervention. Ideally – and inescapably – a good developed programme-level theory is multilevel and distinguishes all key components of the OHP intervention in ways that generate specific hypotheses on how they interact and relate via what underlying mechanisms, and which contextual factors shape the expected relationships. Here, it is important to include potential reciprocal effects (e.g. the context may affect mechanisms, but changes in mechanisms may change the context as well) and a specification of temporality (i.e. some effects can be expected quickly, others take longer). These notions are particularly relevant as OHP interventions typically take place around workplaces with their own social dynamics that unfold over time (van Dijk et al., 2020; Verschuren et al., 2021) and that need to be managed well (e.g. via collaborative and action-learning processes (Wood, 2020)).

Figure 1 presents an example programme-level theory that specifies aspects and mechanisms around an OHP intervention. The programme-level theory includes relevant psychosocial work theories like the job demands-resources model (Bakker & Demerouti, 2014) or the self-determination theory (Deci & Ryan, 2000) to explain underlying psychological mechanisms to explain the beneficial effects of the intervention (for example reducing job demands and increasing job resources).

Figure 1. Example programme-level theory of a hypothetical OHP intervention programme.

The collaborative processes described in Figure 1 (e.g. activities) involve contributions from different forms of knowledge to understand conceptual connections across the work of different stakeholder groups and included intervention elements (e.g. micro-, meso- and macro-level activities can strengthen each other; Moore et al., 2019). Hereby, the collaborative process gains a sense of purpose that aligns the goal of improving well-being and occupational health with the agreed objectives and conditions surrounding the OHP intervention(s). Moreover, clarifications that result from the process and the visual presentation of the programme multilevel theory concerning the expected effects of the OHP intervention(s) can further contribute to actively involving different stakeholder groups. The success of the included OHP intervention(s) can then be evaluated by examining whether theoretical steps happened in practice or relevant psychological mechanisms were tapped and affected through the intervention phases. Furthermore, knowledge can be systematically gathered on whether the intervention programme and its elements worked as expected. A programme-level evaluation framework can then be meaningfully used to monitor the expected steps and knowledge needed for evaluating the overall programme multilevel theory (Moore et al., 2019).

Thus, implementing an OHP intervention within one level can influence and be influenced by other interventions or developments occurring at the same time at other levels (Rivkin & Siggelkow, 2003). Micro-phenomena or so-called micro foundations (Barney & Felin, 2013), such as perceptions of work demands or possibilities for crafting one’s job, are influenced by meso-level factors (e.g. organisation of the workload at the team and organisation level) and macro-level factors (e.g. economic pressures or shortages in the labour market), while macro-phenomena, such as industrial action (e.g. strikes), often emerge through the interactions (and experienced frustrations) at micro (teams) and meso-level (e.g. works council) (Klein & Kozlowski, 2000). System-based views can better recognise the interrelatedness and possible joint or aggregated effects of these micro-, meso- and macro-level factors in creating the impact of implementing OHP interventions in context Barney and Felin (2013).

Furthermore, Nyanchoka et al. (2019) emphasise proactive stakeholders’ involvement in research to understand phenomena or “information gaps” between stakeholders when implementing interventions in organisations. Besides researchers, different audiences including staff, clinicians, policymakers, funders and patients or the public can benefit from understanding research and information gaps, as well as methods or practices on how to identify and display gaps in health research. Based on earlier “multi-stakeholder involvement management” (MSIM) framework research (Waligo et al., 2013), the integration of strategic level stakeholders’ involvement is important in the development and implementation of OHP interventions in three ways to: (1) ensure that attention to the topic of occupational health is sufficiently present at all included levels, (2), developing ways to collaborate and communicate among stakeholders, and (3), to establish sound management for adequate monitoring of the quality of collaboration with stakeholders and intervention implementation and evaluation. Thus, proactive stakeholder mapping and involvement strategies are important in the IPEF suggested phases of intervention design and evaluation.

With regard to specific stakeholders, Helland et al. (2021) have discussed the crucial role of line managers and safety representatives in implementing OHP interventions. Further, Hasson et al. (2014) discussed the important role line managers, senior managers, and HR professionals can play in the preparation, implementation and evaluation phases to better integrate the successful interventions in existing policies and practices.

These diverse stakeholder groups need to be engaged in a participatory way (Abildgaard et al., 2016) or to be informed about the contents of the complex OHP intervention programme during different implementation phases (from the preparation to the start phase, during the implementation process, and in the evaluation phase) to make the implementation process successful across time. Multiple benefits of a participatory process have been identified (Nielsen & Christensen, 2021). First, the process creates employees’ buy-in and ownership. Second, it makes use of workers’ expertise of which demands and resources need to change and it enables workers to make sense of the rationale behind the process. Third, it optimises the fit with the organisational context, and it facilitates the participatory process (Nielsen & Randall, 2013). Fourth, it enables dialogue between managers and workers about what changes need to be made (Christensen et al., 2019). Finally, the dual perspective on both negative and positive aspects of the work environment encourages a balanced understanding of the environment (Christensen et al., 2019).

In sum, the aforementioned frameworks and theories, combined with the explicit inclusion of systems thinking and an explicated programme-level theory, are useful in designing and evaluating complex OHP interventions. To illustrate this combination of perspectives and generate learnings for the development of OHP interventions, the next section considers two OHP intervention programmes. Specifically, this exercise aims to arrive at balanced recommendations that enable gathering high-quality scientific evidence around OHP interventions that simultaneously appreciate and address the complex multilevel and multi-stakeholder context in which they are situated.

Evaluating two OHP intervention programmes

The two intervention projects considered from the aforementioned perspectives are both situated in healthcare. The complexity of interventions in this area stems from the high-stakes and high-constraints situations occurring in this context (e.g. complex care tasks for patients with severe conditions etc.; De Lange, Løvseth, Teoh et al., 2020), and make it a particularly relevant sector to consider OHP interventions in. Additionally, this sector faces extensive OHP-relevant labour market challenges (e.g. labour shortage, excessive demands, deterioration of staff health and wellbeing; Broetje et al., 2020; Hodkinson et al., 2022; Teoh et al., 2023a). Meta-analytical evidence highlights the need for a multilevel perspective as organisational-level interventions (e.g. workload reductions, shift rotations) show larger effect sizes in the reduction of the level of burnout symptoms among healthcare professionals than individual-level interventions (De Simone et al., 2021; Panagioti et al., 2017). Although individual-level interventions (e.g. stress management, mindfulness) show promise in helping healthcare professionals’ wellbeing (Boet et al., 2023; De Simone et al., 2021), they also have been criticised for holding the individual accountable for system-level shortcomings (Bal et al., 2020; Shanafelt et al., 2019).

Both intervention studies were discussed at the 2022 European Academy of Occupational Health Psychology Small Group Meeting, and reflect a range of European countries, vary in their approach to multilevel and multi-stakeholder engagement, and fit differently to the IPEF framework and other relevant theories – allowing for reflection on recommendations for new OHP research in this field.

Intervention 1: the Dutch Healthy Healthcare project: improving the work ability of healthcare workers (The Netherlands)

Background. This multilevel and multi-stakeholder intervention project (De Lange, Pak, Van Vuuren et al., 2020; De Lange, Pak, Osagie et al., 2020) involved 23 different Dutch healthcare institutions, aimed to create a community of practice for balancing staff, patients, and system requirements and to improve reported work ability of employees. Different psychological and system-based theories were used in the development of the empirical data cycle as well as the action-learning cycle for HR professionals. Two qualitatively different interventions were implemented in the project that involved stakeholders at the individual- (micro; i.e. employee), organisation- (macro; i.e. Human Resources), and leader- (meso; i.e. leadership) levels. The first individual-level intervention was training aimed at improving and sustaining employees’ willingness and ability to work by developing their self-leadership competencies. The intervention included group workshops and weekly e-learning modules with exercises on self-leadership (Neck & Manz, 2011), proactive problem-solving (Covey, 1989), positive self-talk (Seligman, 2012) and strength-based thinking and exercises (Linley & Harrington, 2006). The second intervention aimed to develop an organisation-level HR strategy to sustain work ability (i.e. the organisation level), and to improve leaders’ knowledge and skills about sustainable work ability. In the identification and preparation phase, intervention partners used needs-analysis interviews with HR-professionals and desk-research. This participatory approach allowed organisations to adjust the programmes to their own specific developmental needs.

Evaluation. Effect and process evaluations were conducted in both interventions. For effect evaluation, both interventions included a control and experimental (i.e. training) group, resulting in a quasi-experimental complete panel-design (N = 1,478). The self-leadership training was associated with increases in participants’ work ability, firm-internal employability, and vitality. This change was mainly due to improved self-observation skills among participants. No significant effects were found for the top-down leadership intervention. For the process evaluation, one HR manager from each healthcare institution completed a survey and attended a focus group, although no quantitative data was captured. All were positive about participation in the project and saw progress within their organisation concerning the knowledge available on sustainable work ability. There were differences in the recommendations the HR managers formulated for increasing the impact of the two interventions.

Lessons learnt. There was considerable overlap between the IPEF and the evaluation procedures used in the intervention project. In line with the IPEF, the project included all intervention phases, qualitative and quantitative measures, a control versus experimental group during different intervention phases, a needs-analysis, and implementation process evaluation. This resulted in meaningful learnings across the IGLO levels. In terms of programme-level theory, the job demands-resources model (Bakker & Demerouti, 2014) was used to hypothesise expected relations between work exposure and occupational health. Furthermore, psychological mechanisms and theories related to self-intervention informed effect testing and explained the well-being and work-ability improvement across time. Theory also formulated expectations about the intervention implementation process, with personal communication among stakeholders, managerial and financial support, and co-creative design in the intervention design key factors to implement during the intervention.

The project highlighted the importance of paying more explicit attention to stakeholders across different levels of the system(s); e.g. teams (i.e. group), organisation (i.e. organisation) and regional activities with the employer association (i.e. omnibus). Through a better stakeholder involvement strategy, different gaps could be identified among stakeholders (e.g. communication, information, skills or learning goal gaps, possible funding and relational gaps) and monitored across time. This encouraged employees and leaders to participate in interventions, and to develop clearer feedback (action) learning cycles during the implementation process. Furthermore, it enhanced motivation among relevant stakeholders at different levels in the organisation, as well as at the regional or employer association level. This was especially important for the interventions that were created bottom-up (was in this case, the self-leadership training), which are based on free choice participation.

At the leader level, process evaluation confirmed management support as a crucial stakeholder involvement factor for intervention success and to prevent distributive gaps between stakeholders. Leader support was pivotal to maintaining the finances and personnel for project management. Personnel changes reported during the implementation phase resulted in difficulties (e.g. team leaders, HR professionals, at the employer association), affecting the progress and impact of the leadership intervention. All this is important to adjust the intervention to the context and to maintain its sustainability.

This intervention shows how programme-level theory can be used to further theorise about and test possible developmental changes in outcomes among micro-, meso- as well as macro-level system key stakeholders. Adding to IPEF, programme-level theory to hypothesise and test for possible underlying psychological as well as implementation process mechanisms is crucial for developing, designing, and evaluating multilevel OHP interventions. For example, psychological mechanisms and theories related to self-intervention informed the effect testing and of the intervention and explained the well-being and work ability improvement across time. Similarly, adequate stakeholder mapping in the first phase of the IPEF is arguably best followed-up with stakeholder mapping and proactive stakeholder involvement strategies in all IPEF intervention phases (e.g. including qualitative and quantitative measures among key stakeholders on information gaps). Finally, action learning cycles that, for example, examine IGLO learnings among system stakeholders at all levels in all intervention phases can be used to improve the implementation process and effects of the OHP intervention over time. Challenges in this project were mainly changes among the included HR-professionals within context (due to sickness absence, turnover, et cetera), resulting in different pitfalls during the implementation phase (i.e. changed planning, lack of communication among stakeholders, lack of integrating the self-leadership training in existing policies to lack of organising organisational support for the training, et cetera).

Intervention 2: multilevel interventions to promote mental health in SMEs and public workplaces: the H-Work project (Europe)

Background. The H-WORK project (https://h-work.eu/), a Horizon 2020-funded project, aimed to promote mental health in Small and Medium sized Enterprises (SMEs) and public workplaces through multilevel interventions (De Angelis et al., 2020). These workplaces were identified as being particularly vulnerable since they often have a lack of resources to manage workers’ mental health and psychosocial well-being. Altogether, 18 interventions were conducted at 11 different intervention sites in Italy, Spain, The Netherlands, Germany, and the Czech Republic between 2020 and 2022. Nine interventions were conducted at the individual level, three at the group level, four interventions at the leader level, and two at the organisational level.

The project’s focus was on designing, implementing, and validating effective multilevel assessment and intervention toolkits, and to evaluate individual and organisational outcomes of the adopted measures. These toolkits help guide end-users (i.e. managers, practitioners, and stakeholders) to: (i) assess psychosocial risk factors in specific work environments, (ii) decide which interventions to implement, and (iii) evaluate the intervention process and outcomes in terms of increased workplace psychological well-being and cost-effectiveness. An e-learning course has also been developed to facilitate leaders and HR-personnel in promoting mental health and well-being in the workplace.

The H-WORK programme was designed as a multilevel approach using the IGLO model (Nielsen et al., 2018). In addition, the H-WORK programme relied on a participatory bottom-up approach involving employees in identifying problem sources and their potential solutions (Nielsen & Christensen, 2021). Finally, building on the Job Demands–Resources model (Bakker & Demerouti, 2014), the H-WORK programme targeted both the health impairment and health promotion processes in the psychosocial work environment.

Evaluation. The evaluation process looked at both the intervention and the implementation process. First, needs analyses, conducted through the H-WORK Assessment Tool (HAT) – a context measure – provided information about both objective measures as well as employees’ perception of programmes, policies, and practices regarding mental health and well-being. To explore needs at different levels among different stakeholders, focus groups with employees were conducted to identify suggestions for areas that needed improvement regarding mental health at work. Interviews explored middle and senior managers’ perspectives and priorities concerning mental health at work. These results were then presented at a stakeholder meeting to develop, prioritise, and decide upon an action plan and choice of multilevel interventions for improving mental health and well-being in the workplace.

Second, effect, cost-effectiveness, and qualitative and quantitative process evaluation were conducted to measure if and why the interventions worked or not. Quantitative surveys collected distal and proximal measures at baseline, post, and follow-up (six months after implementation). The distal outcome measures represented indirect and causally distant outcomes of the H-WORK interventions. These were used across all intervention sites and were independent of the interventions implemented. The proximal measures represented the direct and causally close outcomes of each intervention and were therefore dependent on the specific interventions implemented. Objective data (e.g. sickness absence, turnover) collected at intervention sites formed the basis for effectiveness and economic evaluations, including the intervention’s value for money and affordability.

Finally, to understand what works, for whom, and under which circumstances, process data were collected using the H-Work Evaluation Tool (HET) every three months. Consistent with the IPEF, which was developed as part of the H-WORK project (Nielsen et al., 2022), different constructs were measured throughout the different intervention process phases depending on the activities in each phase of the intervention. In addition, qualitative interviews were conducted among the different stakeholders participating in the project concerning a more in-depth understanding of the context and the mechanisms leading to the outcome.

Applying this to a specific intervention site for illustration purposes, the needs analyses in a participating Italian hospital achieved three main goals: (i) identifying job demands and resources at different IGLO levels; (ii) identifying similarities and differences across hierarchical positions and departments; and (iii) developing an action plan to enhance workplace mental health within the targeted organisational contexts (Giusino, De Angelis, Mazzetti, Christensen, et al., 2022b; Giusino, De Angelis, Mazzetti, Faiulo, et al., 2022). Through the exercise, workers and their managers provided suggestions, including training programmes (on e.g. job-related techniques, emotional distress), developing improved career development opportunities, refining job rotation plans, revising organisational management structures, and providing psychological support services (Giusino, De Angelis, Mazzetti, Christensen, et al., 2022b).

Lessons learnt. The H-WORK project actually formed the basis of the IPEF, with the Job Demands-Resources model (Bakker & Demerouti, 2014) as the underlying programme-level theory as the basis for their project. The project highlights the necessity of applying a participatory approach, both in the identification of needs and fitting the interventions to system key stakeholders’ needs. Since only the wearer knows where the shoe pinches, future research should identify and include different system key stakeholders across different levels of the organisation and ensure that interventions are congruent to the organisation's needs (Nielsen & Christensen, 2021). Another important aspect to consider for future research is the importance of anchoring the project and process in the management of the organisation and the willingness of the team leaders to motivate their work teams to participate. An important take-home message is to create a communication plan, use it actively and have a clear strategic plan for the whole intervention process, starting with the initial planning and recruitment and following the screening, the action planning, implementation, and evaluation.

The project showed that demands and resources exist at all IGLO-levels, highlighting the need for interventions at multiple levels, providing a more holistic and effective approach to tackling mental health issues at work. Again, pointing to the relevance of adding an overall programme-level theory and methodological recommendations to include more proactive system key stakeholder (co-creative) management and engagement (i.e. action learning on IGLO) strategies across the intervention implementation phases.

Discussion

The present position paper aimed to summarise learnings from re-evaluating two complex European OHP interventions in healthcare in terms of similarities, practical challenges, and possible pitfalls in implementing OHP interventions within practice. The two selected OHP intervention projects offer important insights for using the IPEF model and other systematic and multilevel programme theories in practice. Specifically, the selected interventions advocate a systemic participatory approach in examining and linking micro-, meso- and macro-level factors in designing, implementing, and evaluating OHP multilevel interventions to improve workers’ health and well-being and their aggregated effects on other levels of the system. The most notable lessons learned are (a) the importance of psychological theorising in developing testable underlying programme-level theories, including stepwise procedures, participatory or systemic co-creative approaches in designing and evaluating complex OHP interventions, and (b) including quantitative as well as qualitative data (system inclusive) measures, (action) learning, and feedback loops for different types of key stakeholders. Here, OHP is not only well-placed to embrace interventions from a micro to a macro-perspective, but equally offers considerable expertise and insights to advance the concept further. Through this all, we outline seven recommendations for developing new future OHP intervention research and practice.

1. Embrace a systemic approach by including programme multilevel theory. The two intervention projects considered here reveal that the use of programme multilevel theories would facilitate a multilevel intervention design. Specifically, multilevel theorising enables guided predictions regarding the mechanisms and the expected effects of an intervention programme. This should be done at all relevant levels and should include the social dynamics within and across organisations. Here, both negative intervention undermining and positive intervention supporting social dynamics need to be considered by the programme theory. By using a systematic programme multilevel theory that maps relevant dynamics, crucial aspects are less likely to be overlooked.

2. Use system key stakeholder mapping and include specific stakeholder involvement strategies in all intervention phases. More attention can be given to system key stakeholder involvement and communication during the intervention phases. From the example projects, we found that researchers should create a strategy for including relevant stakeholders at the system level in the intervention development as well as implementation phases. Key system stakeholders could be better informed in understanding the aims, consequences, and context-related necessities for making interventions work in practice, as well as their own potential role in the development and success of an intervention (Skivington et al., 2021). Additionally, interventions can cause a “ripple effect” (i.e. a series of events in a system, resulting in the evolution of new structures of interactions and new shared meanings; Jagosh et al., 2015), where current alignment of elements in the system may have to be re-aligned because of the changes introduced by the intervention (Lundmark et al., 2021).

In practice, successfully achieving a system fitting intervention requires contextualising it through proactive stakeholders’ engagement in both the design and implementation phases (Von Thiele Schwarz et al., 2016). For interventions to be aligned across system levels and functions, considering for whom and what is affected by the change is crucial at the initial stages of the intervention process. Therefore, potential hindrances to change (e.g. lack of fit with associated procedures and goals) can be reduced, and the intervention can be better integrated within the relevant context, increasing chances of reaching outcomes and sustaining the intervention. The three stages for engaging stakeholders from the “multi-stakeholder involvement management framework” (Waligo et al., 2013): (i) scene-setting, (ii) recognising stakeholder involvement capacity, and (iii) stakeholder relationship management, can be useful to support the implementation phases of the intervention.

Intervention Mapping (Bartholomew-Eldredge et al., 2016), as used in planning health promotion programmes, could be relevant as it includes constructing theories of change for all relevant stakeholders. These theories of change delineate the desired behaviours for the main target group as well as relevant agents in and outside the organisation and connect them to their most important psychological determinants. This facilitates connecting specific psychological determinants to intervention strategies based on available scientific evidence.

(3)

Plan for working with complex multilevel data but being able to stay pragmatic in practice. Our review of the two interventions suggests embracing different research methodologies, paradigms and system-based measures across the intervention phases as important. While it is now easier to carry out multilevel analyses, there are nevertheless limitations that need to be accounted for. The first of these is the identification of appropriate data. Data from other levels are inherently more distal (e.g. macro-level topics like the influence of organisational practices), which leads to further time lag, additional confounding factors, and the need for strong theoretical grounding to justify proposed pathways (Teoh et al., 2022). Second, the use of organisational data is implicitly seen as better, given that they are more likely to represent metrics of interest to the organisation and its stakeholders while also overcoming the limitations of self-reported measures (Haeffel & Howard, 2010). However, organisational data, especially when routinely collected, presents its own challenges, where poor collection, disingenuous reporting, and data fabrication and manipulation undermine the validity of the data collected (Edmondson, 2004; Mears, 2014; Teoh et al., 2022). This links with the difficulty of using what are known as flawed, uncertain, proximate, and sparse (FUPS) datasets (Wolpert & Rutter, 2018), where researchers should first acknowledge where data may have characteristics of FUPS; be transparent in the entire data preparation and analysis process; and to draw on other data sources to triangulate data and interpretations made.

(4)

Balancing organisational and intervention demands. Many interventions arise from the need for immediate action to address an issue of concern. This is especially true when the situation is acute and asks for a quick response or solutions. This often clashes with the desire for researchers to take time to plan for a study, incorporate elements such as thorough needs-analyses, control, or comparison groups, administer multiple survey instruments, and engage stakeholders within context. Within a time-pressured complex system, the question is: how do we ensure high-quality designs that can be anchored in already available and existing policies, intervention programmes and practices within context? Here, the IPEF offers a flexible guide to pick out measures across the different intervention stages that might be most readily available or important to that context. The interventions reviewed also reveal that researchers should be pragmatic in their choices, meeting the demands of the context. It also shows researchers and organisations what might be missing within their intervention designs that could potentially be captured as part of a formal process evaluation.

(5)

Embracing new paradigms in research and practice. Beyond methods, the move from designs that emphasise structure (e.g. randomised control trials) to more flexible systemic co-creative design approaches (including intervention implementation process evaluation) represents a move from a more positivist to a more realist epistemology, opening to other research paradigms would encourage different perspectives and methods (Jones, 2018; Porter, 2015). For example, a more critical paradigm could challenge the neoliberal ideology that underpins much of society or the dominance of new public management (Simonet, 2014) that arguably places the individual’s and society’s interests over organisational goals, such as productivity and profitability (Bal et al., 2020; Roper et al., 2022). Such a perspective would account for a number of the other limitations highlighted in this paper and could be particularly salient in the healthcare sector where changes to the system could be to the benefit of the worker and patient as individuals, rather than to benefit the wider organisation.

(6)

Results obtained from different contexts and levels are most useful for learning. First, future research can further acknowledge the context of the complex OHP intervention(s) and raise awareness of all the factors that play a role in the direct actual and subjective environment of the intervention. Second, future research can apply the programme multilevel theory in the design and evaluation of OHP intervention(s). Third, incorporation of all relevant stakeholders at different intervention levels (i.e. micro-, meso- or macro-level) and in different phases (i.e. preparation and identification of intervention, design, implementation and securing it in existing policies) is crucial to create and sustain meaningful effects of the interventions. Researchers may also use the MSIM-framework, the framework for evaluating complex interventions of the Medical Research Council guidance (Skivington et al., 2021) that emphasises the importance of stakeholders’ involvement to ensure attention to multi-layer stakeholders at different stages of the intervention implementation. Fourth, future research can use the identification of key uncertainties that make the implementation of the OHP intervention(s) problematic (i.e. resistance among target groups or stakeholders) so these potential barriers can be addressed adequately across time. Fifth, the economic business case of the planned intervention and follow-up activities must be examined in more detail. In other words, as an international community of researchers we can share valuable insights across different contexts and use relevant implementation evaluation frameworks and theories in publishing our work.

(7)

Capture more the diversity of the studied workforce, context, and its included vulnerable stakeholders. This paper focuses on well-educated healthcare workers, and the H-work project also includes workers in SMEs and public organisations. Nonetheless, new intervention programmes can further examine the possible aggregated effects of OHP interventions among different types of target groups, as well as recognising the challenges of engaging underrepresented groups into the intervention process (like lower-educated workers; Damen et al., 2024; Hussein, 2022). In doing so, future OHP interventions should also address the workers’ perception of being treated as mere objects by the organisation (Nussbaum, 1995). Workers’ objectification has shown to be triggered by a focus on economic issues and asymmetrical power relationships (Baldissari et al., 2022) and to be associated with a decrease in workers’ well-being and performance (e.g. Correia, 2023). Given that organisations face increasingly economic constraints (e.g. Bååthe et al., 2022 for healthcare services) and some of its workers may have positions of low power, workers’ objectification might be an important risk factor for employees that should be considered in interventions.

Focusing on healthcare, across different international countries and regions, there is substantial variation in how healthcare systems and infrastructure are set up and funded; how workforces are trained and recruited; and how national welfare, social security, and public health measures are implemented (Nielsen et al., 2020; Wendt, 2009). To fully recognise the wider systems approach, more international perspectives and examinations are needed to incorporate these. Cross-comparison of studies across international nations also offers opportunities to examine factors at the omnibus macro-levels, such as how other legislation and guidance (e.g. work-related stress), that vary in implementation and enforcement (Jain et al., 2022). Equally, interventions should consider where it may be appropriate to generate activities to influence the omnibus context, such as lobbying regulators, policymakers, and governments (Teoh et al., 2023b).

In sum, re-evaluating international case studies of complex OHP multilevel interventions and reflecting on lessons learned helps develop our understanding further in theorising about, designing and evaluating complex multilevel, multi-stakeholder OHP interventions and to create a meaningful body of knowledge across time.

Conclusion

In this paper we provided an overview of similarities, practical challenges, and possible pitfalls in implementing complex OHP interventions within practice. Building on the experiences obtained in two meaningful case studies, we formulated seven recommendations regarding the design, implementation and evaluation of complex OHP interventions within practice that can be used by researchers as well as intervention partners in the OHP field.

Acknowledgements

In the development of this position paper, we acknowledge the European Academy of Occupational Health Psychology (www.eaohp.org) and connected researchers for making a small group meeting possible in 2022 in Nijmegen. Moreover, we would like to thank an anonymous reviewer for providing constructive and valuable feedback to an earlier draft of this paper. The fruitful discussions during this meeting have helped to develop this position paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).