Validation of the Intervention Preparedness Tool: a short measure to assess important precursors for successful implementation of organisational interventions

ABSTRACT

Process evaluation enables us to gain insights into the complex organisational intervention processes, but has mostly taken place post intervention, thus failing to support implementation. Using the theory of planned behaviour, we developed and validated a 7-item process evaluation questionnaire (the Intervention Preparedness Tool) that aims to evaluate the preparatory phases of the intervention and may be used to optimise the intervention process. The study was conducted in two Italian hospitals participating in an organisational intervention (N = 1,654 healthcare workers).

We conducted exploratory factor analysis (EFA) on one half of the sample and cross-validated the best factor structure identified through confirmatory factor analysis (CFA) on the other half of the sample. Results showed that the Intervention Preparedness Tool composed of 7 items has a three-factor structure (readiness for change, intervention-context fit, and communication). To perform nomological validation, we correlated the Intervention Preparedness Tool with seven psychosocial working conditions (demand, control, peer support, supervisor support, roles, relationships, changes) and job satisfaction. We found that the dimensions of the Intervention Preparedness Tool were significantly related to these working conditions and job satisfaction suggesting that the organisational context may be related to participants’ appraisals of the early phases of participatory interventions.

KEYWORDS:

• Organisational intervention
• validation
• evaluation
• process evaluation
• healthcare
• Intervention Preparedness Tool

Introduction

Organisational interventions can be defined as “planned, behavioural, theory-based actions that aim to improve employee health and well-being through changing the way work is designed, organised and managed” (Nielsen, 2013, p. 1030), i.e. through making changes to work policies, practices and procedures. As organisational interventions address the causes of poor health and wellbeing, they are generally recommended when aiming to improve employee well-being (ETUC, 2004; EU-OSHA, 2010; ILO, 2001). Reviews and meta-analyses of the effectiveness of organisational interventions, however, have shown inconsistent results indicating that organisational interventions do not always achieve their intended outcomes (Bambra et al., 2009; Montano et al., 2014; Richardson & Rothstein, 2008). Nielsen and Miraglia (2017) argued that due to the complexity of organisational interventions, traditional evaluation methods such as the randomised, controlled trial are not suitable on their own. They suggested that evaluation of organisational interventions should include process evaluation (Nielsen & Miraglia, 2017), i.e. the evaluation of “individual, collective and management perceptions and actions in implementing any intervention and their influence on the overall result of the intervention” (Nytrø et al., 2000, p. 214).

Process evaluation offers the opportunity to gain insight into the complex processes of organisational interventions and may be used to develop supportive activities to ensure the successful implementation of the intervention (Nielsen & Randall, 2013). A recent literature review revealed that most process evaluation takes place post intervention to understand how the processes influenced the intervention’s outcomes, thus failing to use process evaluation as a means to optimise intervention implementation (Nielsen et al., 2023). In the present paper, we validate a short process evaluation questionnaire, the Intervention Preparedness Tool (IPT) that aims to evaluate the preparatory phases of the intervention and which may be used to optimise the intervention process.

The main contributions of our study are twofold. First, we add to the literature on quantitative process evaluation, which has been criticised for failing to meet the demand for rigorous validation (Nielsen et al., 2023) and thus failing to ensure the psychometric properties of scales used (Hinkin, 1998). Second, existing measures have failed to capture the contextual factors influencing the subsequent phases of the intervention (Nielsen et al., 2023). Even readiness for change and previous history of interventions, which forms part of the Intervention Process Measure (IPM) (Randall et al., 2009), have only been captured post-intervention. Process evaluation can also be used to implement supportive activities, i.e. interventions to support the subsequent phases of the intervention (Nielsen & Randall, 2013; Nielsen, 2013), however, this use requires data to be collected in the earlier intervention phase.

In this study, we present the IPT, which captures three key elements of the intervention preparation process and we test its reliability, discriminant, convergent and nomological validity. The IPT has been developed to be included in the baseline questionnaire of organisational interventions. Such inclusion has two advantages. First, feedback on the results may facilitate the development of supportive interventions to optimise implementation and second, recency bias may be reduced (Kalm & Norris, 2018).

Measuring intervention preparedness

Organisational interventions are typically implemented using a problem solving cycle approach (Nielsen et al., 2010). In the first phase, the preparatory phase, a steering group is set up, the goals and vision of the intervention are agreed and a communication strategy to support the process is developed. What happens in this phase may influence participants’ engagement with the intervention in later phases, thus ultimately influencing the intervention’s outcomes (Nielsen & Abildgaard, 2013). The theory of planned behaviour (TPB; Ajzen, 1991) may be a useful lens to understand which elements need to be measured at the early phases of the intervention to anticipate how workers may engage in the later phases of the intervention. In essence, TPB focuses on the factors that influence an individual’s intention to enact a certain behaviour, in our case, engage with the intervention and its activities (Ajzen, 1991). According to TPB (Ajzen, 1991), three key underlying factors shape intentions. First, attitudes towards the behaviour/intervention, i.e. whether participants have a positive appraisal of the intervention. Second, subjective norms refer to whether participants feel social pressure to engage with the behaviour/intervention and third, perceived behavioural control (PBC) refers to the extent to which participants feel they have a level of control over the behaviour/intervention. We did not measure TPB directly but followed the recommendation of Ajzen (2015) to tailor measures capturing TPB in the context it is used. We therefore developed measures of attitudes, subjective norms and perceived behavioural control as they would be translated into the mechanisms in the early phases of the intervention process which may influence participants’ engagement in the subsequent phases of the intervention. Mechanisms in organisational intervention research are the reasonings and reactions of participants in response to the stimuli they encounter (Nielsen & Miraglia, 2017). In the present paper, we validate three key elements of the preparatory phase, which we suggest shape participants’ intentions to engage with the intervention process and its activities. We suggest that measuring these in the early phases of the intervention can function as a temperature check of whether supportive activities are needed to enhance participants’ intentions and motivation to engage in the intervention and its activities.

Readiness for change concerns the extent to which participants welcome the intervention and the planned changes and perceive they can use the intervention to improve their working conditions (Weiner et al., 2008). Readiness for change is thus about positive appraisals or attitudes towards the intervention. Participants who favourably evaluate the planned intervention and the changes that are planned as part of the intervention, or in other words, have a positive attitude towards the intervention will be more likely to engage in the intervention’s later activities. Readiness for change also taps into theory of planned behaviours’ element of perceived behavioural control. If participants believe they can use the intervention to improve their own working conditions, a specific form of PBC directly related to control over the course of the intervention, then they may be more motivated to engage in the intervention’s activities.

Randall et al. (2009) included a measure of readiness for change in their process evaluation questionnaire, however, included it at the follow-up 18 months later. To optimise the use of this measure it should be included in the baseline survey. We use the same measure as Randall et al. (2009) as it demonstrated good psychometric properties and was found to be related to self-efficacy and well-being post-intervention. Previous research has also employed this measure. Augustsson et al. (2015) found that low levels of readiness for change meant the intervention was not implemented according to plan. Schelvis et al. (2016) found that readiness for change decreased over time as workers felt disappointed by the intervention. Neither Augustsson et al. (2015) nor Schelvis et al. (2016) tested the reliability or validity of the measure.

Building on principles of person-environment fit (P-E fit), i.e. the compatibility between an individual and their environment (Kristof-Brown et al., 2005), Nielsen and Randall (2015) proposed that context-intervention fit is important, i.e. that the intervention needs to be tailored to and integrated into the organisational context in which the intervention takes place. When the intervention is integrated into and aligned with organisational objectives and context, the intervention is more likely to achieve its outcomes and therefore fit needs to be addressed in the preparation phase (Nielsen & Randall, 2015). Two elements of fit could be important in the preparatory phases of the intervention. First, it has been suggested that the intervention must be aligned with the organisation’s goals (von Thiele Schwarz et al., 2021). Alignment with the organisation’s goals serves several purposes. Workers may see alignment as PBC as they can see how the intervention may help them achieve organisational goals and do their jobs in a better way. Alignment may also help reduce the risk of unintended effects as alignment implies analysis of how the intervention may impact other key operations within the organisation (Bamberger et al., 2016). Thus alignment may be linked to positive attitudes towards the intervention as it may minimise workers’ fears of unintended effects.

Second, another important element of context-intervention fit is whether the intervention is perceived to be useful in addressing the challenges the organisation is facing. Aligning this type of fit with TPB’s dimension of attitudes (Ajzen, 1991), fit may be an important precursor of participants’ intention to engage with the intervention. If participants have a positive appraisal of how the intervention is aligned with the organisation’s goals, they may be more likely to engage in the intervention and its activities. The perceived usefulness of the intervention may also reflect BPC as workers believe the intervention may be useful to minimise the challenges the organisation faces. If workers believe they exert control over changes introduced to make them work to their advantage, then they are more likely to engage in the intervention’s subsequent phases.

Previous research has demonstrated the importance of fit. Using qualitative methods, Biron et al. (2011) found that line managers felt that surveying workers was inappropriate as they felt that they themselves were in greater need of intervention. As a result, line managers did not distribute the survey to their workers. In a qualitative study, Nielsen et al. (2014) found that using a tailored questionnaire facilitated sensemaking of results and enabled the development of detailed actions. Measuring fit at follow-up, Lundmark et al. (2018) found that workers who reported that the intervention corresponded to their needs for change and they saw benefit of the changes introduced reported higher levels of intrinsic motivation.

Communication may also be an important precursor shaping participants’ engagement with the subsequent intervention phases. Communication about the intervention may enhance participants’ understanding of the rationale behind the intervention, facilitate sensemaking and help ensure buy-in and engagement (Nielsen et al., 2021; Nytrø et al., 2000). Communication plans should be developed on how to inform participants about the interventions’ aims and objectives and the type of involvement required by participants at different phases of intervention, e.g. completing a survey or participating in action planning workshops (Nielsen & Noblet, 2018). Communication links into TPB’s subjective norms (Ajzen, 1991) as an important precursor of participants’ intentions to engage with the intervention. Subjective norms refer to whether individuals believe others want them to perform a certain behaviour and whether one’s network performs the behaviour. If leaders and worker representatives communicate what is expected of participants at different phases of the project, e.g. completing a questionnaire, then participants may be more likely to succumb to these expectations and behave accordingly, i.e. engage with the intervention. Having received information about the goals of the project links into attitudes towards the interventions. If workers understand the aims and objectives of the interventions; how the intervention may be of benefit to them, they are more likely to have a positive attitude towards the intervention and be more motivated to engage with the intervention. Communication about what is expected of participants may also link into PBC: participants may feel control over what they need to do as part of the intervention and may also feel in control if it is communicated to them how they can exert influence, e.g. by voicing concerns in the screening phase or participation in action planning workshops.

Post-intervention process evaluation has found that communication was related to increased autonomy and job satisfaction (Bakhuys Roozeboom et al., 2020) and provided clarity of the roles and responsibilities of the intervention and led to the intervention achieving its intended outcomes (Augustsson et al., 2015). Schelvis et al. (2016) included a range of single item measures about communication at different phases of the intervention, including communication at the start up, about survey results, and about content and progress of action plans. Schelvis et al. (2016) found that while communication was successful in both intervention sites in the first phases, communication differed as the project progressed and one site failed to communicate effectively in the action planning phase.

Hypothesis 1: The IPT will have a three-factor structure, i.e. readiness for change, intervention-context fit, and communication.

Key to construct validation is the identification of the probable links between constructs of interest and measures of other constructs, i.e. to test the constructs’ nomological validity (Schwab, 1980). Translating this to the context of organisational interventions, Nielsen and Miraglia (2017) suggested that pre-exiting working conditions may be closely related whether intervention processes are implemented successfully. The constructs in the IPT can be seen as the working mechanisms of the first phase of the intervention. If workers are ready for change, feel the intervention is well aligned with the organisational goals and that communication about the intervention has been good, these are the mechanisms that may trigger participants’ engagement in later phases of the intervention, however, contextual factors may either hinder or facilitate that these mechanisms are triggered (Nielsen & Miraglia, 2017).

As an early test of the nomological network that the IPT may for part of, we tested whether the IPT was related to the seven dimensions of the HSE Indicator tool (HSEIT; Edwards et al., 2008) and job satisfaction. Previous studies have found that job satisfaction, control (Nielsen & Randall, 2013), role clarity, peer social support (Nielsen & Randall, 2009) and leadership (Lundmark et al., 2017) predicted later intervention processes.

The job demands-resources model (JD-R; Demerouti et al., 2001) and conservation of resources theory (COR, Hobfoll, 1989) may be useful in predicting the relationships between working conditions, job satisfaction and the IPT. The JD-R suggests that job demands require sustained effort or skills and if not managed may lead to a depletion of resources (Demerouti et al., 2001). Job demands may thus be negatively related to the dimensions of the IPT as workers who experience high job demands may feel drained and thus less prepared to engage with the intervention, they may react negatively to any communication and may not feel able to use the intervention to their benefit. Job resources stimulate growth and may lead to resources caravans, whereby workers may invest resources to gain additional resources (Demerouti et al., 2001; Hobfoll, 1989). Relationships, peer and supervisor support, role clarity and control are all job resources (Nielsen et al., 2017). If workers perceive these resources are present then they are more likely to see the intervention as an opportunity to accumulate additional resources (Hobfoll, 1989). For example, if workers feel they already have high levels of control, they may be more confident they can use the intervention to their advantage, i.e. they will be ready for change. If workers perceive they have good peer and supervisor support they are more likely to feel they have received the necessary information about the intervention. If workers experience role clarity, they may feel confident that they know what are expected of them as part of the intervention and they may feel more ready. Furthermore, if workers perceive that changes in general are managed well, they are more likely to be ready for the changes the particular intervention will bring about. Therefore, workers may feel that also in connection with this intervention have they received good communication and that the intervention aligns with the organisation’s goals. As a recent review revealed that only few studies have explored how the context may influence intervention processes and none have tested the nomological validity of process mechanisms (Nielsen et al., 2023), we only hypothesise the direction of the relationships and not the strengths of relationships between the HSEIT and IPT.

Hypothesis 2: The dimensions of the IPT will be negatively correlated with job demands, and positively correlated with control, peer support, supervisor support, role, positive relationships at work, change and job satisfaction.

Methods

Procedure and participants

The sample consisted of healthcare workers employed in two Italian hospitals undergoing an organisational intervention. We collected data on the processes of the preparation phase in the next phase of the organisational intervention, the screening phase, in the baseline survey. Questionnaires were distributed to 6,687 staff and 1,905 questionnaires were returned, yielding a response rate of 29%. Almost three-quarters (74.2%) were females, and 25.8% men. The majority of workers were aged between 31 and 50 (48.7%), and almost the entire sample was composed of Italian workers (98.9%). In terms of occupational roles, 79.7% were healthcare professionals, while the remaining 20.3% held administrative positions. As regards to work contracts, 93.2% of workers had a permanent contract, followed by fixed term (4.7%) and interim contracts (1.2%). A total of 52.1% of workers were employed in shift-work, and 54.5% of these worked both in day and night shifts. Finally, in terms of average job tenure, participants worked in the same unit for 128.96 months (SD = 114.56), and in the same company for 207.12 months (SD = 139.39).

We assessed data for missing values, univariate and multivariate normality. Participants who left the survey without answering any question about context indicators were excluded from the analysis (N = 251). The remaining sample (N = 1,654) were evaluated for missing data at item-level to see if absent responses were systematic (Tabachnick & Fidell, 2007). For the purpose, the Missing Value Analysis (MVA) function of SPSS was applied to assess Little’s test of missing data using estimated means (Little, 1988). Missing values percentages were very low, ranging from 0.1–0.4%, and Little’s Test was non-significant (χ² = 46.22, df = 47, p > .05), suggesting that data were missing completely at random (MCAR). These results supported the adoption of a Listwise method of deletion.

The participatory organisational intervention

The two hospitals were involved in a participatory organisational intervention using the INAIL’s approach to managing psychosocial working conditions. The INAIL methodology has been developed to enable organisations to comply with the national legal requirement to assess and manage psychosocial risks along with health and safety risks in the workplace. After a first phase of preparation aiming to set up the organisational intervention, the second and third phases (screening) assess psychosocial risks through objective and verifiable indicators of work-related stress through a checklist (Barbaranelli et al., 2018), and by surveying workers’ perceptions of psychosocial working conditions using the Italian version of HSEIT (Rondinone et al., 2013; Wood et al., 2019). The screening phase is followed by action planning identifying actions to improve working conditions based on the screening results. This process is cyclical and requires the evaluation of the process effectiveness before starting a new assessment, thus recently a fifth phase has been included aiming to examine if the intervention works (Di Tecco et al., 2015; Di Tecco et al., 2020).

Development of the IPT scales

We followed the guidelines recommended by Hinkin (1998) in developing the three scales. In a review of the intervention implementation literature, Nielsen (2023) identified communication and intervention-context fit as key underlying principles and readiness for change as particularly important in the preparation phase. She argued that in the earlier phases of intervention, workers need to be informed about the intervention and its processes to ensure buy-in and participation in later phases and that the intervention should be aligned with the goals of the organisation. Based on this review, we identified three potential factors: readiness for change, communication and intervention-context fit. We then reviewed Nielsen et al. (2023), a systematic literature review on the quantitative measures used in multi-level interventions and focused on the measures used in interventions at the organisational level. We were unable to find suitable, validated measures of fit and communication. We included the existing scale of Randall et al. (2009) to measure Readiness for change, reformulating it to reflect the inclusion in the baseline questionnaire (sample item: “I have high expectations that the intervention will improve my working conditions”). Next, we selected items for the communication and the intervention-context fit based on the existing literature (a sample item for communication is: “It has been clearly communicated what is expected of me in the project” and a sample item for intervention-context fit is: “It is clear to me how the intervention is related to the organisation’s overall goals”). Statements were answered on a 5-point Likert-type agreement response scales from 1 = strongly disagree, 5 = strongly agree.

Measures to test nomological validity

Psychosocial risks and job satisfaction were measured at baseline along with the baseline IPT. The seven dimensions of psychosocial risks were measured by 35 items from the Italian version of the HSEIT (Rondinone et al., 2013; Wood et al., 2019).

Demands explores issues around workload, work patterns and work environment (8 items; α = .85; a sample item is: “I have unachievable deadlines”).

Control concerns the control workers can exercise over their own work activities (6 items; α = .82; a sample item is: “I have a choice in deciding how I do my work”).

Supervisor support addresses aspects of supervisors’ behaviours such as encouragement and support to workers (5 items; α = .87; a sample item is: “I am given supportive feedback on the work I do”).

Peer support addresses the encouragement and support provided by colleagues (4 items, α = .86; a sample item is: “I get the help and support I need from colleagues”).

Positive relationships at work captures the perceptions of interpersonal conflict at work (4 items; α = .77; sample item “Relationships at work are strained”). The scale was reversed such that a high score signifies positive relationships.

Roles captures the understanding of the worker’s own role in avoiding role conflict (5 items; α = .80; sample item “I am clear what my duties and responsibilities are”).

Change addresses the organisational change processes and how changes are communicated, (3 items; α = .78; sample item “I have sufficient opportunities to question managers about change at work”). According with the original measure (Edwards et al., 2008; HSE, 2019), workers were asked to answer statements on 5-point Likert-type response scales using two alternative response formats: frequency ones using from 1 = never to 5 = always and agreement ones from 1 = strongly disagree, 5 = strongly agree.

Finally, we included a five-item measure of job satisfaction as a measure of employee general satisfaction with the job (Hackman & Oldham, 1974). An example of an item is: “Generally speaking, I’m very satisfied with my job” (α = .67). Statements were answered on a 5-point Likert-type agreement response scales from 1 = strongly disagree to 5 = strongly agree.

Data analysis

First, we provided descriptive statistics for the IPT in terms of item-level and scale-level arithmetic means, standard deviations (SD) and medians to obtain some preliminary information on participants’ responses. We checked univariate normality by inspecting skewness and kurtosis scores, while multivariate normality was investigated through Mardia’s multivariate test (Mardia, 1970). To analyse the factorial validity of the IPT, we randomly split the sample in two parts using IBM SPSS 23 facilities. Independent t-tests and chi-square tests showed no significant differences between the two split samples.

To test for Hypothesis 1, we tested the dimensionality of the IPT in two ways. First, we performed exploratory factor analysis (EFA) using Oblimin oblique rotation on one half of the sample. Second, we cross-validated the best factor structure identified through confirmatory factor analysis (CFA) on the other half of the sample. This was made since using both EFA and CFA for tool validation can determine more accurate measurement (Hinkin, 1998). EFA and CFA were performed using Mplus 8.2 (Muthén & Muthén, 2017).

Prior to EFA, we examined sample adequacy through the Kaiser-Meyer-Olkin (KMO) test and the Bartlett’s test of Sphericity. We developed the IPT to consist of three different domains and therefore we assessed and compared models from 1 to 3 factors to determine which was best-suited. Factor retention was guided by the following model fit indices: Comparative Fit Index (CFI), Tucker and Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA) and Standardised Root Mean Square Residual (SRMR). We also reported Chi-square test of model fit, however, since Chi-square is strongly affected by large sample size and model complexity, we mainly referred to the four indicators of goodness-of-fit already specified (Garrido et al., 2016). We adopted the following criteria to assess model fit: CFI ≥ 0.90, TLI ≥ 0.90, RMSEA ≤ 0.08, SRMR ≤ 0.06 (Hu & Bentler, 1999). Moreover, rotated factor loadings were assessed to determine if an item should be removed. In specific, we decided a priori to drop items with factor loadings < .40, or > .40 on more than one factor to avoid cross-loadings (Field, 2013).

We tested the best solution identified through EFA with a CFA using the second sub-sample. We assessed the CFI, TLI, RMSEA and SRMR to examine the model fit. Internal consistency reliability was assessed using Cronbach’s alpha (minimum level accepted: α = .70; Nunnally, 1978) and Spearman-Brown coefficient (ρ), which is the recommended measure for two-item scales (Eisinga et al., 2013). Moreover, we computed Composite Reliability (CR), which describes the extent to which latent construct items share the measurement of a construct, and Average Variance Extracted (AVE), which measures the level of variance captured by a construct versus the level due to measurement error (Fornell & Larcker, 1981). Values for CR and AVE greater than > .60 and > .50, respectively, are considered acceptable (Bagozzi & Yi, 1988; Hair et al., 2010). Based on the assumption that appraisals of interventions are complex and multifaceted (Nielsen et al., 2007), we expected to identify different factors that would exhibit discriminant validity. As a check of the discriminant validity of the factors, we performed a CFA model with a single factor which was then compared with our final model.

We used the overall sample to test nomological validity (Hypothesis 2) by computing the correlations between each IPT scale and the seven psychosocial working conditions and job satisfaction. Finally, we statistically compared the correlations between each working condition and job satisfaction and the different IPT subscales using the Steiger’s (1980) Z test as implemented in the cocor R package (Diedenhofen & Musch, 2015).

Results

Sensitivity of the scales

Item means ranged from 2.89 to 3.70 (Table 1); the majority were around the mid-point (5-point Likert scale). The most positive evaluations were found for items on intervention-context fit (3.26 out of 5), while the most negative evaluations, on average, were found for items on communication (2.89 out of 5). Measures of kurtosis and skewness were used to determine if items and scales met univariate normality assumptions (Kline, 2005). All the items exhibited non-significant skewness and kurtosis, except for one item from the readiness for change dimension (kurtosis = 1.57). Moreover, Mardia’s test suggested that multivariate normality was not reached (multivariate kurtosis: 119.25, p < . 001). Consequently, the Robust Maximum Likelihood estimator (MLR) included in Mplus 8.2 software was used for EFA and CFA to take into account multivariate nonnormality.

Table 1. Item-level and scale-level statistics.

			Item-level statistics				Scale-level statistics
	Item code		Mean (SD)	Skew	Kurt	Median	Mean (SD)	Skew	Kurt
Readiness for change	Rfc1	I feel confident that I can use the intervention to improve my working conditions	3.13 (1.04)	−0.44	−0.41	3	3.25 (0.82)	−0.36	0.02
	Rfc2	I have high expectations that the intervention will improve my working conditions	2.99 (1.04)	−0.23	−0.46	3
	Rfc3	I look forward to the changes brought about by the intervention	3.19 (1.06)	−0.44	−0.43	3
	Rfc4	I am ready to accept the changes brought about by the intervention	3.70 (0.79)	−0.81	1.57	4
Communication	Co1	It has been clearly communicated what is expected of me in the project	2.89 (1.04)	−0.13	−0.62	3	2.89 (1.00)	−0.18	−0.57
	Co2	I have received information about the goals of the project	2.89 (1.06)	−0.17	−0.75	3
Intervention-context fit	Icf1	It is clear to me how the intervention is related to the organisation’s overall goals	3.29 (0.90)	−0.56	0.20	3	3.26 (0.80)	−0.54	0.43
	Icf2	The intervention is relevant for solving important problems in my organisation	3.24 (0.92)	−0.53	0.23	3

Note: Skew = skewness; kurt = kurtosis.

Exploratory factor analysis

Prior to EFA, we performed the KMO test and the Bartlett’s test of Sphericity to investigate if the items in the IPT could be represented in a simpler structure. The results stated the database suitability for factor analysis (KMO = .85; Bartlett: χ²= 4057.33, df = 28, p < .001). Testing Hypothesis 1, a first EFA showed that one item from the readiness for change dimension (“I am ready to accept the changes brought about by the intervention”) had a factor loading < .40. Therefore, the second EFA was performed after deleting this item. As shown in Table 2, fit indices clearly suggested that the three-factor EFA model, compared to the one-factor and the two-factor solutions, represented the best fit for the data, CFI = 1.00, TLI = 0.99, RMSEA = 0.017, SRMR = 0.003. Factor 1 (“Readiness for change”), namely the extent to which participants are confident that their engagement in the intervention can lead to improvements in working conditions, consisted of 3 items. Factor 2 (“Communication”), describing the extent to which the intervention characteristics have been communicated to workers, consisted of 2 items. Finally, Factor 3 (“Intervention-context fit”), namely the extent to which the intervention fits with the goals of the organisations involved, consisted of the remaining 2 items. All the primary factor loadings were adequate (> .40); moreover, no further cross-loadings were found (Table 3). All inter-factor correlations exceed .32, ranging from .42 to .60; this confirmed the suitability of the oblique rotation over the orthogonal one (Tabachnick & Fidell, 2012). The EFA thus supported our first hypothesis that there are three factors in the IPT.

Table 2. EFA and CFA models comparison.

EFA – Models	χ^2	df	p	CFI	TLI	RMSEA	SRMR
1-factor model	552.94	14	<.001	0.71	0.57	0.217	0.108
2-factor model	77.31	8	<.001	0.96	0.90	0.103	0.038
3-factor model	3.74	3	>.05	1.00	0.99	0.017	0.003
CFA – Models	χ^2	df	p	CFI	TLI	RMSEA	SRMR
1-factor model	594.24	14	<.001	0.73	0.60	0.335	0.091
3-factor model	117.28	11	<.001	0.95	0.91	0.108	0.040
3-factor model*	56.86	10	<.001	0.98	0.96	0.076	0.030

Note: * = with modification.

Table 3. Factor structure for the tool.

	Subsample 1 (EFA; N = 821)					Subsample 2 (CFA; N = 821)
Scale and constituent items	Item-total correlations	h^2	1	2	3	1	2	3
Readiness for change
I feel confident that I can use the intervention to improve my working conditions	.76	.72	.74	.09	.09	.92
I have high expectations that the intervention will improve my working conditions	.70	.82	.92	.02	−.05	.82
I look forward to the changes brought about by the intervention	.71	.76	.90	−.07	.01	.78
Communication
It has been clearly communicated what is expected of me in the project	.69	.79	.02	.87	.02		.92
I have received information about the goals of the project	.67	.86	−.03	.94	.00		.90
Intervention-organisation Fit
It is clear to me how the intervention is related to the organisation’s overall goals	.66	.42	.33	.01	.42			.74
The intervention is relevant for solving important problems in my organisation	.67	.84	−.02	.02	.92			.81

Note: h² = Communalities.

Confirmatory factor analysis, internal consistency and construct validity

In an additional test of Hypothesis 1, we performed a CFA with MLR estimator to test the three-factor solution suggested by EFA. Initial assessment of the three-factor model showed that it could be improved, CFI = 0.95, TLI = 0.91, RMSEA = 0.108, SRMR = 0.040. Consequently, by inspecting modification indices and items with similar wording, the errors of two items of the readiness for change subscale were allowed to correlate (“I have high expectations that the intervention will improve my working conditions” and “I look forward to the changes brought about by the intervention”; r = .47; p < .001). This modification resulted in an improvement of the model, which now showed adequate fit, CFI = 0.98, TLI = 0.96, RMSEA = 0.076, SRMR = 0.030. The factor loadings were all higher than .74, ranging from .74 to .92 (Table 3). Internal consistency was adequate for all the scales (readiness for change: α = .91; communication: ρ = .90; intervention-context fit: ρ = .75). Simultaneously, all factors obtained adequate values of CR (readiness for change = .88; communication = .90; intervention-context fit = .75) and AVE (readiness for change = .71; communication = .82; intervention-context fit = .61), providing a confirmation of convergent validity. The inter-correlations between the factors were generally high (Cohen, 1988), thus indicating a solid stem common to the three dimensions: the highest correlation was between readiness for change and intervention-context fit (r = .82, p < .001), followed by the correlation between intervention-context fit and communication (r = .67, p < .001) and between communication and readiness for change (r = .64, p < .001). However, the model where all items were set to load on one factor generated a much poorer fit to the data than the three-factor solution (CFI = 0.73, TLI = 0.60, RMSEA = 0.335, SRMR = 0.091), with a Satorra-Bentler Δχ²=  178.48,df = 1, p < .001. This result, together with the absence of significant cross-loadings, provided a confirmation of discriminant validity among the factors.

Relationships between IPT, psychosocial working conditions and job satisfaction

Testing Hypothesis 2, Table 4 shows the intercorrelations between the IPT, psychosocial working conditions and job satisfaction. The results confirmed Hypothesis 2, as all correlations between the three IPT indicators, the seven working conditions and job satisfaction were in the expected direction and statistically significant. In detail, we found that demands were especially correlated with communication (r = −.15, p < .001) and readiness for change (r = −.13, p < .001). Among resources, supervisor support (r = .32, p < .001), positive relationships at work (r = .18, p < .001) and change (r = .35, p < .001) were more strongly correlated with communication, while peer support was similarly correlated with intervention-context fit (r = .19, p < .001), communication (r = .20, p < .001) and readiness for change (r = .21, p < .001). Analogously, control (r = .16, p < .001) and role (r = .26, p < .001) were mainly associated with communication, although these values were not significantly different from the correlation coefficients with intervention-context fit. Finally, job satisfaction was especially correlated with readiness for change (r = .20, p < .001) and communication (r = .18, p < . 001).

Table 4. Correlations between psychosocial working conditions, job satisfaction and IPT scales (Listwise; N = 1569).

	Intervention-context Fit	Communication	Readiness for change
Demands	−.08^bc	−.15^a	−.13^a
Control	.14	.16^c	.10^b
Peer support	.19	.20	.21
Supervisor support	.24^b	.32^ac	.26^b
Positive relationships at work	.09^b	.18^ac	.11^b
Role	.25	.26^c	.21^b
Change	.28^b	.35^ac	.29^b
Job satisfaction	.13^bc	.18^a	.20^a

Note: All correlations significant at p < .01. ^{a (b) (c)} = this correlation differs significantly from the corresponding correlation of fit (comm) (read) with this concept.

As additional analysis, we investigated the relative importance of the seven working conditions in affecting the IPT dimensions using Dominance Analysis (DA; Groemping, 2006). Further information can be found in FigShare (10.6084/m9.figshare.23759037).

Discussion

Based on TPB (Ajzen, 1991) and reviews of the organisational intervention literature (Nielsen, 2023; Nielsen et al., 2023), we proposed three factors in the preparation phase of organisational interventions, which may influence the subsequent phases of the intervention, namely readiness for change, intervention-context fit and communication. We collected information about these three constructs in the screening phase. We tested the reliability, discriminant, convergent and nomological validity of these three factors. We found support for Hypothesis 1 that our items captured three distinct processes. Unlike Randall et al. (2009) who found support for a four-item readiness for change scale (measured at follow-up), our EFA showed that one item “I am ready to accept the changes brought about by the intervention”, did not fit well with the scale. One possible explanation is that the three other items more clearly capture proactive attitudes, and therefore may be more important at the preparatory phase of the intervention as they are more reflective of planned behaviour.

We also found support for our second Hypothesis. The seven dimensions of the HSEIT and job satisfaction were significantly associated with the three dimensions of the IPT. As expected, demands were negatively related to the dimensions of the IPT, while resources and job satisfaction were positively related to the dimensions of the IPT. These correlations support the assumption that our constructs are part of a wider nomological network. We found communication was more significantly correlated with demands than intervention-context fit, with control, roles and change than readiness for change, with supervisor support and relationships than the other two dimensions of the IPT, and job satisfaction than intervention-context fit. Furthermore, readiness for change was more significantly related to demands and job satisfaction than intervention-context fit. The results offer a nomological perspective that may facilitate the development of hypotheses about which working conditions in the organisational context may be important when planning the intervention process. For example, if demands are high, and resources such as social support, roles and responsibilities are low and there is generally poor communication about change then we would expect workers to also report poor communication about the intervention. In such context, more comprehensive communication plans should be developed.

Implications for research and practice

Our results have important implications for research and practice. We did not find support for the four-item measure of readiness for change. Our study calls for further tests of the reliability and validity of the scales when included at baseline. Our results suggest that three constructs in the IPT should be included in the baseline survey of organisational interventions.

Although previous studies (e.g. Augustsson et al., 2015; Schelvis et al., 2016) included some of these three dimensions in their baseline survey, none of these reported developing supportive activities to ensure a smooth intervention process in the subsequent phases. Results should be fed back to organisations and steering groups to enable them to develop supportive activities to ensure subsequent intervention phases run smoothly. If participants report not being ready for change, not being clear about their role in the intervention and do not see how the intervention is related to the organisation’s goals, then supportive activities should be initiated. The steering group could review whether the communication strategy ensures sufficient information is communicated, whether the means of communication are appropriate, e.g. if workers do not read corporate emails, updates at team meetings may be more effective. If workers do not feel confident how they can best make use of the intervention, psychoeducation could be needed for workers to learn about psychosocial working conditions and how these may be addressed through making changes to the way work is organised, designed, and managed.

Strengths and limitations

The main strength of our study is the rigorous validation in a large sample across two hospitals. Our study, however, is not without its limitations. First, we based the development of our IPT on reviews of the existing literature (Nielsen, 2023; Nielsen et al., 2023). Conducting interviews in the participating hospitals may have revealed other factors important in the preparation phase, however, several reviews have found these three factors to be important (Nielsen & Noblet, 2018; Nielsen, 2023; Nielsen et al., 2023).

Second, we only collated information in the baseline survey on the contextual factors which may influence these early intervention mechanisms, we are thus unable to test the predictive validity of the IPT on future intervention processes, e.g. whether good communication leads to participants completing questionnaires or participating in action planning workshops. As argued by Ajzen (2015), we would not necessarily expect a strong relationship between the IPT and engagement in intervention activities in the later phases. The IPT should be used to identify supportive activities and these activities is believed to lead to increased engagement. For example, if participants felt communication was poor at the early phase of the intervention, then improving this communication should motivate participants to engage with the intervention activities at the later stage.

Third, our communication and intervention-context measures only contained two items. As the intention is to include these measures in the baseline screening survey, we decided to include as few items as possible. The baseline survey already measures demographics, psychosocial working conditions and wellbeing outcomes and it is therefore crucial that the IPT is as short as possible, while still capturing key elements of the process.

Finally, we only tested the nomological validity of the IPT in relations to the seven dimensions of the HSEIT and job satisfaction. We chose these dimensions as they represent generic working conditions that are supposed to be important across a range of sectors and occupations (Edwards et al., 2008) and because there is some support in the intervention literature that some of these may influence processes in the later stages of the intervention (Lundmark et al., 2018; Nielsen et al., 2010; Nielsen & Randall, 2009). Cronbach and Meehl (1955) suggested that in the early history of a construct, the network will be limited and future research should explore the relationships with other contextual factors. The Workplace Integrated Safety and Health assessment tool aimed at managers identify a range of contextual factors around existing procedures for managing employee health and well-being, e.g. leadership commitment to managing health and safety, adherence to health and safety legislation and involving workers in health and safety initiatives (Sorensen et al., 2018). These factors may all influence workers’ readiness for change, the appraisals of communication and the extent to which they feel the intervention is aligned with the organisation’s goals.

Conclusion

The main contributions of this paper are that we developed and validated the IPT for evaluating key elements of the preparation phase of organisational interventions. The literature has found that three elements, readiness for change, communication and intervention-context fit, play a key role in shaping the intervention, however, these have most often been measured using single items post intervention. We conducted a rigorous test of the reliability and validity of the three measures and found support that these are three distinct scales and they interact in a nomological network with psychosocial working conditions and job satisfaction. It is our hope that IPT can be used to develop supportive activities to ensure the subsequent phases of the organisational intervention run smoothly thus increasing the chances that the intervention succeeds.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was funded by the National Workers Compensation Authority under the grant BRIC 2020-2022- ID n. 20 / 2019