Construct Validity and Test–Retest Reliability of Questionnaires to Assess Workplace Violence Risk Factors (QAWRF): A Tripartite Perspective of Worksite-Specific Determinants in Healthcare Settings

Abstract

Introduction

Among available workplace violence (WPV) interventions, only data-driven, worksite-based, and risk-based approach WPV interventions had moderate evidence for effectiveness in decreasing the risk of WPV. The Questionnaires to Assess Workplace Violence Risk Factors (QAWRF) had been previously developed to determine the level of WPV risk factors in each healthcare setting based on the tripartite perspective of key stakeholders to enable effective WPV interventions. This study aimed to determine the construct validity and test–retest validity of QAWRF.

Methods

QAWRF, a three-component instrument consisting of QAWRF-Administrators, QAWRF-Workers, and QAWRF-Clients, had previously undergone content validation, face validation, and internal consistency reliability testing. 965 respondents were recruited to examine the construct validity of QAWRF, and a subset of these (n = 90) were retested again at an interval of three weeks to assess its test–retest reliability. Confirmatory factor analysis (CFA) was performed, and fitness indices, average variance extracted, correlation coefficient, composite reliability, and intraclass correlation coefficient were determined.

Results

QAWRF-Administrator, QAWRF-Worker, and QAWRF-Client had acceptable factor loadings (≥0.6), absolute fit (Root Mean Square Error of Approximation > 0.1), incremental fit (Confirmatory Fit Index and Tucker Lewis Index > 0.9), parsimonious fit (Chi-square/degree of freedom < 5), correlation coefficient between construct (≤0.85), discriminant validity index, and construct reliability (≥0.6). CFA supported a four-factor model for QAWRF-Administrator and QAWRF-Worker, and a two-factor model for QAWRF-Client.

Conclusion

QAWRF holds good construct validity and test–retest reliability. By using QAWRF, healthcare managers can identify specific WPV risk factors that are perceived by stakeholders as prevalent at a particular workplace, and these findings can contribute towards data-driven, worksite-specific, and targeted WPV interventions in healthcare settings that are expected to be resource-efficient and more effective than general WPV interventions.

Keywords:

• workplace violence
• risk factors
• risk assessment
• questionnaire
• instrument
• tool
• validation
• healthcare workers

Introduction

Workplace violence (WPV) encompasses acts or threats of violence directed towards individuals while they are at work or on duty, which range from verbal abuse to physical assaults.¹ Among various professions, healthcare workers (HCWs) face a significant WPV risk, with a 20% higher overall prevalence than all other occupations.² The most common form of WPV experienced by HCWs is Type 2 WPV, where patients or their dissatisfied family members are the perpetrators.^3–5 Indeed, a previous meta-analysis reported a pooled one-year prevalence of 19.3% for Type 2 WPV among HCWs, which indicates that approximately one in every five HCWs experience WPV initiated by patients or visitors annually.⁶ The pervasiveness of WPV among HCWs is deeply concerning due to its potential adverse impacts. This includes individual effects such as physical ill health, depression, anxiety, post-traumatic stress disorder, and financial instability,^3,7–11 as well as organisational effects such as prolonged work absenteeism, subpar work performance, reduced quality of care, burnout, and job dissatisfaction.⁷

Given that WPV incidences are commonly experienced by HCWs with detrimental effects on their health, well-being, and work functioning,⁷ there is an urgent need to develop effective WPV interventions to prevent these incidences. Previous systematic review has indicated that among available WPV interventions, only data-driven, worksite-based, and risk-based approach WPV interventions had moderate evidence for effectiveness in decreasing the risk of WPV.¹² Furthermore, given that WPV risk factors might vary from setting to setting, current recommendations to address WPV include incorporating stakeholders at the regional or local level and taking into account their aggregate perspectives to promote better planning of WPV interventions.¹³

At present, workplaces currently collect information to identify possible WPV risk factors through records analysis, job hazard analysis, and specified WPV program checklist.² Although these conventional WPV risk factors identification methods are helpful, they require training and may be time-consuming.¹⁴ On the other hand, the available survey questionnaires have some limitations as they either only focused on non-modifiable WPV risk factors such as the sociodemographic profile of workers, had insufficient validation, or used open-ended questions that may be difficult to interpret and score. Moreover, existing survey questionnaires mainly explore workers’ perspectives and rarely consider clients’ or administrators’ perspectives on the WPV incident.

Consequently, the Questionnaires to Assess Workplace Violence Risk Factors (QAWRF), an instrument that considers the tripartite perspectives of worksite-specific WPV risk factors in healthcare settings, was developed and validated as an initiative to address WPV among HCWs.¹⁵ The QAWRF is a three-component set of structured questionnaires designed to obtain information on the possible risk factors for WPV in any given healthcare setting based on the perspective of three main healthcare stakeholders: administrators, workers, and clients. QAWRF enables WPV risk factors to be rated using scales according to the perceptions of three main healthcare stakeholders.¹⁵ Using input from QAWRF, healthcare management can then develop targeted WPV interventions based on the WPV risk factors identified as prevalent in their healthcare settings.

The QAWRF was developed based on the Chappell and Di Martino’s Interactive Model of Workplace Violence.⁸ In this model, the risk factors for WPV are multifactorial and can be divided into the individual risk factors of the victims and the perpetrators, workplace risk factors, and the wider contextual and societal risk factors that indirectly affect WPV. These categories, with wider contextual and societal risk factors combined as one domain, formed the domains for QAWRF.¹⁵ The items for each domain were extracted from a systematic review of WPV risk factors, which were then grouped under the suitable domain as suggested by the Chappell and Di Martino’s WPV model and previous literature.¹⁵

Initial validation suggests that the QAWRF has satisfactory psychometric indices for content validity, face validity, and internal consistency reliability.¹⁵ To ensure a valid, reliable, and informed measure of WPV risk factors, further testing to assess the factor structure and test–retest reliability is required. Construct validity measures how well an instrument measures the trait or theoretical construct it intends to measure,¹⁶ whereas the test–retest reliability is important to demonstrate that an instrument is stable over time.¹⁶ Therefore, the present study aimed to determine the construct validity and the test–retest reliability for all QAWRF components, namely the QAWRF-Administrator, QAWRF-Worker, and QAWRF-Client.

Materials and Methods

Ethical Approval

Prior to the conduct of this study, ethical clearance was obtained from the Medical Research & Ethics Committee [NMMR-21-537-58890 (IIR)] and Universiti Teknologi MARA Research Ethics Committee [REC/07/2021 (MR/629)].

Study Population

A validation study was conducted among administrators, workers, and clients of 39 government outpatient and health clinics in Terengganu, a state in the eastern part of Peninsular Malaysia. Terengganu consists of eight districts in which there were 52 health clinics as well as six hospitals offering inpatient and outpatient clinical services.¹⁷ The inclusion and exclusion criteria for administrators, workers, and clients are outlined in Table 1. Individuals with past psychiatric history were screened by asking all potential respondents whether they had been diagnosed with any psychiatric disorder prior to enrolment into the study.

Table 1 Inclusion and Exclusion Criteria of the Study

QAWRF Component	Inclusion Criteria	Exclusion Criteria
Administrator	Individuals who were involved in administrative work at a healthcare department, including head of departments, head of medical officers, head of nurses, and head of paramedics. Able to read, write, and understand the Malay language.	Individuals with working experience of fewer than six months at the healthcare department during sample recruitment. Individuals with past psychiatric history.
Worker	Individuals who were involved clinically with clients, including doctors, paramedics, and nurses. Able to read, write, and understand the Malay language.	Individuals with working experience of fewer than six months at the healthcare department during sample recruitment. Individuals with past psychiatric history.
Client	Patients, relatives, and/or individuals who accompanied a patient to a healthcare department and had completed the visit to the health department. Age 18 years old and above. Able to read, write and understand the Malay language.	Intoxicated at the time of study conduct Individuals with past psychiatric history. Individuals who came to the healthcare department but did not meet any clinical healthcare personnel during the visit (eg, came to renew medication).

Sample Size

At the end of the initial validation study of the QAWRF, the domains and items of the QAWRF components were as follows: four domains and 45 items in QAWRF-Administrator, four domains and 43 items in QAWRF-Worker, and two domains and 14 items in QAWRF-Client (Table 2).¹⁵ Each domain had more than three items. Based on these numbers and considering recommendations from previous literature collectively,^18–24 a sample size of 300 respondents for each QAWRF component was determined to be the minimum sample size required to run confirmatory factor analysis (CFA) for each QAWRF component questionnaire. Meanwhile, for test–retest reliability, the sample size was calculated using Arifin et al online sample size calculator.²⁵ The parameters used for the calculation include an expected intraclass correlation coefficient (ICC) of 0.80, a minimum acceptable ICC of 0.5, a significance level α of 0.05 for a two-tailed test, a power of 80%, and a repetition per subject of 2. Based on the calculation, the minimum number of respondents required to determine the test–retest reliability of QAWRF was 28 respondents for each QAWRF component.

Table 2 Domains and Items in QAWRF at the End of Initial Validation Study

QAWRF Component	Domain	Item	Total Items
QAWRF-Administrator	Client	c1, c2, c3, c4, c5, c6, c7, c8, c9, c10	10
	Worker	w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12, w13, w14, w15, w16	16
	Workplace	p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13	13
	Societal	s1, s2, s3, s4, s5, s6	6
QAWRF-Worker	Client	c1, c2, c3, c4, c5, c6, c7, c8, c9, c10	10
	Worker	w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12, w13, w14, w15, w16	16
	Workplace	p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12	12
	Societal	s1, s2, s3, s4, s5	5
QAWRF-Client	Worker	w1, w2, w3, w4, w5, w6	6
	Workplace	p1, p2, p3, p4, p5, p6, p7, p8	8

Sampling Mechanism and Conduct of Study

Two-stage sampling was used to sample respondents for CFA. In the first stage, cluster random sampling was conducted in which three out of eight districts in Terengganu state, Malaysia were randomly selected using a random number generator. There were a total of 39 government outpatient and health clinics in the three selected districts. In the second stage, respondents from government hospital outpatient clinics and health clinics in the three selected districts were selected via systematic random sampling for workers and clients and universal sampling for administrators. For workers, firstly, the interval number for respondent selection was calculated separately for each clinic depending on the number of workers available at the clinic and the desired sample size of the clinic. Workers were then systematically selected from the worker’s name list, whereby the starting number for worker selection was generated using a random number generator. For clients, firstly, the interval number for respondent selection was calculated by taking the total number of drug prescriptions collected at the clinic’s pharmacy the week before to serve as a proxy for the client’s weekly population size at the clinic. Clients were then systematically selected from the client’s waiting list at the pharmacy, whereby the starting number for client selection was generated using a random number generator. Finally, for administrators, universal sampling was performed where all eligible administrators were invited to join the study to ensure a sufficient sample due to limited number of administrators available at the study sites.

Following the selection of respondents, they were invited to participate in the study and provided written consent. For respondents comprising of administrators and workers, links to online versions (Google Form) of QAWRF-administrator and QAWRF-worker were provided via WhatsApp Messenger application. All items requiring validation in the online questionnaire were set as required to avoid missing data. Respondents were given two weeks to complete the questionnaire, with a reminder given after one week. For respondents comprising of clients, they were approached by the researcher and invited to participate in the study. After providing written consent, they were asked to complete the QAWRF-Client and to return it to the researcher upon completion. The researcher would then briefly screen through the questionnaire to ensure respondents complete responses, whereby any unanswered items were verified with the respective respondents and proceeded to place the completed questionnaire in a brown opaque envelope. All respondents were given a small gift as a token of appreciation post-completion of the questionnaire.

A subset of respondents (n = 90) who completed the QAWRF during the CFA stage were recruited again for a retest at an interval of three weeks. These respondents were given a link for an online questionnaire and asked to complete it within one day. The researcher sent a reminder through the WhatsApp messenger application after one day. If no response was received after two days, a phone call was made to the respective respondent to remind the respondent again.

Study Instruments

The QAWRF is a WPV instrument developed in the Malay language and comprises of QAWRF-Administrator, QAWRF-Worker, and QAWRF-Client.¹⁵ The Malay language was used to ensure a better understanding of the questionnaires among the target population. All items in the QAWRF were rated using a 5-point Likert scale, with 1 being “highly not apparent at the facility” and 5 being “highly apparent at the facility”. A sociodemographic section was added to each QAWRF component.

Statistical Analysis

Data obtained was compiled in IBM SPSS version 28. Prior to running CFA, model checking was conducted based on previous literature.^26–28 Sample size adequacy and missingness were assessed. Multivariate normality of data for each questionnaire was determined by examining skewness values in IBM SPSS Amos version 28 output on Assessment of Normality. Distribution was deemed normal when skewness values of items fall between −1.5 and 1.5.²³ Outliers were examined by assessing the Mahalanobis distance in the IBM SPSS Amos version 28 output and were only dealt with if non-normal distribution existed.

Following this, CFA was conducted using IBM SPSS Amos version 28. The pooled CFA was run to test whether the data fit the hypothesised measurement model²⁹ by assessing each questionnaire construct’s unidimensionality, validity (convergent validity, construct validity, discriminant validity), and composite reliability,^23,29 The measurement models for all three QAWRF components, which were constructed based on the Chappell and Di Martino’s WPV model and systematic review findings, were modified iteratively (item deletion/items constrained) until all indices requirements (Table 3) for the respective models were achieved. The CFA was conducted initially to achieve the model fit for construct validity, first by deleting items with low factor loading (one at a time) and then by constraining items with the highest modification indices (MI) after discussing the level of importance of maintaining each item in the questionnaire.²³ High MI indicates redundant items within the same construct. Once the model fit for construct validity had been achieved, indices for convergent validity, discriminant validity, and composite reliability (CR) were calculated to ensure these indices met the level of acceptance.²³ Further modifications (item deletion/items constrained) were only made if any of the indices did not meet the level of acceptance.

Table 3 Confirmatory Factor Analysis Indices and Their Values of Acceptance^23,29

Name of Category	Name of Index	Level of Acceptance
Unidimensionality	Factor Loading (Standardized Regression Weight)	Weight ≥ 0.6
Validity
1. Convergent validity	Average Variance Extracted (AVE) AVE = ∑Қ^2/n*	AVE ≥ 0.5
2. Construct validity	1. Absolute Fit Root Mean Square Error of Approximation (RMSEA) 2. Incremental Fit Comparative Fit Index (CFI) Tucker Lewis Index (TLI) 3. Parsimonious Fit Chi-square/degree of freedom (Chisq/df)	RMSEA > 0.1 acceptable, best if < 0.08 CFI > 0.9 TLI > 0.9 Chisq/df < 5.0, best if < 3.0
3. Discriminant validity	Correlation coefficient between construct Discriminant validity index (DVI)	Correlation coefficient between construct ≤ 0.85 The discriminant validity index of the respective construct must have higher values than correlation coefficient values between the constructs involved
Construct Reliability	Composite Reliability (CR) CR = (∑Қ)^2/[(∑Қ)^2 + (∑1-Қ^2)]*	CR ≥ 0.6

Notes: *Қ was the factor loading for every item, while n was the number of items in the construct. Indexes and the level of acceptance for each index were suggested by Awang (2015)²² and Aimran & Ahmad (2013)²⁸.

For test–retest reliability, the data obtained was analysed using IBM SPSS version 28. The test–retest reliability of the QAWRF was determined by computing the intraclass correlation coefficient (ICC). An ICC value of below 0.50 was regarded as having poor test–retest reliability, 0.5 to 0.74 as having fair test–retest reliability, 0.75 to 0.89 as having good test–retest reliability, and 0.90 to 1.00 as having excellent test–retest reliability.³⁰ Calculations were made at the scale and domain level for each QAWRF component using two-way mixed-effect model, average measurement, and absolute agreement in IBM SPSS version 28, as suggested by previous literature.³¹

Results

Preliminary analysis indicated multivariate normality, sufficient sample size, and no missing data. A total of 302 administrator’s data for QAWRF-Administrator, 356 worker’s data for QAWRF-Worker, and 307 client’s data for QAWRF-client were used for analysis. For test–retest reliability, data from 30 administrators, 30 workers, and 30 clients were used for analysis.

Construct Validity of the QAWRF-Administrator

The initial model, Model Administrator-A, achieved the fitness index for absolute fit (RMSEA = 0.072) and parsimonious fit (Chisq/df = 2.539). However, the model did not achieve acceptable fitness indices for incremental fit, as observed by CFI of 0.839 and TLI of 0.830 (Table 4). After deleting nine items with factor loading <0.600 one by one (c8, w8, w11, w14, p5, p6, p9, s1, s2), the second model, Model Administrator-B still did not achieve the incremental fit indices as the CFI was 0.885 and TLI was 0.876 (Table 4). The incremental fit indices were only achieved after two pairs of items with the highest MI values were constrained (p10 with p11, w5 with w6) as illustrated in the final model, Model Administrator-C (Figure 1). The fitness values for construct validity of the final model were all acceptable (Table 4).

Figure 1 Final model for QAWRF-administrator.

Notes: Domains/constructs/latent variables (oval), items/measured variables (rectangles), factor loadings (single-headed arrows), correlations (double-headed arrows), error terms (circle).

Abbreviations: RMSEA, root mean square error of approximation; CFI, confirmatory fit index; TLI, Tucker Lewis Index; Chisq/df, chi-square/degree of freedom.

Table 4 Confirmatory Factor Analysis Fitness Indices

QAWRF Component	Model	Absolute Fit	Incremental Fit		Parsimonious Fit
		RMSEA	CFI	TLI	Chisq/df
QAWRF-Administrator	Administrator-A (Initial)	0.072	0.839	0.830	2.539
	Administrator-B	0.070	0.885	0.876	2.476
	Administrator-C (Final)	0.063	0.908	0.901	2.186
QAWRF-Worker	Worker-A (Initial)	0.076	0.842	0.833	3.031
	Worker-B	0.075	0.883	0.874	2.980
	Worker-C (Final)	0.066	0.910	0.903	2.528
QAWRF-Client	Client	0.071	0.965	0.958	2.545

The factor loadings of items, AVEs, and CRs for every domain of the final model exceeded the minimum values required (Table 5). Therefore, the unidimensionality, convergent validity and construct reliability for all domains were satisfactory for QAWRF-Administrator. Based on the DVI and correlation coefficients between constructs, the discriminant validity for QAWRF-Administrator was also satisfactory (Table 6), and there were no multicollinearity issues between constructs. The final validated QAWRF-Administrator has four domains and 36 items, with nine items in the client domain, 13 items in the worker domain, ten items in the workplace domain, and four items in the societal domain.

Table 5 Factor Loadings, Average Variance Extracted, and Composite Reliability for QAWRF-Administrator Final Model

Domain/Construct	Item	Factor Loading	AVE	CR
Client	c1	0.81	0.56	0.92
	c2	0.80
	c3	0.70
	c4	0.71
	c5	0.68
	c6	0.76
	c7	0.75
	c9	0.76
	c10	0.75
Worker	w1	0.75	0.57	0.94
	w2	0.77
	w3	0.73
	w4	0.68
	w5	0.70
	w6	0.75
	w7	0.80
	w9	0.73
	w10	0.88
	w12	0.73
	w13	0.78
	w15	0.76
	w16	0.69
Workplace	p1	0.70	0.53	0.92
	p2	0.73
	p3	0.84
	p4	0.69
	p7	0.72
	p8	0.81
	p10	0.67
	p11	0.63
	p12	0.74
	p13	0.73
Societal	s3	0.81	0.69	0.90
	s4	0.92
	s5	0.84
	s6	0.74

Table 6 Discriminant Validity Index of QAWRF

QAWRF Component	Domain	Client	Worker	Workplace	Societal
QAWRF-Administrator	Client	0.75
	Worker	0.68	0.75
	Workplace	0.60	0.67	0.73
	Societal	0.41	0.35	0.59	0.83
QAWRF-Worker	Client	0.74
	Worker	0.67	0.76
	Workplace	0.72	0.69	0.77
	Societal	0.65	0.56	0.67	0.87
QAWRF-Client	Worker		0.86
	Workplace		0.72	0.77

Notes: Bolded values are the DVI of the respective construct, while non-bolded values are the correlation coefficient values between constructs.

Construct Validity of the QAWRF-Worker

The initial model, Model Worker-A also achieved the fitness index for absolute fit (RMSEA = 0.076) and parsimonious fit (Chisq/df = 3.031) but did not achieve acceptable fitness indices for incremental fit (Table 4). The incremental fit indices were 0.842 for CFI and 0.833 for TLI. After deleting eight items one by one with factor loading <0.60 (c8, w9, w11, 14, p5, p6, p9, s1), the second model, Model Worker-B still did not achieve the acceptable incremental fit indices as the CFI was 0.883 and TLI was 0.874 (Table 4). After three pairs of items with the highest MI values were constrained (p10 with p11, w1 with w2, w5 with w7) as in the final model, Model Worker-C, the incremental fit indices were satisfactory (Figure 2). The fitness values for construct validity of the final model were all acceptable (Table 4).

Figure 2 Final model for QAWRF-worker.

Notes: Domains/constructs/latent variables (oval), items/measured variables (rectangles), factor loadings (single-headed arrows), correlations (double-headed arrows), error terms (circle).

Abbreviations: RMSEA, root mean square error of approximation; CFI, confirmatory fit index; TLI, Tucker Lewis Index; Chisq/df, chi-square/degree of freedom.

The factor loadings of items, AVEs, and CRs for every domain achieved the minimum values required (Table 7). Therefore, the unidimensionality, convergent validity, and construct reliability for all domains were satisfactory for QAWRF-Worker. Based on the DVI and correlation coefficients between constructs, the discriminant validity for QAWRF-Worker was also adequate (Table 6), and there were no multicollinearity issues between constructs. The final validated QAWRF-Worker has four domains and 35 items, with nine items in the client domain, 13 items in the worker domain, nine items in the workplace domain, and four items in the societal domain.

Table 7 Factor Loadings, Average Variance Extracted, and Composite Reliability for QAWRF-Worker Final Model

Domain/Construct	Item	Factor Loading	AVE	CR
Client	c1	0.75	0.54	0.91
	c2	0.65
	c3	0.70
	c4	0.76
	c5	0.74
	c6	0.72
	c7	0.77
	c9	0.83
	c10	0.71
Worker	w1	0.76	0.57	0.94
	w2	0.81
	w3	0.80
	w4	0.79
	w5	0.81
	w6	0.80
	w7	0.69
	w8	0.73
	w10	0.84
	w12	0.69
	w13	0.71
	w15	0.65
	w16	0.68
Workplace	p1	0.78	0.59	0.93
	p2	0.73
	p3	0.85
	p4	0.82
	p7	0.75
	p8	0.76
	p10	0.68
	p11	0.69
	p12	0.82
Societal	s3	0.78	0.75	0.92
	s4	0.85
	s5	0.91
	s6	0.91

Construct Validity of the QAWRF-Client

The QAWRF-Client achieved acceptable values for all CFA indices without modifying the initial model (Model Client, Figure 3). The model fitness values for construct validity, such as absolute fit (RMSEA = 0.071), incremental fit(CFI = 0.965, TLI = 0.958), and parsimonious fit (Chisq/df = 2.545) values all exceeded the required values (Table 4).

Figure 3 Final model for QAWRF-client.

Notes: Domains/constructs/latent variables (oval), items/measured variables (rectangles), factor loadings (single-headed arrows), correlations (double-headed arrows), error terms (circle).

Abbreviations: RMSEA, root mean square error of approximation; CFI, confirmatory fit index; TLI, Tucker Lewis Index; Chisq/df, chi-square/degree of freedom.

The factor loadings of the items, AVEs, and CRs for both domains achieved acceptable indices values (Table 8). Therefore, the unidimensionality, the convergent validity and construct reliability for both domains in QAWRF-client were established. Based on the DVI and correlation coefficients between constructs (Table 6), the discriminant validity for QAWRF-Client was adequate and there were no multicollinearity issues between constructs. The final validated QAWRF-Client has two domains and 14 items, with six items in the worker domain and eight items in the workplace domain.

Table 8 Factor Loadings, Average Variance Extracted, and Composite Reliability for QAWRF-Client Final Model

Domain/Construct	Item	Factor Loading	AVE	CR
Worker	w1	0.86	0.74	0.95
	w2	0.87
	w3	0.82
	w4	0.88
	w5	0.86
	w6	0.87
Workplace	p1	0.76	0.60	0.92
	p2	0.77
	p3	0.8
	p4	0.76
	p5	0.69
	p6	0.81
	p7	0.81
	p8	0.77

Test–Retest Reliability of QAWRF

All three components of QAWRF had satisfactory test–retest reliability indices for all domains and overall scales (Table 9). For QAWRF-Administrator, the client domain [ICC = 0.897 (95% CI: 0.785,0.951)], worker domain [ICC = 0.768 (95% CI: 0.519,0.889)], workplace domain [ICC = 0.854 (95% CI: 0.697,0.930)] and societal domain [ICC = 0.773 (95% CI: 0.529,0.891)] all had good test–retest reliability. Meanwhile, the test–retest reliability for QAWRF-Administrator overall scale was excellent [ICC = 0.910 (95% CI: 0.813,0.957)].

Table 9 Test–Retest Reliability of QAWRF

QAWRF Component	Domain	ICC	95% Confidence Interval		F test with True Value 0
			Lower Bound	Upper Bound	Value	df1	df2	Sig
QAWRF-Administrator	Client	0.897	0.785	0.951	9.653	29	29	<0.001
	Worker	0.768	0.519	0.889	4.390	29	29	<0.001
	Workplace	0.854	0.697	0.930	6.976	29	29	<0.001
	Societal	0.773	0.529	0.891	4.544	29	29	<0.001
	Scale	0.910	0.813	0.957	11.464	29	29	<0.001
QAWRF-Worker	Client	0.849	0.685	0.928	6.940	29	29	<0.001
	Worker	0.797	0.577	0.903	4.933	29	29	<0.001
	Workplace	0.805	0.592	0.907	5.071	29	29	<0.001
	Societal	0.806	0.593	0.907	5.083	29	29	<0.001
	Scale	0.904	0.799	0.954	10.301	29	29	<0.001
QAWRF-Client	Worker	0.836	0.645	0.924	5.963	27	27	<0.001
	Workplace	0.779	0.522	0.898	4.449	27	27	<0.001
	Scale	0.857	0.690	0.934	6.804	27	27	<0.001

Notes: Test–retest reliability testing using two-way mixed-effect model, average measurement, and absolute agreement.

Similar findings were noted for QAWRF-Worker. The client domain [ICC = 0.849 (95% CI: 0.685,0.928)], worker domain [ICC = 0.797 (95% CI: 0.577,0.903)], workplace domain [ICC = 0.805 (95% CI: 0.592,0.907)] and societal domain [ICC = 0.806 (95% CI: 0.593,0.907)] had good test–retest reliability. The test–retest reliability was excellent for QAWRF-Worker overall scale [ICC = 0.904 (95% CI: 0.799,0.954)].

For QAWRF-Client, the worker domain [ICC = 0.836 (95% CI: 0.645,0.924)] and the workplace domain [ICC = 0.779 (95% CI: 0.522,0.898)] both had good test–retest reliability. Meanwhile, the ICC indicated that the test–retest reliability for QAWRF-Client overall scale was good [ICC = 0.857 (95% CI: 0.690,0.934)].

Discussion

The main objective of the present study was to examine the construct validity and test–retest reliability of QAWRF, a set of structured survey questionnaires that can assess WPV risk factors in healthcare settings based on a tripartite perspective of key stakeholders. Based on the study findings, QAWRF-Administrator, QAWRF-Worker, and QAWRF-Client had acceptable construct validity and test–retest reliability. Given the established content validity, face validity, construct validity, internal consistency reliability, and test–retest reliability of QAWRF,¹⁵ it is a validated instrument that is able to evaluate WPV risk factors in healthcare settings based on the perceptions of three key healthcare stakeholders, both effectively and reliably.

In this study, CFA was utilised to evaluate the construct validity of QAWRF instead of performing exploratory approaches first such as exploratory factor analysis (EFA) or principal component analysis (PCA). Exploratory approaches are appropriate when the number of factors and the allocation of items to factors are unknown.³² However, for most psychological measures, performing an exploratory approach may not be appropriate, as most of these measures were already designed to capture specific constructs with a clear hypothesis that would best be tested using a confirmatory technique.³² Additionally, most discussions by scholars on whether conducting exploratory approach is compulsory before performing CFA for scale development scholars fail to meet a clear consensus.³³ However, scholars have made it clear that before conducting CFA, the researchers must have a priori theory or strong theoretical assumption regarding what factors should be in the data and what variables should define each factor.^33,34 The Chappell and Di Martino’s Interactive Model of WPV, used as the study’s theoretical framework, was considered useful for research studies as it provides evidence for improving preventive strategies for WPV.^35–37 Indeed, to further demonstrate the model’s strength, the domains in the model were similar to how most narrative reviews and systematic reviews categorised WPV risk factors,^12,38–42 and discussions with experts revealed that the WPV risk factors in each model domains were almost exclusive to the particular domain. Given the priori theory underpinning QAWRF development, CFA was preferable as exploratory approaches such as EFA have been associated with poor item distributions and difficulty in factor interpretation.^33,43 Indeed, trial EFA findings using the same data resulted in item loadings that deviated from factors in the original domains. Moreover, because this study involved assessing the construct validity of three component questionnaires with the same aim (to assess WPV risk factors) but differing perspectives (administrators, workers, and clients) simultaneously, conducting EFA or PCA may lead to three different models, which could complicate study findings. Therefore, this study performed CFA directly using the prespecified Chappell and Di Martino’s Interactive Model of WPV.

The satisfactory indices for test–retest reliability of QAWRF also indicate that it can provide consistent results over time. Therefore, healthcare stakeholders should be able to use QAWRF data confidently to develop specific WPV interventions without the risk of wasting valuable resources for unnecessary or unreliable interventions. Interestingly, the study findings suggest that respondent exposure to WPV risk factors was important in determining a scale’s test–retest reliability.⁴⁴ In this regard, clients who spent the least time at a healthcare facility resulted in slightly lower test–retest reliability overall scale indices compared to administrators and workers who were only selected if they had working experience of more than six months in healthcare facilities. Nevertheless, it is important to consider clients’ perspectives of WPV because, based on literature reviews, clients were the perpetrators for most WPVs.^3–5 Identifying factors that they perceive to be important WPV risk factors may enable the design and implementation of more effective WPV program.²

Based on the literature review, the most commonly used instrument used to assess WPV risk factors was a questionnaire jointly developed by the International Labour Office (ILO), International Council of Nurses (ICN), World Health Organization (WHO), and Public Services International (PSI) named the Workplace Violence in the Health Sector Country Case Studies Research Instrument.^45,46 In this questionnaire, the risk factors of WPV are assessed by associating the demographic profile of respondents with WPV incidents and by exploring answers to the two open-ended questions on the potential risk factors for physical and psychological violence. Unfortunately, most studies using the ILO/ICN/WHO/PSI instrument only reported the demographic factors associated with WPV rather than WPV risk factors that are potentially modifiable such as worker, workplace, and organisational factors. This may likely be due to difficulty in extracting, categorising, and analysing responses on WPV risk factors from the two open-ended questions⁴⁷ or lack of responses to the open-ended questions.⁴⁸ Therefore, the more modifiable risk factors of WPV, which were previously difficult to explore, are the main contents of the structured QAWRF. As such, by using the validated QAWRF, managers at each healthcare facility unit can conveniently and confidently assess WPV risk factors at their units and develop their own data-driven, worksite-based, and risk-based approach WPV intervention, the only WPV intervention shown to have moderate effectiveness.¹²

The QAWRF can also be used to complement WPV incident investigations, in which the questionnaires can be distributed to involved parties at the incident location to assess the most probable WPV determinants of the incident, without the need for interviews or walkthrough surveys. In addition, the QAWRF, being a standardised and structured set of questionnaires, can also be used to compare WPV risk factors findings for different times and places. For example, if the QAWRF is utilised before and after a WPV intervention has been implemented, it can be used to ascertain the effectiveness of the intervention, whereby the WPV risk factors that are the focuses of the intervention are expected to be rated as less prevalent after the intervention. QAWRF can also be utilised to compare WPV risk factors between healthcare settings, including between clinics in the same hospitals, between clinics in rural and urban areas, as well as between hospital outpatient clinics and health clinics. Finally, the QAWRF can be a helpful tool to educate and increase healthcare stakeholders’ awareness on WPV risk factors, especially if it is used periodically and consistently.

To ensure a holistic understanding of WPV risk factors in a given healthcare setting, the three QAWRF component questionnaires should be used simultaneously when assessing WPV risk factors, as the literature indicates that engaging all stakeholders and considering their collective perspectives can facilitate better policymaking and improved measures for preventing WPV.¹³ QAWRF users should then develop the necessary WPV interventions based on the overall findings. However, in the event of time or human resource constraints, QAWRF users may prioritise using QAWRF-Worker as workers who work directly on the ground are more familiar with process activities, facility operations, and potential WPV risk factors,² as well as the QAWRF-Administrator because administrators are more exposed to the particular workplace and might be involved in WPV data management as compared to clients.

There are a few limitations to this study. Firstly, as QAWRF was developed in the Malay language and the risk factors included were specific to Type 2 WPV in healthcare, QAWRF may lack generalizability to non-Malay speaking populations and other types of WPV.¹⁵ Secondly, as multiple WPV risk factors had to be included in QAWRF-Administrator and QAWRF-Worker to ensure the completeness of the questionnaires, they were both quite lengthy. However, measures to word each item as concisely as possible were made to reduce the likelihood of respondent fatigue. Thirdly, social desirability bias might influence the responses received during the validation of QAWRF. QAWRF’s respondents, especially administrators and workers, may provide responses according to social expectations instead of responses according to their true perception of WPV risk factors for impression management so that their workplaces would be viewed more favourably by others.^49,50 However, this may have likely been minimised as prior to the validation process, respondents were explained that this study was for validation purposes only and not for reporting the magnitude of the WPV risk factors present at their workplaces.

Conclusions and Recommendations

This study has completed the validation for a set of questionnaires, the QAWRF, that can be used to assess risk factors of WPV in healthcare facilities based on the perspectives of three key healthcare stakeholders. In terms of implication for future research, the QAWRF can be used by future researchers examining WPV risk factors in healthcare settings. In this regard, the QAWRF can be combined with other tools, such as the ILO/ICN/WHO/PSI instrument,⁴⁶ to provide a better understanding of WPV. Furthermore, future studies should translate and validate an English version of the QAWRF to improve its applicability to healthcare settings in other parts of the world and ensure its usefulness in developing successful WPV interventions. The QAWRF can also be expanded in the future by including other WPV risk factors that were not covered by the current study, such as risk factors for type 3 WPV (WPV between coworkers), as well as risk factors for WPV related to inpatient settings.

In terms of implications for policy and practice, findings from QAWRF can contribute to the development of WPV policies in healthcare settings that outlines specific risk reduction measures based on the prevalent WPV risk factors identified in the particular health setting. In addition, healthcare administrators may also consider using QAWRF as a part of a risk assessment matrix or risk index in which workplaces can be classified as having a high, moderate, or low probability of WPV.⁵¹ This can be done by considering WPV risk factors in QAWRF as a likelihood of an event where each risk factor is assigned a certain score based on the level of contribution of the risk factors to WPV. The total likelihood scores can then be multiplied with another score assigned to the effects of WPV, such as frequency of WPV occurrence, staff turnover rates, staff burnout, or stress level, to provide the risk probability of WPV at the worksite.⁵² Overall, findings from QAWRF can contribute towards data-driven, worksite-specific, and targeted WPV interventions in healthcare settings that are expected to be resource-efficient and more effective than general WPV interventions.

Data Sharing Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to restrictions from data owners.

Ethics Statement

The study was conducted according to the guidelines of the Declaration of Helsinki. It was approved by the Medical Research & Ethics Committee [NMMR-21-537-58890 (IIR), approval date 18 May 2021] and Universiti Teknologi MARA (UiTM) Research Ethic Committee [REC/07/2021 (MR/629), approval date 2 August 2021].

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis, and interpretation, or in all these areas; took part in drafting, revising, or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

The authors declare no conflicts of interest in this work.

Acknowledgment

We would like to express our special gratitude to the panel of experts for their involvement in validating the questionnaires.

Additional information

Funding

This study is funded by Universiti Teknologi MARA’s Faculty of Medicine Research Grant [600-TNCPI 5/3/DDF (MEDIC) (010/2021)].