Evidence of validity for the Norwegian version of the interprofessional collaborative competency attainment survey (ICCAS)

ABSTRACT

This was a validation study of the Norwegian version of The Interprofessional Collaborative Competency Attainment Survey (ICCAS). ICCAS consists of 20 retrospective pre- and post-questions, where respondents rate their agreement with regard to self-assessed competencies after participating in interprofessional education courses. It has been validated across various settings. The questionnaire was translated using the back-translation technique. We investigated evidence of validity regarding content, response process, and internal structure. Data were obtained from health and social care students (n = 1440, response rate 42.8%) participating in 12 different interprofessional courses in seven education institutions in Norway using a cross-sectional design. Exploratory factor analysis indicated one retracted factor for pre-scores and one retracted factor for post-scores. High McDonald’s omega values indicated good internal consistency. Item deletion did not improve the scale’s overall consistency on pre- or post-scores. We observed higher mean post-scores than pre-scores with moderate-to-large effect sizes, indicating a positive change in self-assessed interprofessional capabilities after training. Our findings indicate that the Norwegian version of ICCAS is a valid tool that may be implemented across a wide range of interprofessional education courses. Finally, our findings support earlier recommendations that ICCAS should be analyzed at an overall level to address change in interprofessional capabilities.

KEYWORDS:

• Interprofessional education
• student assessment
• instrument validation
• ICCAS

Introduction

In 2010, the World Health Organization (WHO) highlighted that, to meet new challenges associated with optimized patient care, health-care professionals must work in interprofessional teams. The Commission on Education of Health Professionals for the 21^st Century emphasized that health-care graduates were capable of interprofessional team-based care (Frenk et al., 2010). Increasing complexity in health promotion and public health worldwide has contributed to expanding interprofessional education (IPE) and interprofessional collaboration (IPC) beyond health and social care educations to include professions such as police and teachers (Barr et al., 2005).

Although there is worldwide agreement on the importance of IPE and IPC (Reeves, Palaganas et al., 2017; Reeves, Pelone et al., 2017), there is no clear international or national consensus regarding how such competencies should be assessed. Several tools are available for use in the self-assessment of individual attitudes and skills in IPC. However, most of the available assessment tools are dependent on the users’ contexts and situations, and have scarcely been validated (Boet et al., 2019; Nelson et al., 2017; Shoemaker et al., 2016). The use of validated assessment tools that are less context-sensitive can help educators’ measure students’ self-reported achieved competence in IPE and, consequently, develop the optimal IPE for future professionals.

Background

In Norway, IPC has been highlighted in several white papers as a key factor in improving health and social services, with requirements for graduates’ knowledge of IPC and collaborative competencies (The Norwegian Ministry of Education and Research, 2012, 2017b; The Norwegian Ministry of Health and Care Services, 2009). Subsequently, national frameworks, learning outcomes, and recommendations for IPC are being developed (The Norwegian Association of Higher Education Institutions, 2016; The Norwegian Ministry of Education and Research, 2017a). Several IPE courses and initiatives have emerged in recent years at Norwegian Universities and University Colleges. In an effort to synthesize experiences and contribute to expanding the IPE field in Norway, a collaboration between seven educational institutions offering different IPE courses was formed.

One way to assess students’ competencies in IPC and consequently evaluate, and compare educational courses is to use a validated self-reported questionnaire that allows comparison across various courses. We chose to translate The Interprofessional Collaborative Competency Attainment Survey (ICCAS) to Norwegian as it has been deemed valid and reliable in a variety of IPE settings (Archibald et al., 2014; Schmitz et al., 2017; Violato & King, 2019). Use of existing questionnaires does not guarantee validity after translation and cultural modification. Thus, a translated version needs to be validated (i.e., the degree to which an instrument measures what it says it should measure for the intended purpose; Cook & Beckman, 2006). Validity can be evaluated through five sources; content, response process, internal structure, relations with other sources, and consequences (Cook & Lineberry, 2016; Cook et al., 2014).

The aim of the present study was to assess the evidence for validity of the Norwegian version of ICCAS across several different educational courses as an instrument for measuring self-reported achieved competence in IPE.

The interprofessional collaborative competency attainment survey

The Interprofessional Collaborative Competency Attainment Survey (ICCAS) was developed in response to the call for validated assessment instruments for IPE evaluation (Archibald et al., 2014). ICCAS was based on the interprofessional care competencies: communication, collaboration, patient-/family-centered approach, roles, and responsibilities, conflict resolution and management, and team functioning (Canadian Interprofessional Health Collaborative, 2010). Participants self-assess changes in their interprofessional competencies’ levels after completing an IPE intervention. The questionnaire has 20 retrospective pre- and post-questions. Respondents rate their agreement using a 7-point Likert-type scale, thus adopting a retrospective pre-/posttest measurement format. It also includes a “not applicable” option. The initial psychometric study of ICCAS in English and French consisted of 584 participants from 15 different IPE programs with 19 different professions in Canada and New Zealand (Archibald et al., 2014). The authors found good internal consistency and reliability, with two retracted pre-score factors (Cronbach’s alpha.96 and .94) and one post-score factor (Cronbach’s alpha .98). The authors concluded that ICCAS can be used to measure participants’ competencies across several IPE programs (Archibald et al., 2014).

In a replication validation study, 785 students from various health-care professions participated after completing an educational program in Minnesota, USA (Schmitz et al., 2017). The authors made two changes to the questionnaire: the rating scale was changed from a 7-point to a 5-point qualitative Likert-type scale, and an item designed to capture how much the students’ overall abilities had changed was included. The retrospective pre-/posttest measurement format was retained for collecting the data. Good internal consistency and reliability were observed with retraction of one factor for post-scores (Cronbach’s alpha .96), supporting the use of ICCAS. The use of an overall sum score was recommended due to strong conceptual overlap between constructs (Schmitz et al., 2017).

In a Canadian validation study, 991 students from various health programs participated after completing a mandatory three-hour IPE course during their first weeks of education (Violato & King, 2019). Schmitz et al. (2017) version was used, and retrospective pre-/posttest measurement format was chosen for collection of data. The study provided additional validity evidence for ICCAS, supporting a single-factor structure for pre-scores (Cronbach’s alpha .97) and post-scores (Cronbach’s alpha .95; Violato & King, 2019). ICCAS has been used as an evaluation tool in multiple different settings in recent years such as interactive case-based IPE sessions for pre-licensure health science students (Langford et al., 2019), academic day devoted to introductory IPE experiences for first-year health-care students (Singer et al., 2018) and IPE clinical simulations for students from non-clinical disciplines and health-care disciplines (Champagne-Langabeer et al., 2019). However, to date, most of the studies are based on the original English version (Archibald et al., 2014).

Method

This validation study was a national collaboration between seven academic institutions in Norway responsible for training health and social science students in IPC.¹

Questionnaire and scoring

The Norwegian ICCAS translation was based on the English version developed by Schmitz et al. (2017) with a 5-point Likert-type scale. We retained the wording of the responses from the original study (Archibald et al., 2014), namely: 1 = strongly disagree, 2 = slightly disagree, 3 = neutral, 4 = slightly agree, and 5 = strongly agree, rather than using the qualitative responses (1 = poor, 2 = fair, 3 = good, 4 = very good, 5 = excellent) developed by Schmitz et al. (2017). In the Norwegian translation, the students rate their agreement, which is better answered with agree-disagree than poor-excellent. We also retained the response n/a = not applicable from the original questionnaire. Higher scores reflect a more positive evaluation of the students’ self-assessed interprofessional capabilities (Archibald et al., 2014). An overall sum score is recommended because of strong conceptual overlap among constructs (Schmitz et al., 2017). However, because this is the first validation of the Norwegian version, we found it necessary to replicate pre-post response rate on item level for comparison.

Translation

The Center for Interdisciplinary Work-Place Learning (TVEPS) at the University of Bergen (UiB) translated the first draft of ICCAS into Norwegian, with independent back-translation into English (Beaton et al., 2000). After adjusting some grammatical discrepancies, TVEPS finalized a preliminary Norwegian version. Through a national Delphi process with representatives from all seven institutions, a cultural validation took place. A Delphi process is a means of obtaining structured group opinion and group consensus from experts in a given field (Hsu & Sandford, 2010). We discussed, incorporated, and agreed upon proposed amendments during Skype® meetings and via e-mail over a period of 6 months. Cognitive interviews were conducted with two students. Based on their interpretations of the individual questions as they filled out the form, only small changes were made. Following the students’ review, we agreed upon a final version with 20 items.

Data collection

Data collection took place in a cross-sectional study from September 2018 to January 2019. We included students from the IPE courses, a total of 12, currently in effect at our seven institutions, to obtain a variety of professions and courses. The courses varied in duration and spanned a wide range of applied pedagogy, from interprofessional learning in the workplace, through learning in simulated environments, and case-based learning on campus.

A web-based survey, developed in SurveyXact® (Rambøll Management Consulting, 2019) was used. The students were provided with a link to the survey directly after completing an IPE course, thus maintaining the retrospective pre-/posttest measurement format for data collection. In addition to the ICCAS questions, the students stated gender, age, place of study, field of study (IPE course and profession), and academic year.

Data analysis

Sample characteristics of the respondents and basic statistics were calculated to visualize the data material. We treated the response category “not applicable” as a missing value in the data analysis. Items missing on item level were not imputed.

The validity of the Norwegian version of ICCAS was assessed by content (Delphi experts), response process (cognitive interviews), and internal structure (factor analysis, internal consistency, and paired t-tests). Content validity evidence was evaluated in a Delphi process with IPE experts. Evidence for response process validity was assessed by cognitive interviews with two students (Cook & Lineberry, 2016; Cook et al., 2014).

Evidence for internal structure validity was evaluated by factor analysis, internal consistency, and paired t-test (Cook & Lineberry, 2016; Cook et al., 2014). Internal structure was assessed by Exploratory Factor Analysis (EFA) using Principal Axis Factoring (PAF) with an oblique oblimin rotation to attain the best fitting structure and number of factors (Field, 2018). We chose PAF over other extraction procedures because it makes no assumption regarding the variables’ distribution, and oblique rotation in view of the distinct possibility that any underlying factors might be correlated (Field, 2018). PAF was conducted separately on both pre- and post-scores. The suitability of the data for factor analysis was assessed by inspecting the correlation matrix for coefficients of .3 and above (Field, 2018) and Kaiser-Meyer-Olkin (KMO) values above .7 to justify good sampling adequacy (Dziuban et al., 1974). Bartlett’s test of sphericity must reach statistical significance to further support the factorability of the correlation matrix (Field, 2018). To further assess the number of factors to retain, we did a Parallel Analysis. Only factors with eigenvalues exceeding the corresponding eigenvalues from a random data set of the same size are suggested for retention (O’Connor, 2000).

Internal consistency was assessed using McDonald’s omega coefficient and item-total correlation (Dunn et al., 2014). If the different items on the scale measure the same concept, the internal consistency must be greater than or equal to .7. Additionally, on a reliable scale, all items should correlate with the total score (r ≥ .30) (Field, 2018; Kline, 2000).

Paired t-tests on pre-and post-scores for each item were assessed to evaluate the Norwegian versions’ ability to detect changes in perceived IPC competencies. We analyzed differences in pre- and post-scores in terms of standardized effect sizes, based on Cohen d calculations and 95% confidence limits. We interpreted “large” differences as those over .80, “moderate” differences between .79-.50, and “small” differences between .2-.49 (Cohen, 1988).

The SurveyXact® file was converted via Excel into an SPSS (Statistical Package for Social Science) file for statistical analysis in SPSS 26 (IBM Corp, 2017) and R (R Core Team, 2017).

Ethical considerations

Approval from the Norwegian Center for Research Data was obtained (project number 61063). Participation was voluntary, and we gained informed consent from the participants.

Results

Invitations to participate were sent to 3,367 students. Of 1,900 opened surveys, 1,012 were completed in their entirety, and 428 had most of ICCAS items completed. Consequently, 1,440 surveys were included in the analysis (42.8%). Of the respondents, 1,165 (80.9%) were female and 275 (19.1%) were male. The median age was 23 (range 18–52, mean 24.5, SD 5.32). Details of location, professional program, and academic year are listed in Table 1.

Table 1. Characteristics of the respondents (n = 1440). Number of respondents in each institution, program and academic year (n), and percentage of total sample (%)

	n (%)
Location
UiB	46 (3.2)
HVL Bergen	687 (47.7)
HVL Førde	5 (0.3)
UiS	63 (4.4)
OsloMet	11 (0.8)
UiO	15 (1.0)
NTNU Trondheim	558 (38.8)
UiT	55 (3.8)
Professional Program
Audiology	29 (2.0)
Biom edical Laboratory	96 (6.7)
Child Welfare	2 (0.1)
Dentistry*	8 (0.6)
Dental Hygiene	26 (1.8)
Geriatric Nursing	5 (0.3)
Kindergarten Teacher	3 (0.3)
Medicine*	123 (8.5)
Music Therapy	2 (0.1)
Nutrition*	8 (0.6)
Nursing	422 (29.3)
Occupational Therapy	136 (9.4)
Pharmacy*	11 (0.8)
Physiotherapy	147 (10.2)
Psychology*	5 (0.3)
Radiography	89 (6.2)
Social Education	179 (12.4)
Social Work	149 (10.3)
Academic year
1	268 (18.6)
2	205 (14.2)
3	914 (63.5)
4	18 (1.3)
5 6	22 (1.5) 13 (0.9)

UiB = University of Bergen, HVL = Western Norway University of Applied Sciences, UiS = University of Stavanger, OsloMet = Oslo Metropolitan University, UiO = University of Oslo, NTNU = Norwegian University of Science and Technology, UiT = Arctic University of Norway.

*Professional program of more than three years duration

Pre-score item means ranged from 3.6 to 4.2 and post-score item mean ranged from 4.2 to 4.6. Missing item responses for pre- and post-scores ranged between 0.6 and 10.5 and 0.4–11.3%, respectively. In a comprehensive manual review of missing items, there were no obvious systematic missing except items 15 and 18. Item 15 concerns the influence of the patient/user/family in decision-making, and item 18 deals with addressing team conflict in a respectful manner. Missing percentages were 8.1% pre and 8.8% post on item 15, and 10.5% pre and 11.3% post on item 18. Item descriptions may be found in Tables 2 and 3.

Table 2. Summary of exploratory factor analysis results

ICCAS item	Pre-scores Factor	Post-scores Factor
1. Promote effective communication among IP members	.62	.71
2. Actively listen to IP team members’ ideas and concerns	.63	.77
3. Express my ideas and concerns without being judgmental	.62	.71
4. Provide constructive feedback to IP members	.63	.68
5. Express my ideas and concerns in a clear, concise manner	.66	.70
6. Seek out IP team members to address issues	.69	.76
7. Work effectively with IP team members to enhance care	.74	.80
8. Learn with, from and about IP team members to enhance care	.73	.80
9. Identify and describe my abilities and contributions to the IP team	.69	.73
10. Be accountable for my contributions to the IP team	.72	.77
11. Understand the abilities and contributions of IP team members	.72	.81
12. Recognize how others’ skills and knowledge complement and overlap with my own	.70	.78
13. Use an IP team approach with the patient to assess the health situation	.72	.75
14. Use an IP team approach with the patient to provide whole person care	.70	.76
15. Include the patient/user/family in decision-making	.62	.67
16. Actively listen to the perspectives of IP team members	.72	.81
17. Take into account the ideas of IP team members	.72	.81
18. Address team conflict in a respectful manner	.65	.72
19. Develop an effective care plan with IP team members	.66	.68
20. Negotiate responsibilities within overlapping scopes of practice	.69	.73

* Factor loadings after rotation. Factor loadings over 0.40 appear in bold

Table 3. Item total score correlation and omega if item deleted based on students’ responses pre- and post-score

	Pre-scores		Post-scores
ICCAS item	Item total r	Omega if item deleted	Item total r	Omega if item deleted
1. Promote effective communication among IP members	.63	.90	.74	.92
2. Actively listen to IP team members’ ideas and concerns	.65	.90	.79	.92
3. Express my ideas and concerns without being judgmental	.63	.91	.73	.92
4. Provide constructive feedback to IP members	.64	.91	.69	.92
5. Express my ideas and concerns in a clear, concise manner	.67	.91	.74	.92
6. Seek out IP team members to address issues	.69	.91	.78	.92
7. Work effectively with IP team members to enhance care	.73	.91	.82	.92
8. Learn with, from and about IP team members to enhance care	.74	.91	.82	.92
9. Identify and describe my abilities and contributions to the IP team	.70	.91	.76	.92
10. Be accountable for my contributions to the IP team	.72	.91	.79	.92
11. Understand the abilities and contributions of IP team members	.72	.91	.81	.92
12. Recognize how others’ skills and knowledge complement and overlap with my own	.71	.91	.79	.92
13. Use an IP team approach with the patient to assess the health situation	.71	.91	.76	.92
14. Use an IP team approach with the patient to provide whole person care	.70	.91	.78	.92
15. Include the patient/user/family in decision-making	.62	.91	.70	.92
16. Actively listen to the perspectives of IP team members	.72	.91	.82	.92
17. Take into account the ideas of IP team members	.73	.91	.82	.92
18. Address team conflict in a respectful manner	.65	.91	.72	.92
19. Develop an effective care plan with IP team members	.67	.91	.70	.92
20. Negotiate responsibilities within overlapping scopes of practice	.70	.91	.76	.92

Content and response process validity

Content validity was deemed adequate in the Delphi process conducted over a 6 month period by IPE experts from the seven institutions. The constructs in the questionnaire were also found to align with the learning outcomes described in the IPE courses offered in Norway. The two students’ verbal assessment of the questionnaire indicated that they understood the questionnaire and the response format, thus supporting response process validity.

Internal structure

Exploratory factor analysis (EFA)

Evaluation of the correlation matrix-supported retention of all 20 items on ICCAS. The KMO values were .96 and .97 for the pre- and post-scores, indicating that the correlation matrix was appropriate for factor analysis. Bartlett’s test of sphericity reached statistical significance (p < .001) for pre- and post-scores, further supporting the factorability of the correlation matrix.

In the PAF analysis with oblique rotation (direct oblimin), three factors had eigenvalues over Kaiser’s criterion of 1 in the pre-scores and, in combination, accounted for 60.8% of the variance. The scree plot showed inflections that could justify the retention of two factors. The Parallel Analysis showed that only one component had an eigenvalue exceeding eigenvalues from the randomly generated data matrix of the same size. Because previous studies did not support the suggested theoretical five-factor structure, we found it suitable to retain one factor for pre-scores, accounting for 47% of the total variance.

For the post-score analysis, PAF showed eigenvalues of 11.68 and 0.95 for the highest-ranking factors, explaining 58.4% and 4.7% of the variance, respectively. Only one factor had an eigenvalue over Kaiser’s criterion of 1, and Parallel Analysis showed one component with eigenvalues exceeding the eigenvalues from the randomly generated data matrix. Considering a theoretical support of one factor for post-scores from previous studies, a single factor was extracted. The factor retained accounted for 56.3% of total variance. Table 2 shows the results of the factor analysis for the pre- and post-scores.

Internal consistency

The internal consistency of the ICCAS using McDonald’s omega reliability coefficient was .91 for pre-scores and .92 for post-scores. Item deletion did not improve the overall consistency of the scale on either pre- or post-scores. Table 3 illustrates the item-total correlations for pre- and post-scores.

The internal consistency of factors using McDonald’s omega reliability coefficient was .91 for the pre-score factor and .92 for the post-score factor.

Paired t-tests

There was a significant difference (p < .001, two-tailed paired sample t-test) in mean scores for each pre- and post-item pair. Overall, we observed moderate-to-large effect sizes for 18 of the 20 items indicating responsiveness of the Norwegian version (Table 4).

Table 4. Paired-samples t-tests between pre- and post-scores on each item

Construct	Items¹	Mean (SD) Retrospective	Mean (SD)	Mean difference [95% CI]	Effect size²	Difference³
		Pre-score	Post- score	Post-pre
Communication	1	3.55 (0.85)	4.30(0.75)	0.75 [0.71, 0.80]	.89	Large
	2	4.23 (0.85)	4.60 (0.73)	0.37 [0.33, 0.41]	.50	Moderate
	3	4.09 (0.90)	4.42 (0.80)	0.33 [0.29, 0.37]	.44	Small
	4	3.78 (0.95)	4.18 (0.85)	0.40 [0.36, 0.44]	.53	Moderate
	5	3.75 (0.89)	4.18 (0.84)	0.43 [0.39, 0.46]	.57	Moderate
Collaboration	6	3.78 (0.97)	4.36 (0.80)	0.58 [0.53, 0.62]	.67	Moderate
	7	3.84 (0.90)	4.44 (0.75)	0.60 [0.56, 0.65]	.74	Moderate
	8	3.74 (0.90)	4.47 (0.76)	0.73 [0.69, 0.77]	.84	Large
Roles and responsibilities	9	3.68 (0.92)	4.28 (0.80)	0.60 [0.55, 0.64]	.73	Moderate
	10	3.92 (0.94)	4.38 (0.78)	0.46 [0.42, 0.50]	.62	Moderate
	11	4.08 (0.92)	4.58 (0.74)	0.50 [0.46, 0.54]	.63	Moderate
	12	3.85 (0.93)	4.49 (0.76)	0.64 [0.60, 0.69]	.76	Moderate
Patient-centered care	13	3.58 (0.95)	4.29 (0.83)	0.71 [0.67, 0.76]	.82	Large
	14	3.64 (0.93)	4.29 (0.83)	0.65 [0.60, 0.69]	.77	Moderate
	15	3.85 (0.98)	4.28 (0.88)	0.43 [0.39, 0.47]	.56	Moderate
Conflict management, team functioning	16	4.20 (0.86)	4.58 (0.73)	0.38 [0.34, 0.42]	.54	Moderate
	17	4.19 (0.87)	4.54 (0.75)	0.35 [0.31, 0.38]	.52	Moderate
	18	4.00 (0.96)	4.31 (0.88)	0.31 [0.27, 0.34]	.46	Small
	19	3.62 (1.01)	4.18 (0.93)	0.56 [0.51, 0.61]	.67	Moderate
	20	3.67 (0.94)	4.20 (0.88)	0.53 [0.49, 0.57]	.64	Moderate

¹Item description is found in table 2 & 3. ²Cohen d, ³Qualitative differences: “Large” = values of ≥0.8, “Moderate” = values between 0.79-0.50 and “Small” values between 0.2-0.49 (Cohen, 1988)

Items on ICCAS were scored on a five-point Likert-type scale; 1= strongly disagree, 2= slightly disagree, 3= neutral, 4= slightly agree, 5= strongly agree, n/a= not applicable (registered as missing)

Discussion

Our study provides validity evidence for the Norwegian version of ICCAS in content, response process, and internal structure in alignment with findings from previous validation studies. The consequence of our study is further recommendation to analyze ICCAS at an overall level to address change in interprofessional capabilities.

A thorough Delphi process involving the IPE experts responsible for conducting the various IPE courses in Norway ensured that the content in the Norwegian version aligned with the construct, thus providing evidence of content validity. Evidence of response process was provided with cognitive interviews of two students indicating that they understood the questionnaire and the response format. In retrospect, we could have expanded the pilot test to involve several students in an effort to further assess clarity of the questionnaire. However, at the time, we deemed the answers from those students to be sufficient.

As with previous studies (Archibald et al., 2014; Schmitz et al., 2017; Violato & King, 2019), the factors emerging in our study did not support the theoretical five-factor construction of communication, collaboration, roles, and responsibility, patient/family-centered care, conflict management/resolution, and team functioning (Canadian Interprofessional Health Collaborative, 2010). We extracted one factor from the pre-scores and one factor for post-scores. High McDonald’s omega for the factor from pre-scores and the factor from post-scores in our material further demonstrates good internal consistency, supporting evidence of internal structure validity. However, as with previous studies (Archibald et al., 2014; Schmitz et al., 2017; Violato & King, 2019), a single-factor structure in our material suggests a strong conceptual overlap between constructs.

The internal consistency of the Norwegian version of ICCAS was found to be good, with high McDonald’s omega coefficient values, supporting evidence for the internal structure’s validity. Item deletion from the Norwegian version of the scale would neither increase nor decrease the McDonald’s omega. This is consistent with the results from previous validation studies (Archibald et al., 2014; Schmitz et al., 2017; Violato & King, 2019), further supporting retention of the items. A desirable value for internal consistency is usually between .70 and .90. Very high omega coefficients might be associated with redundancy of elements (Dunn et al., 2014). In our material, the McDonald’s omega coefficient is just above the upper value of .90. ICCAS has proved to contain sets of measures that are naturally correlated, and therefore, can be expected to be highly interrelated. Based on that assumption, we argue that our findings suggest a good internal consistency.

We found that mean post-scores were higher than pre-scores, reflecting a positive change in self-assessed interprofessional capabilities. Our study indicates a slightly larger effect size on the different items than Archibald et al. (2014) and Schmitz et al. (2017), with 18 items scoring moderate or large effect size. Violato and King (2019) reported higher effect sizes than our study, with the majority of items scoring large effect size. The authors argued that the large effect sizes may be accounted for by low levels of previous IPE and IPC experience among their participants (Violato & King, 2019). Several factors may explain the differences in our material, including previous IPE experience or the absence thereof. However, we did not ask the students about their prior experience. Another reason could be that the participating educational institutions have an above-average involvement and focus on interprofessional collaborative learning, which might in turn affect the students’ learning favorably, regardless of previous experience. Nevertheless, we report a positive change in self-assessed IPC competencies, adding to the evidence that the ICCAS is responsive and sensitive in measuring change.

Limitations

A retrospective pre-/post-measurement format offers a means of limiting recall bias in self-reported questionnaires (Skeff et al., 1992). Recall bias can occur when participants do not remember their pre-ratings and/or have changed their understanding of the concepts being measured. When the pretest is completed at the same time as the posttest, directly after the intervention, it is possible to reduce this response-shift bias, because the students have better perspectives on their improvement in IPC when they complete the questionnaire (Skeff et al., 1992). With this measurement format, however, some students may wish to maximize the pre- and post-difference. Results from our material show that the students indicate the IPE programs had a moderate-to-large effect. However, the description of each variable supports the finding that the students did not maximize their differences, with pre-score item mean ranging from 3.6 to 4.2 and post-score item mean ranged from 4.2 to 4.6. Therefore, ICCAS has provided us with positive evidence for student learning of interprofessional competencies. We are also aware that the item effect sizes could reflect some differences in sample as well as random coincidences. In future studies, it could be interesting to see if actual repeat-measurement of the questionnaire replicates the change-score evidence found in the pre-post format.

When assessing results from the self-reported questionnaire, we must be mindful that the responding students might be the most receptive and positive to the IPE course they attended. Our sample might not consist of the students that were dissatisfied with the courses, thus enhancing the positive results. Nothing is known of the study’s non-responders, and we acknowledge that their responses could have altered our results. The opportunity to compare our results with three other thorough validation studies strengthened our study. The use of a self-reported instrument without any objective measures of how the students’ abilities changed is also an important limitation. However, until we obtain a validated scoring tool for objective evaluation of IPC, use of self-report instruments is one way to assess competencies across settings.

The educational courses involved in our study varied in length and format. Not all courses contained elements of patient/family involvement, which might explain why items about the involvement of patient/family had the highest missing values. Furthermore, relatively short courses, such as many of the courses in our study, might not represent an arena for the emergence of conflicts. Thus, management of interpersonal conflicts might not arise, resulting in high missing values. Another limitation is that the mean score we refer to is based on courses that vary in duration, content, and participants. We have not compared results based on institutions, courses, or professions to explore possible differences or similarities. However, in an effort to validate the Norwegian version of ICCAS across several different IPE courses, we deemed it necessary to analyze the material as a whole.

Although there are limitations to our study, we found sound evidence for validity of the Norwegian version of ICCAS across settings. We will continue to use ICCAS for evaluation of our educational programs, and to compare results across courses, educations, and sites.

Practical and theoretical implications

One of the strengths of this study is that it contributes further evidence for the validity of ICCAS across several courses. We have also shown support for regarding ICCAS as a questionnaire that is responsive and sensitive in measuring change in students’ competencies.

Our results do not support the theoretical five-factor construct the ICCAS is based on, but rather a single-factor solution for pre- and post-scores. Thus, we question the validity of constructing sub-competencies and learning outcomes to interprofessional capabilities. Use of social learning theories can be one way of understanding interprofessional competence, whereby the students learn through participation in social activities (Lingard, 2012). From a socio-cultural perspective, for example, learning is regarded as a process whereby people reconfigure their relationships in practice. Knowledge and skills that define experts are based on knowledge and values that matter in practice, and are made up of shared experiences or shared understandings (Edwards, 2012). Their competence, therefore, is developed in the context of teamwork, through which multifaceted interactions and actions accumulate into interprofessional competence. Thus, we support Schmitz et al. (2017) in that ICCAS must be analyzed at an overall level, rather than divided into different competencies or items. To our understanding, this does not mean that ICCAS is inappropriate for measuring interprofessional capabilities. On the contrary, our results support the complexity and interconnectedness of interprofessional competencies and indicate that ICCAS can be used to measure the change of overall collaborative interprofessional abilities.

The national requirement for learning outcome descriptions for IPC in Norway can help reduce disparities in education and facilitate IPE (The Norwegian Ministry of Education and Research, 2017a). Institutions are free to choose educational and assessment methods to ensure that the defined final competence is achieved. Therefore, our combined efforts to translate and validate the Norwegian version of ICCAS ensures that institutions offering IPE courses have access to a validated tool with which to assess students’ self-reported competencies in IPC when the requirements for IPC learning outcomes are in effect. A shared, validated measurement tool could prove valuable for measuring students’ self-reported acquired competencies, but also to evaluate and compare educational courses across institutions.

Conclusion

Knowledge and expertise in interprofessional collaboration is a desired outcome highlighted by national white papers and international frameworks to prepare students for an increasingly complex health and social services. Using validated, less context-specific assessment tools that are easy to use and implement helps educators to obtain students’ self-reported achieved competence in IPE. Those results can be useful in evaluating IPE courses, and consequently, be used to develop the best possible IPE for future professionals.

Our study provides validation evidence for the Norwegian version of ICCAS regarding content, response process, internal structure, and consequences. Although validation is an ongoing process and further evidence is needed, especially concerning comparison to objective measures, our study contributes to the accumulated validation of ICCAS to measure students’ acquired, self-reported IPE competencies across several different IPE courses. As with earlier validation studies, we recommend that ICCAS be analyzed at an overall level, supporting the use of ICCAS to measure changes in overall collaborative interprofessional ability.

Declaration of interest

We report no conflict of interest and are responsible for the content and writing of this article.

Acknowledgments

We wish to thank all the students who participated in this study. We also wish to thank Ibrahimu Mdala from the University of Oslo for valuable discussions regarding statistical analysis.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This study is funded by internal funds at the various institutions.

Notes on contributors

Lene Lunde

Lene Lunde, RN, MSc, PhD Candidate, Department of Nursing Science, Institute of Health and Society, Faculty of Medicine, University of Oslo, Norway.

Anders Bærheim

Anders Bærheim, MD, Professor Emeritus, Department of Global Public Health and Primary Care, Faculty of Medicine, University of Bergen, Bergen, Norway.

Ane Johannessen

Ane Johannessen, Associate Professor, Department of Global Public Health and Primary Care, Faculty of Medicine, University of Bergen, Bergen, Norway.

Ingunn Aase

Ingunn Aase, RN, Associate Professor, SHARE - Centre for Resilience in Healthcare, Faculty of Health Sciences, University of Stavanger, Stavanger, Norway.

Kari Almendingen

Kari Almendingen, CD, Professor, Department of Nursing and Health Promotion, Faculty of Health Sciences, Oslo Metropolitan University, Oslo, Norway.

Irene Aasen Andersen

Irene Aasen Andersen, RN, Associate Professor, Department of Health and Caring Sciences, Western Norway University of Applied Sciences, Førde, Norway.

Rutt Bengtsson

Rutt Bengtsson, SW, Associate Professor, Department of Social Studies, Faculty of Social Science, University of Stavanger, Stavanger, Norway.

Sissel Johansson Brenna

Sissel Johansson Brenna, RN, Assistant Professor, Department of Health and Caring Sciences, Western Norway University of Applied Sciences, Bergen, Norway.

Nanna Hauksdottir

Nanna Hauksdottir, PT, Assistant Professor, Centre for Faculty Development, Faculty of Health Sciences, The Arctic University of Norway, Tromsø, Norway.

Aslak Steinsbekk

Aslak Steinsbekk, Professor, Department of Public Health and Nursing, Norwegian University of Science and Technology, Trondheim, Norway.

Elin Olaug Rosvold

Elin Olaug Rosvold, MD, Professor, Department of General Practice, Institute of Health and Society, Faculty of Medicine, University of Oslo, Norway.

Notes

1. University of Bergen (UiB), Western Norway University of Applied Sciences (HVL), University of Stavanger (UiS), Oslo Metropolitan University (OsloMet), University of Oslo (UiO), Norwegian University of Science and Technology (NTNU) and the Arctic University of Norway (UiT)