The Balancing Act of Assessment Validity in Interprofessional Healthcare Education: A Qualitative Evaluation Study

Figures & data

Table

Participants and Inclusion criteria

Teachers had an active role in IP education, e.g., as an assessor in the IP course; had experience in working in IP practice OR experience in student guidance during IP internships; had experience in conducting IP team meetings and writing IP care plans. Students had passed the IP modules in educational years One and Two; participated in groups containing at least one student from each of the educational programs of physiotherapy, occupational therapy, nursing, speech therapy, and arts therapy; had completed one internship in which they had worked together with (students from) other professions. IP Education and assessment experts had been involved in IP assessment and international research on IP education or IP assessment for at least two years; OR had at least two years’ experience in designing performance assessments in the domain of healthcare education.

Table 1. Interpretation/use arguments for the validity evaluation of the IP assessment prototype.

Aspect of validity	Definition	Interpretation/Use Argument - claims
Authenticity	The way in which assessment tasks align with IP practice	The assessment tasks reflect real world practice. The assessment cases reflect real world client cases. The team composition in the assessment task reflects a real-world team composition. The set of assessment tasks and client cases used are as diverse as the types of tasks a student encounters in real world IP practice.
Scoring	The way in which all aspects of the IP performance are scored (i.e., compared to a notional ideal) during an assessment and the assessment procedure matches the assessment goals.	The rubrics, criteria, and scoring rules are clear to students and assessors. The criteria used in the assessment reflect relevant IP competencies. The construct is at the appropriate level of difficulty. The assessment prepares for an IP internship. All students are able to participate in the assessment task regardless of professional background.

Table

Group interview guide questions

Assessment task Does this task, conducting an IP team meeting, reflect real practice? What aspects of this assessment mean that it corresponds / does not correspond to a team meeting as happens in practice? Does the diversity of the team composition in this assessment correspond to the team composition you encounter in practice? Which profession(s) should also participate, so that the team composition is more in line with the team composition you encounter in practice? Do the instructions provided with this assessment give sufficient guidance to assessors? What else do assessors need to be able to assess students? Preparation for internship Does this task, conducting a team meeting, sufficiently prepare students for participating in an IP team meeting during their internship? What causes this assessment to sufficiently / insufficiently prepare students to participate in an IP team meeting during their internship? Client cases Are the client cases used in this assessment in line with a client case as you would encounter in practice? What makes the cases correspond / not correspond to client cases as you encounter in practice? Are the two client cases sufficiently diverse, given that cases differ in real practice? What kinds of cases are missing? What type of case should be included in the assessment concerning writing a care plan? Are there enough leads in the case for all the professions involved? For which professions does this case provide sufficient starting points and for which does it not? How can the cases be improved? Assessment procedure Are the assessment criteria clear? Which assessment criteria are clear / unclear? Is the assessment procedure clear? What aspects of the assessment procedure are clear / unclear? Can you use this assessment procedure to decide whether a student is able to conduct an interprofessional team meeting? Can you use this assessment procedure to decide whether a student is able to write an interprofessional care plan? What makes it (im)possible to determine the individual competence of the student? Can you use this assessment procedure to decide whether a group of students can conduct an interprofessional team meeting? Can you use this assessment procedure to decide whether a group of students can write an interprofessional care plan? What makes it (im)possible to make a decision about the group competence of the students?

Table 2. Analysis matrix.

Validity aspect	Evidence for:	Threat to:
Authenticity	Authentic tasks The assessment task, the IP team meeting, reflects real practice Authentic client cases Client cases reflect real client cases as seen in IP practice Authentic team composition The student team composition reflects a real IP team composition	Authentic tasks The assessment does not represent a real IP team meeting An IP team meeting does not reflect real practice in some healthcare and wellbeing settings Using an IP care plan does not always reflect real-life IP practice Authentic client cases The paper format of the client cases does not reflect real client cases as seen in IP practice Authentic team composition One or more professions are missing in the IP team composition compared to real IP practice
Scoring	Clarity of the assessment The assessment criteria are clear The assessment instructions are clear Alignment between assessment purpose and procedure The group product, the IP care plan, can be assessed using this assessment procedure Appropriateness of difficulty The assessment task is a good first introduction to an IP team meeting Variability of the assessment cases The assessment cases are sufficiently diverse for all students from different professions to participate in the assessment task and thus be scored on their performance	Clarity of the assessment The assessment procedure lacks clarity in the purpose of the assessment (i.e., insufficiently identifies what has to be assessed) The assessment criteria lack clarity in the difference between sufficient and excellent The decision rules lack clarity in determining a pass or fail Alignment between assessment purpose and procedure The IP collaboration process of participating in an IP team meeting cannot be assessed using this assessment The individual student cannot be assessed using this assessment The team of students cannot be assessed using this assessment Appropriateness of difficulty Students need more practice in IP collaboration before it can be assessed One assessment is insufficient to determine IP competencies Students need more practice in IP collaboration before they enter an internship The task of writing an IP care plan is too difficult for second-year students Variability of the assessment cases One or more professions lack professional connection to the cases, hindering them from participation in the assessment task and ability to be scored on IP performance The student team composition is unbalanced, leading to groups with some professions better represented than others. Assessors The role of assessor should be executed by others rather than left exclusively to teachers

Figure 1. Evidence for and threats to authenticity.

Figure 2. Evidence for and threats to scoring.