Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

Figures & data

Figure 1. Example of a generic scoring form for EPA-based assessments.

Figure 2. General flow of workplace-based assessment data to support summative entrustment decisions making.

Table

Required elements	Recommended elements
Learner name or ID Rater name or ID Date/time of the assessment EPA of interest Specialty, rotation, clerkship Level of supervision provided and/or recommended of ES-scale Narrative feedback or learning goal	Level of complexity of case or situation Relevant domains of competence for feedback, if defined for the EPA Space for written, or, preferably, recorded feedback Optional explanatory texts (about the EPA, about supervision levels, about criteria)

Figure 3. Procedure and data flow in one MSF procedure, annually or semi-annually performed.

Figure 4. Schematic excerpt from an individualized portfolio, showing expected increase in autonomy for various EPAs across four postgraduate years (ten Cate 2014) (reprinted with permission).

Table 1. A Checklist of considerations for review when choosing or designing WBA mobile technology.

Considerations Prior To Review or Design
Purpose	Will the assessments be used formatively (low-stakes), summatively (high-stakes), or both? Will qualitative comments/feedback be captured? Will growth from feedback be needed/tracked? Will quantitative data be collected only? Will progression toward a specific outcome be tracked? Will data be linked to a dashboard containing data from other sources for an overall snapshot of a trainee? Will there be a need for trainees to reflect on their performance? Will the assessment be top-down only or also bottom-up (evaluation of teaching interaction)? Will digital badging or certification processes be needed? How important is compatibility with other systems for collecting and aggregating assessment data?
Context	Will the WBA technology be used in undergraduate medical education, graduate medical education, other educational settings, or a combination (e.g. longitudinal assessment strategy)? Will the WBA technology be used at a single institution, multiple institutions, or as part of larger national or international program? Will the assessments be captured predominantly in a busy clinical setting with high patient traffic, slower clinical setting, or varied settings? Will the assessments be captured in person or remotely? Will the assessments be captured real-time in the clinical setting, or will it be asynchronous and postponed? What network infrastructure and capabilities are available? Is offline capture available? Is the upload of offline data manual or automatic? What technical aspects need to be covered: e.g. iOS vs. Android or both operating systems, only mobile or also web-version? What web browser(s) are supported? Will the users use their own device, or will the training program provide devices? What are the data requirements as WIFI may not be free/accessible in all clinical settings? What institutional resources, personnel, and budget is available?
Users	Who will be the users (e.g. students, residents, preceptor, master assessor, clinical director, other staff as part of the healthcare team, etc.)? Who will be assessors? Only supervisors or will trainees be able to self-assess their performance? Combination of both? Who will be reviewing/accessing the data or reports from the data (e.g. Clinical Competency Committee)? National institutions (e.g. national boards of examiners)
Considerations During the Review or Design
User Interface	Will the technology be offered as a mobile application only? Will users be able to enter data or access information through a desktop version? Will mobile or desktop versions be different for each stakeholder (e.g. trainees – mobile only and evaluators- desktop or mobile)?
System Adaptability	Can multiple supervision scales be used (e.g. ability to use ten Cate scale, Chen scale, Ottawa scale, Zwisch scale, etc.(ten Cate et al. 2020))? Does the system support instant additions/modifications (pushed immediately to users) to assessment elements, scales, descriptors, etc.? Can the branding be changed to support institutional branding? Can the system be modified with the look and feel of the institution?
User Experience	How many total screens will the user have to navigate for main functionality (“number of clicks”)? How intuitive is the workflow? Do open comment areas allow dictation and speech-to-text functionality (e.g. through a microphone built into the system or the microphone on the mobile device keyboard)? How much time do users have to spend in documenting WBAs (e.g. 30 s)? Can students share data easily with others (e.g. assessors)? How is the data shared (e.g. email, Quick Response (QR) code, etc.)? Can students easily see data submitted about them? Can assessors share data about a student with another assessor or staff person)? Can stakeholders see this data through the mobile application, browser, or both? Are further definitions of terms or scale attributes readily available through dropdowns or pop-outs?
Security and Validation	What will be required for authentication? Will integration with an institutional authentication system be needed? Will the authentication be self-contained as part of the system? Will authentication synchronization be available? Will import or bulk uploads be available? Can users be added on the fly? What will be the method of validation (if any) of the assessor? For example, signature, picture of badge, QR code, finger scan, etc. Will the assessor be entering an assessment on the trainee’s device, their own device, a device provided by the institution, or other method? Who will own the data? The trainee? The institution? National authorities? Where will the data be stored? On the smartphone? Local server? In the cloud?
Reporting	What reports will be needed (e.g. total EPAs completed by rotation, average growth curve by EPA, EPA Submissions, etc.)? What filters will be needed for each report (e.g. trainee, assessor, course/rotation, EPA, formative versus summative, supervision scale, case complexity, elements of trustworthiness, start and end dates, etc.)? Will open comments be exported verbatim or will a report with thematic analysis be needed? What formats for data exportation will be needed (e.g. xls, cvs, pdf, doc, etc.)? Will connection to another reporting database/portfolio be needed?
Other considerations	Can the trainee complete a self-assessment? What other elements can be assessed (e.g. elements of trustworthiness, case complexity, etc.) that might inform entrustment decisions? Does the application need or have other uses (e.g. documentation of case or procedure logs, view of course/rotation schedules, multi-source feedback, other assessments, etc.)?
Considerations Following the Review or Design
Timing and selection of assessments	How often will assessments be collected? Do users select the elements to be assessed or are the required elements outlined in a syllabus or other formal document? How often is feedback given/communicated? Is there a set meeting time for feedback (e.g. quarterly) or is feedback given on the fly as it is be entered or dictated into the system?
Training	How will training be conducted? Who will provide the training? Who will need to be trained (e.g. student/assessor/clinical director/ administrator)? What type of model will be used (e.g. direct, scaffolded, train the trainer, etc.)? Will modules need to be included as part of the mobile application?
Support Structure	What will the support structure look like? Will the support structure be direct, scaffolded, handled by the vendor or a third party, etc.? What will be the mechanism for communicating support (e.g. website, email, link in the mobile application, QR code, combination)? How is technical support and ongoing updating of the application guaranteed?
Legal Issues	Are there any legal issues that need to be considered, addressed in policy, discussed for shared understanding? Are policies in place that outline requirements, expectations, licensing, copyright, etc.?
Evaluation	How will the implementation of MT be evaluated? (e.g. using frameworks of implementation science)

Table

1. Lawfulness, fairness, and transparency	Processing must be lawful, fair, and transparent to the data subject
2. Purpose limitation	You must process data for the legitimate purposes specified explicitly to the data subject when you collected it.
3. Data minimization	You should collect and process only as much data as absolutely necessary for the purposes specified
4. Accuracy	You must keep personal data accurate and up to date
5. Storage limitation	You may only store personally identifying data for as long as necessary for the specified purpose
6. Integrity and confidentiality	Processing must ensure appropriate security, integrity, and confidentiality (e.g. by using encryption)
7. Accountability	The data controller is responsible for being able to demonstrate GDPR compliance with all of these principles

ten Cate O. 2014. AM last page: what entrustable professional activities add to a competency-based curriculum. Acad Med. 89(4):691.

 PubMed Web of Science ®Google Scholar

ten Cate O, Schwartz A, Chen HC. 2020. Assessing trainees and making entrustment decisions: on the nature and use of entrustment-supervision scales. Acad Med. 95(11):1662–1669.

 PubMed Web of Science ®Google Scholar