Perceive, Recall, Plan and Perform (PRPP)-Assessment Based on Parent-Provided Videos of Children with Mitochondrial Disorder: Action Design Research on Implementation Challenges

Figures & data

Table 1. Example of the PRPP-Assessment on a selected task with the criterion and the scoring of the observed performance for stage 1 and an example of a PRPP-Assessment item for each quadrant.

Task	Coloring
Criterion	Colors on the paper, does not have to draw/color something specific; as long as she doesn’t color outside the paper (e.g. not on the table). She also has to put the markers away.
STAGE 1: TASK ANALYSIS
Steps		Errors
Gets marker		No error
Remove cap from marker		Accuracy error
Draws/colors on paper		Accuracy error
Put cap on marker		Accuracy error
Put markers away		Dismission error
Mastery score		1 out of 5 steps correct: 20%
STAGE 2: COGNITIVE STRATEGY USE
Quadrant	PRPP-item (cognitive strategy behavior)	Standardized definition of PRPP-item	Example of ineffective use in-task performance (score =1)
Perceive	Modulates	Spontaneous narrowing and broadening of focus, shifting attention from one part of the task to another	Focuses on the whole area, doesn’t narrow focus to the markers and paper
Recall	Uses body	Knows how to use the body to move, act or behave in a manner that is consistent with what the body is designed to do and meets task or situation requirements	Doesn’t remember how to use her left hand to stabilize the paper, leading to the paper moving around and resulting in coloring outside the paper
Plan	Calibrates	Grades the forces, speed and extend of responses in planning and performing an action, step or expression	Uses too little force to take cap of the marker
Perform	Adjusts	Makes small or large musculoskeletal adjustments to: match the plan or automatic adjustment to muscle stretch	Doesn’t make the necessary body adjustments to stay with the marker on the paper

Figure 1. Overview of the iterative action design cycle (adapted from De Villiers, 2005) used in this research.

Table 2. Overview of the action design research (ADR)-cycles and steps.

		ADR-Step
Aim	Cycle	1) Problem Analysis	2) Develop Improvement	3) Implement Improvement	4) Gather data on situation	5) Analyze data
Optimizing the process of gaining video material resulting in feasible video material for assessing the everyday functioning of children with mitochondrial disorders	1: improving the instruction to parents in order to collect adequate video material	Instructions did not result in feasible video material	Adapting existing written instruction & establish short questionnaire to gather parental information	The instructions and questionnaire were emailed to four parents and four professionals	Written feedback on relevance, comprehensiveness and comprehensibility of the text and comment on the ethical considerations and feasibility of transferring the video material	Feedback was mainly on the level of formulating sentences. The process of transferring video material seemed time consuming. Professionals worried on the ethical considerations.
	2: improving the feasibility of the process of gaining video material and making it ethically approved	The process must be digital safe and ethically approved (in line GDPR. Transfer needs to be easy.	A form was added to give to other persons that were videotaped but not subject of study. The medium arQive was found to be a digital safe way to transfer the videos.	The new procedure was implemented in the MitoRoute. Two weeks before the appointment parents received the invite with all forms.	Three parents were interviewed after the appointment. Two parents were successful in uploading videos, one parent was not.	Interviews were analyzed by content analysis and gave directions to further improving the instructions and the process of gaining video material (see Appendix 1). However, problems with instructions, transfer and storage still existed.
	3: further improve the feasibility of the process of gaining video material	Clarification on the difference between ‘usual care’ and ADR was needed. Problems with storage and usage of arQive needed to be solved.	Instructions were refined based on cycle 2. Invite for care and research were split. arQive invented an app to directly film from the phone or tablet.	The new process and instructions were implemented in the MitoRoute. A rehabilitation center assisted in including participants with similar symptoms.	Four parents and one teacher were interviewed who all were successful in uploading videos.	Interviews were analyzed by content analysis and showed that the instruction was as suitable as possible (see Appendix 1)
Adequate appliance of the PRPP-Assessment based on video material by OTs	4: identify challenges when assessing children’s performance based on parent-provided videos with the PRPP-Assessment	It is unknown which challenges OTs experience when scoring suitable video material.	Not yet possible; first more data collection and analysis necessary	Not yet possible; first more data collection and analysis necessary	Seven PRPP-Assessment certified pediatric-OTs scored parent-provided videos and participated in a focus group moderated by two PRPP-experts.	Analysis uncovered several challenges were uncovered (see Appendix 1) and showed that specific training of OTs was necessary.
	5: identify and overcome remaining challenges after training based on cycle 4	OTs find it difficult to score criterium-based and rule out overall knowledge about normal developmental.	A refresher training for pediatric OTs was designed by three PRPP-experts based on cycle 4.	The refresher training was followed by 13 OTs in the psychometric study (Lindenschot et al, 2022) in which 11 OTs scored five or six children with an average of 14 videos in total.	Nine out of 11 OTs participated in interviews. Seven OTs were individually interviewed. Five OTs (of which three also participated in individual interviews) participated in a focus group interview.	Data was analyzed by inductive coding. Results showed several challenges and recommendations (see Drueten et al., 2021) Recommendations should be integrated in OT-training and in the manual for applying the PRPP-Assessment by parent-provided videos

ADR: action design research; GDPR: General Data Protection Regulation; OT: Occupational Therapist; PRPP-Assessment: Perceive, Recall, Plan and Perform System of Task Analysis; PT: Physical Therapist.

Table 3. Overall lessons learned.

Theme	Topic	Explanation
Instructing parents	Balanced information	Just as much as needed to know what is expected from them. Also, some information on the PRPP-Assessment. Parents should know that they should try not to intervene.
	Choice of activity	Activities should fit with the conditions, represent different occupational areas, and match the PRPP-Assessment criteria (challenging, give room for improvement, and more than two steps in task analysis)
	Method of filming	The used device for filming should not (or minimize) impact on actual performance and child behavior. The whole body of the child should and the objects or people the child interacts with should be captured. The task should be filmed in its whole including instructions (when given).
	Privacy Legislations & Ethical considerations	Everyone captured on the video material needs to give informed consent, should be blurred or removed from the video. Parents should not film their children in a vulnerable situation.
Handling video material	Capacity for storage	Device should be able to capture whole activity, considering an usage between 5 to 700 MB (in average 200 MB will be sufficient)
	Transferring method	Transferring method should be easy, understandable and non-time-consuming.
	Privacy Legislations	Storage and transferring method should adhere to privacy legislations. The process needs to be ethics approved and digital safe.
PRPP-Assessment based on parent-provided videos	Additional information needed from parents	Additional information on the activities is needed and should give knowledge on the familiarity of the activity, the (usual) context, the assignment and the criterion of each filmed activity. Preferably, additional information on the child, the overall functioning and the areas of concern are also known.
	Scoring what you see in relation to the criterion	OTs should only score behavior that they see on the video material and impacts the criterion. When parents intervene too fast, OTs can check the taken over cognitive strategy that is taken in a different step of the activity or write it down with the results.
PRPP-Assessment with children (with very limited functional abilities)	Knowledge of child development	Knowledge on children’s typical development or the observed child (dis)abilities should not impact the scoring as the PRPP-Assessment is criterion-referenced.
	Knowledge of interventions	Knowledge on possible interventions or improvements should not influence scoring of the PRPP-Assessment. The rule of thumb is that you only score what impacts the task and criterion. Also; phase 1 and 2 are connected; when phase 1 is 100%, the score in phase 2 is also 100%.
	Behavior versus activity	When scoring children that have limited abilities and/or who receive a lot of assistance, it can be helpful to formulate a criterion on behavioral aspects and conduct a task analysis (phase 1) on behavior.

De Villiers, M. (2005). Three approaches as pillars for interpretive information systems research: Development research, action research and grounded theory. Proceedings of the 2005 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT '05) on IT Research in Developing Countries (pp. 142–151). South African Institute for Computer Scientists and Information Technologists.

 Google Scholar

Lindenschot, M., Koene, S., Nott, M. T., Nijhuis-Van Der Sanden, M. W., de Groot, I. J., Steultjens, E. M., & Graff, M. J. (2022). The reliability and validity of the perceive, recall, plan and perform assessment in children with a mitochondrial disorder.

 Google Scholar

Drueten, M. V., Eijkelkamp, S., & Grinsven, P. V. (2021). PRPP challenge accepted. HAN University of Applied Sciences.

 Google Scholar

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request. It should be noted that all qualitative data are in Dutch.