The reliability and validity of the perceive, recall, plan and perform assessment in children with a mitochondrial disorder

Abstract

Purpose

To investigate the reliability and validity of the Perceive, Recall, Plan and Perform System of Task Analysis (PRPP-Assessment) by parent-provided videos of children with mitochondrial disorders.

Methods

Videos of 13 children performing 3–7 activities each were the subject of study, resulting in 65 activities. These were scored using the PRPP-Assessment by 11 raters. Internal consistency was calculated with Cronbach’s alpha. Intra-rater reliability was evaluated by Bland-Altman Plots on 44 PRPP-Assessment scores. Inter-rater reliability was calculated with ICC_Agreement on 128 PRPP-Assessment scores. Construct validity was assessed by comparing the PRPP-Assessment scores to the Canadian Occupational Performance Measure using Cohen’s Kappa. PRPP-Assessments scores were evaluated with a multi-faceted Rasch Analysis.

Results

Internal consistency was high (0.92). Intra-rater reliability was sufficient to be good (92–96% within the 95%-Limits of the Agreement). The ICC_Agreement for stage 1 Mastery Score showed acceptable inter-rater reliability (0.646). Stage 2 of the PRPP-Assessment showed low ICCs due to a lack of variability within the sample. Four out of six hypotheses on validity were accepted. Rasch's analysis demonstrated sound goodness-of-fit, and supported the validity of the PRPP-Assessment.

Conclusion

The PRPP-Assessment by parent-provided videos in this heterogenic group showed sufficient to good psychometric properties. In practice, careful task selection and formulating criterion is recommended.

Implications for Rehabilitation

• PRPP-Assessment by parent-provided videos is reliable and valid in the complex, heterogenous group of children with mitochondrial disorders

• The PRPP-Assessment is suitable for children with mitochondrial disorders as it showed to contain familiar, functional and meaningful tasks and activities that fit with their level of functioning

• Professionals should be aware that parents might not be used to the criterium-based frame of reference of the PRPP-Assessment

• When applying the PRPP-Assessment, it is recommended to be careful in task selection and formulating the criterion.

Keywords:

• Children
• mitochondrial disease
• assessment
• occupational therapy
• activities
• psychometric properties

Introduction

Mitochondrial disorders are rare diseases that influence cellular energy metabolism. These disorders are one of the most common inherited errors of metabolism that can be caused by mutations in more than 230 different genes [1,2]. This genetic heterogeneity is also reflected in the large range of symptoms and impairments associated with mitochondrial disorders, varying from motor impairments (such as muscle weakness or balance problems) to cognitive impairments (such as concentration problems and intellectual disability) [3–6]. These impairments have a tremendous impact on daily functioning and influence participation in school, leisure activities, the neighborhood, and the community [6]. It is a challenge for clinicians to assess the functioning of children with mitochondrial disorders with ‘standardized’ assessments due to the heterogenicity of the population.

Best practice should include assessment of the child’s ability to perform meaningful activities and evaluate the specific motor and/or cognitive difficulties that impact task performance. Ideally, assessments should inform how well children can participate according to their own needs, in line with personalized and value-based care [7]. Furthermore, norm-referenced tests do not provide the information needed for personalized and value-based care, and most existing tests are not applicable due to the heterogeneity of the group or that these children do not perform activities in the ‘standardized’ way [8].

In contrast, criterion-referenced measures focus on a more individualized outcome. Each child’s performance level is not compared to others [9] but is assessed against what is required for them to achieve their personalized goal. Criterion-based assessments are encouraged in pediatric occupational therapy as they can contribute to client-centered practice and allow self-determination and self-definition [10,11], and overcome difficulties in standardized testing of children with atypical development [12–14]. Hence, information gained from criterion-referenced assessments can guide personalized treatment, which makes the outcome more clinically relevant. Therefore, criterion-referenced tests in which children are compared to a personalized criterion seem more appropriate for the heterogenic group of children with mitochondrial disorders [9].

A suitable criterion-referenced assessment for measuring the quality of performance in children with limited cognitive and communication abilities could be the Perceive, Recall, Plan, and Perform (PRPP)-System of Task Analysis [15]. This assessment is used by occupational therapists (OT) to gain insight into occupational performance by observing children performing meaningful activities. When administering the PRPP-Assessment, the OT observes and analyses in a structured way what children do, how they do it and which cognitive strategies they use. Cognitive strategies are essential to human functioning as they are the thinking processes required to think through task performance [16,17]. The cognitive strategies that are described in the PRPP-Assessment align with the information-processing model presented by Lerner [17]. Assessing cognitive strategies is not only relevant for children with limitations in cognition but could also be useful in assessing how children use cognitive strategies to cope with motor limitations or fatigue. Prior to the PRPP-assessment, the client and the therapist, and in the case of children with the involvement of a parent, agree on a meaningful task to be observed and what a satisfactory level of task performance would be in their unique context (the criterion). The PRPP-Assessment can be performed in person or via observation of parent-provided videos [15,18,19] which aligns well with current COVID-19 restrictions.

The psychometric properties of the PRPP-Assessment have been studied in several homogenous and heterogenic target groups, for example, adults and children, with and without disabilities [20,21] children with learning difficulties [22], school-aged children [23], children with autism and intellectual disabilities [24], and adults with traumatic brain injury [25–27], Parkinson’s’ disease [28,29], schizophrenia [30] and dementia [31,32]. Based on these studies, psychometric properties of the PRPP-Assessment have been demonstrated to be sufficient to good.

Because of the multidimensional and heterogeneous limitations in children with mitochondrial disorders, the PRPP-Assessment seems a promising tailor-made instrument to evaluate their performance in everyday activities in their own context which can enhance personalized and value-based care. However, the psychometric properties of the PRPP-Assessment in a heterogeneous group of children with physical, cognitive and communication disabilities are not known. Therefore, the objective of this study was to evaluate the internal consistency, inter- and intra-rater reliability and construct validity of the PRPP-Assessment for children with mitochondrial disorders by parent-provided videos.

Methods

This study used a cross-sectional design using quantitative data from standardized ratings of parent-provided videos of children who performed meaningful everyday activities. Parents provided videos of three to seven activities per child, thus 65 videos of occupational performance in total. Performance was rated with the Canadian Occupational Performance Measure (COPM) by children and parents, and the videos were rated by OTs using the PRPP-Assessment. In addition, parents' and expert opinions were collected.

Participants

Children

Videos of children with a genetically confirmed mitochondrial disorder were included in this study. Dutch children, aged between two and 18 years, who were registered patients in the specialist center for mitochondrial medicine of the university hospital, were eligible for participation in the study. No exclusion criteria considering physical, cognitive and communication (dis)abilities were formulated to include a clinically representative sample. The treating physician sent a patient information letter to 25 eligible children and their parents; 13 parents responded positively. All children lived at home with their parents. Children were in the age of preschool (one child), primary school (five children) and secondary school (six children). Age, gender, functional capacities, and the underlying etiology (see Table 1) varied. To differentiate between the functional capacities of the children, three profiles were used; global low functioning, low cognitive functioning with moderate-to-normal ambulation (referred to as ‘mixed’), and, global moderate functioning [33]. The functional capacities were extracted from patient files and based on age-related norm-referenced tests.

Table 1. Example of the PRPP-Assessment on a selected task with the criterion and the scoring of the observed performance based on each quadrant and for a PRPP-Assessment item.

A girl with a mitochondrial disorder wants to independently drink a glass of milk and put the glass back on the table, without spilling or making a mess.
Quadrant	PRPP-item (cognitive strategy behaviour)	Standardized definition of PRPP-item	Example of ineffective use in-task performance (score =1)
Perceive	Modulates	Spontaneous narrowing and broadening of focus, shifting attention from one part of the task to another	Focuses on the whole kitchen-area, doesn’t narrow focus to the glass, which hinders picking up the glass.
Recall	Uses body	Knows how to use the body to move, act or behave in a manner that is consistent with what the body is designed to do and meets task or situation requirements	Leaves arms on the table and tries to drink with moving head to glass/table, which leads to spilling milk.
Plan	Calibrates	Grades the forces, speed and extend of responses in planning and performing an action, step or expression	Uses too little force to tilt glass, resulting in making a mess.
Perform	Adjusts	Makes small or large musculoskeletal adjustments to: match the plan or automatic adjustment to muscle stretch	Doesn’t make the necessary body adjustments to limit spilling.

Occupational therapists

Dutch pediatric occupational therapists (OTs) certified in the PRPP-Assessment were invited to participate in this study. Inclusion criteria were (1) experience in pediatric occupational therapy, (2) frequent usage of the PRPP-assessment with children (at least once a month) and (3) being willing to complete an additional four-hour training session. This refresher training was executed by two experienced OTs and focused on scoring the PRPP-assessment in children with the mitochondrial disorder by parent-provided videos, and on standardized scoring rules of the PRPP-Assessment and focused on additional points for scoring video material (in comparison to ‘live observation’) as identified in previous research [18]. After training, 11 OTs scored the video material. OTs had an average age of 43 years (range 31–54), an average Pediatric-OT work experience of 15 years (range 5–23 years), and the mean PRPP-Assessment certification was four years (range 1–7 years). As OTs rated child performance with the PRPP-Assessment, they will be called ‘raters’ in this study.

Measures

PRPP-Assessment

The PRPP-System of Task Analysis (PRPP-Assessment) is a two-stage, criterion-referenced measure instrument that uses task analysis methods to examine the effectiveness of cognitive information processing during the performance of daily activities. The PRPP-Assessment incorporates a task analysis procedure in stage one, in which an observed task is divided into relevant steps and types of errors are measured. The level of independence in performing the task can be calculated by dividing the number of error-free steps by the number of all steps of the task. Stage two of the PRPP-Assessment uses a cognitive task analysis and incorporates 35 items divided into the subscales of the PRPP-Assessment, which connects to a specific conceptual stage of information processing (see Figure 1). The 35 strategies include attention and sensory perception (Perceive Quadrant), memory (Recall Quadrant), response planning and evaluation (Plan Quadrant), and performance monitoring (Perform Quadrant). Each cognitive strategy is criterion-referenced and evaluated on a three-point scale indicating how effectively the child used that cognitive strategy (score 1 for ineffective use, score 2 for questionable use or slow use, and score 3 for effective use); resulting in a total score between 35 and 105 (higher scores indicate more effective cognitive strategy application), as well as a PRPP-Assessment quadrant score (max 24 for performance quadrant or max 27 for other quadrants). To illustrate, Table 2 provides an example of a selected task with the criterion and the scoring of the observed performance based on each quadrant and for that PRPP-Assessment item. All scores can be transferred to percentages using the equation: the number of points scored divided by the total number of points possible [34,35].

Figure 1. They perceive, recall, plan and perform (PRPP) system of task analysis conceptual model [15,34].

A pie chart that divides the process of information processing into the four quadrants Perceive, Recall, Plan and Perform, surrounded by the 12 sub-quadrants and the corresponding 35 items, so-called descriptors.

Table 2. Demographics children.

Characteristics	Total group	Global low functioning	Low cognitive functioning and moderate-to-normal ambulation	Global moderate functioning
Participants, n	13	5	4	4
Age, years, range (median)	3–17 (14)	3–16 (10)	6–17 (14, 16)	8–16 (15, 16)
Boys, n	7	3	1	3
nDNA/mtDNA, n	5/8	5/0	2/2	1/3
Genetic defects, gene (mutation)		FBXL4 (c.1361A > C); OPA1 (c.910C > T); PDHX (deletion); SDHA (c.64-2A > G); SERAC1 (c.1765G > A)	MTATP6 (m.8993T > G); OPA1 (c.910C > T); RARS2 (c.442A > G- c.1519G > A); MTTL1 (m.3243A > G)	MTATP6 (m.8993T > G); MTND5 (m.13094T > C); TAZ (c.788_794del); MTTL1 (m.3243A > G)

FBXL4 = F-box and Leucine-rich repeat protein 4; MTATP6 = ATP synthase 6; MTND5= Mitochondrial Complex I, subunit ND5; MTTLI = Transfer RNA, mitochondrial, leucine 1; OPA1= optic atrophy 1; PDHX = Pyruvate dehydrogenase complex, component X; RARS2= Arginyl-tRNAse synthetase 2; SDHA = succinate dehydrogenase complex, subunit A; SERAC1 = Serine active site-containing protein 1; TAZ = Tafazzin.

In this study, the Dutch version of the PRPP-Assessment was used with scoring criteria according to the PRPP-Assessment manual, that is instructed in the five-day PRPP-Assessment course for occupational therapists [36]. The PRPP-Assessment was administered based on video material provided by the parents. The process of gaining eligible parent-provided video material and standardized scoring of video material has been previously studied [18].

Canadian occupational performance measure (COPM)

The COPM [37] is a reliable and valid instrument for measuring the perceived quality of occupational performance and level of satisfaction [38]. The COPM was rated by parents and if possible, separately by the child. For the children, the child version of the COPM was used [39]. In this study, the videotaped activities were assessed with a short version of the COPM. Both versions score activities on importance, performance, and satisfaction on a scale ranging from 1 to 10. Initially, it was planned to score these three domains for each activity. However, the level of parental burden due to COVID-19 and associated additional caring responsibilities required a change in the data collection and resulted in the focus on scoring performance ratings only.

Expert opinions

Prior to the data-analysis five experts, individually, formulated statements focusing on the PRPP-descriptors to identify which would be the most difficult for children with a mitochondrial disorder. To guide them in making the statements they used the hierarchy of the descriptors for children in general [20]. Experts were an OT, an OT-researcher, a pediatric rehabilitation doctor, a clinical geneticist, and a psychologist. All experts were part of the multidisciplinary care around children with a mitochondrial disorder.

Parent opinions

To take into account the parent’s view on the validity of the PRPP-Assessment, parents were asked if they recognized their child in the PRPP-report that was produced following the PRPP-Assessment. The main question to parents was: “Is the PRPP-report consistent with your observations on the everyday functioning of the child, and especially with the focus on the child’s abilities and difficulties?” Answering options were: yes, partially, and no. Parents were asked to elaborate on their answers.

Procedure

Parents were contacted by the first author. After informed consent, they received a written instruction to videotape everyday activities. This instruction was optimized based on a case report [18] and action-design-study [19] with parents and OTs. The instruction contained the assignment to videotape at least three meaningful activities of self-care, play or school activities that are challenging in everyday life. Challenging was explained by mentioning that it required ‘thinking’ of the child. Parents were encouraged to choose a device that would interfere the least with usual performance. They were instructed to film the ‘whole’ activity, from preparing the activity (for instance, gathering the pencils for coloring) to finishing the activity (for instance, clearing up the table after coloring). They were instructed to film the whole body and environment that was relevant to the performance. Ethical considerations and privacy legislation were explained and adhered to by using safe, online storage. This system was used to view the videos: parents could only see their own videos, and raters could only see the videos they were assigned to. When videos had been uploaded, parents and the researcher (ML) discussed the criterion, by questioning: “For this specific task, what level of performance or ability would be satisfying or acceptable for you?” and “In your opinion, what level of performance or ability would be satisfying or acceptable for your child?” Where possible, children themselves were also involved in setting the PRPP-Assessment criterion from their perspective.

Parents provided videos of three to seven activities per child, thus 65 activities in total. Within two weeks of videoing their child, the researcher (ML) contacted parents to collect the COPM-data. This data has been collected at the home of the child/parent or (due to regulations around COVID-19) by telephone, video call or e-mail. All parents of the 13 children were assigned a COPM-performance score for the different activities, which resulted in 65 scored activities, exceeding the minimum requirement (n = 50) to achieve sufficient power for validity measurement [40]. Due to COVID-19 regulations in combination with communication difficulties of children, it was not feasible to collect data from all children only four children on 20 scored activities.

Next, one rater scored all the videotaped activities of one unique child with the PRPP-Assessment. This rater performed the task analysis by conducting Stage One of the PRPP-Assessment, which resulted in the fixed activity steps for the other raters. Next, two random videos of that child were randomly assigned to four other raters. Videos were randomized in such a way that each of the 11 raters scored videos of six different children which is likely to give good power for reliability measurement [41]. The order of the children and videotapes was different for each rater.

Lastly, parents received a copy of the PRPP-Assessment Report completed by the first rater. Parents were called within two weeks after receiving the PRPP-Assessment Report to ask if the child described in the report was consistent with their own observations and understandings of their child and their occupational performance. All 13 parents gave their opinion on recognition of their child in the PRPP-report.

Intra-rater reliability

For intra-rater reliability, all 11 raters scored two videos of one unique child twice, with an interval of six weeks, a total of 44 videos. In order to avoid remembrance of first-time scoring, they scored the videos of the remaining children for inter-rater reliability in the six weeks between re-scoring for intra-rater reliability. The raters were instructed to score independently, without looking at previous scoring.

Internal consistency

The scores of the first rater were used to determine internal consistency, resulting in 65 PRPP-Assessment scores.

Inter-rater reliability

The two random videos of each child were rated by five different raters. These were used for inter-rater reliability. Due to two missing scores, 128 PRPP-Assessments scores could be used.

Construct validity

Construct validity was assessed on three types, based on the COSMIN principles [42]: 1) Convergent validity was assessed by comparing the PRPP-Assessment scores to the Canadian Occupational Performance Measure (COPM) and the opinion of parents, and both 2) Structural validity, and 3) Known group validity, were evaluated with Rasch Analysis and compared with expert opinions and the underlying information processing model that informs the PRPP-Assessment.

Data analysis

Tasks performed by children in each video were categorized using the Occupational Performance Model Australia (OPMA; the conceptual model that is the foundation for the PRPP-Assessment) [43], the International Classification of Functioning, Disability and Health for Children and Youth (ICF-CY) [44] and previous research on the meaning of activities [8,33]. An overview of the task categories and all the activities in the sample are presented in Supplementry Appendix 1. The ICF-CY categories were used in analyzing the data. For the overall performance of children, the scores were summarized with descriptive statistics for the total group of children and the subgroups per functioning profile.

Intra-rater reliability

For intra-rater reliability, the Level of Agreement was determined with a Bland-Altman Plot and calculated for PRPP-Assessment stage 1 mastery score, PRPP-Assessment stage 1 error score (percentage of the overall number of task steps), PRPP-Assessment stage 2 sum score, and PRPP-Assessment (sub)quadrants score. Reliability is acceptable when 95% of the differences scores are within the 95% limits of agreement [40].

Internal consistency

Internal consistency was determined only for stage 2 as for stage 1 it is not possible due to the various task analyses. Internal consistency was determined with Cronbach’s Alpha and estimated at 0.7 or higher for an adequate level of consistency [40]. Two calculations were made; one with all 35 descriptors (items) and one where the descriptor ‘matches’ is excluded as this descriptor does not apply to all activities, and therefore can get ‘no score’.

Inter-rater reliability

For inter-rater reliability, the ICC_agreement and ICC_consistency with a two-way random model, single measures, were determined for PRPP-Assessment stage 1 mastery score, PRPP-Assessment stage 2 sum score, and PRPP-Assessment quadrants score. A priori, an acceptable agreement was defined as 0.6 or higher [45]. Data were analyzed using SPSS for Windows version 26.0 (New York, IBM corporation, 2019).

Construct validity

Following guidelines of the COSMIN [42,46,47,48], a priori hypotheses were formulated to support construct validity by assessing convergent, structural, and known group validity. Hypotheses are presented in Table 3. A traditional statistical procedure was conducted for convergent validity (hypothesis 1 and 2) using SPSS for Windows version 26.0 (New York, IBM corporation, 2019). To make rank-correlation possible, COPM and PRPP-mastery data were ranked in three categories: inadequate, sufficient, and good. The COPM performance categories were based on the Dutch grading system: inadequate = 0–4, sufficient = 5–6, good = 7–10. For the PRPP mastery score, categories were based on knowledge of the assessment and set on inadequate = 0–40, sufficient = 41–84, and good = 85–100. Rank-correlations were calculated using Kappa, where a kappa of 0.4–0.6 is seen as moderate, 0.6–0.8 as substantial, and above 0.8 as almost perfect [45].

Table 3. Hypotheses are formulated a priori.

Aspect of construct validity	Hypotheses
Convergent validity	1. The rank-correlation between the COPM-performance score (parent rated) and the PRPP-assessment mastery score (stage 1) calculated by kappa, shows a moderate positive correlation (0.50 < r > 0.70).
	2. The rank-correlation between the COPM-performance score (child rated) and the PRPP-assessment mastery score (stage 1) calculated by kappa, shows a moderate positive correlation (0.50 < r > 0.70).
	3. The child-recognition in the PRPP-reports expressed by parents is positive (‘yes’) in at least 80% of the cases
Structural validity	4. Item-fit statistics are considered acceptable (mean-square range 0.5–1.7, infit and outfit statistics are between −2 and +2)
	5. Expert-statements on the difficultness of descriptors for children with mitochondrial disorders fit with the Rasch calculated hierarchy of hardest to easiest descriptors (items) for at least 80%
Known group validity	6. The difference in efficiency of cognitive strategy application between the three profiles is noticeable based on the Rasch generated interaction between items and profile

For structural and known group validity, all data was entered into excel and then analyzed using FACETS 3.71.4. A four-faceted Rasch model was generated with facets measuring children, tasks, raters, and the PRPP-Assessment items [49]. The RASCH-generated hierarchy of the four facets was visually presented using the ‘Wright map for the many-facets Rasch analysis’ (in FACETS called the vertical rulers) to give an overview of how each facet is calibrated and to ensure that all facets perform sufficiently predictably to be considered a unidimensional construct [50]. Rasch analysis converts ordinal-level ratings to interval-level measures, the so-called logits. When interpreting Rasch analysis, the mean of the person's abilities, task difficulties, rater severities, and item difficulties, is adopted by default as the 0 points. To interpret, person locations are plotted so that any person has a 50% probability of succeeding with an item located at the same point on the logit scale. For example, a person with an ability estimate of 0 logits has a 50% probability of succeeding (or failing) on an item that has a difficulty estimate of 0 logits. That same person would have a greater than 50% chance of succeeding on items less difficult than logits and less than a 50% probability of succeeding on items that are more difficult 0 logits.

Next, each PRPP item was examined for “fit” with the Rasch mathematical model, considering both infit and outfit, as mean square values and standardized z-scores. For clinical observations, a mean-square range of 0.5–1.7 is considered acceptable, with normalized z-score [51]. Infit and outfit statistics are also reported in normalized form (z), where their expected value is 0, and standardized z values less than −2 or greater than +2 are generally interpreted as having less compatibility with the model than expected (p < 0.05). Negative values indicate less variation than modelled. While positive values indicate more variation than modelled, and greater than +2 means they are too erratic to be useful [50]. Generally, positive values are considered more problematic during interpretation.

In addition, the Rasch calibrated hierarchy of 35 PRPP items (stage 2) was generated following the procedure of Bond & Fox [50]. Interaction plots constructed by FACETS were used to visually present the differences in the efficiency of cognitive strategy application between the three. Due to the small sample size, a between-group analysis of variance was not performed. The article of Nott & Chapparo [26] can be used for more detailed information on analyzing PRPP-Assessment scores with Rasch.

Ethical considerations

All procedures performed in this study were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards [52]. The ethical board of the regional research committee provided their approval (ethical board approval number 2019-5493). All parents signed informed consent and all raters signed a confidentially agreement. Where indicated, the children also signed the informed consent to demonstrate their volition/agency [53].

Results

Overall for all 65 activities, the mean score of stage 1 performance mastery was 60% (SD 25%), and for stage 2, the total score on the use of information processing strategies was 85% (SD 12%). Table 4 shows detailed information. Children scored relatively high on the Quadrant Recall (93%) and the Quadrant Perceive (91%) while the Quadrant Plan (76%) and Quadrant Perform (79%) received lower scores. Most of the errors in stage 1 were due to inaccurate performance (31%, SD 25%), whereas the other errors (repetition, omission and times) ranged from two to four percent.

Table 4. Overall scores on PRPP-Assessment.

	Total group	Subgroups based on functioning profiles
		Global low functioning	Low cognitive functioning with moderate-to-normal ambulation	Global moderate functioning
Children, n	13	5	4	4
Activities, n	65	26	20	19
Mastery score range (median); %	0–100 (60)	20–100 (55)	0–100 (60)	0–100 (71)
Total score range (median); %	45–100 (89)	45–100 (78)	79–96 (90)	70–100 (90)
Quadrant perceive score range (median); %	43–100 (100)	43–100 (82)	81–100 (100)	81–100 (100)
Quadrant recall score range (median); %	44–100 (100)	44–100 (93)	81–100 (100)	85–100 (100)
Quadrant plan score range (median); %	33–100 (81)	33–100 (67)	52–96 (89)	41–100 (81)
Quadrant perform score range (median); %	33–100 (79)	33–100 (75)	67–96 (79)	63–100 (83)

Intra-rater reliability

As shown by the in Bland-Altman plots in Figure 2, all stage one mastery scores lay within the 95% limits of agreement (Figure 2a), in which for 15 out of 22 repeated ratings the difference of scores was zero. For the stage, two Sum score, 96% of all observations lay within the 95% limits of agreement (Figure 2b). In six out of 22 scores, the difference was zero, and 12 scores differed by less than 5%. For the Error scores (stage 1), and the quadrant scores (stage 2), the percentage observations within the 95% limits of agreement are 93%, and 92%, respectively. A more detailed overview of the intra-rater reliability statistics is presented in Table 5.

Figure 2. Bland-Altman Plots for the PRPP-Assessment Mastery score* (stage 1) and Sumscore* (stage 2). *When exactly the same, points were jittered.

Figure displaying the distribution of mean scores against the difference between the first- and second-time scoring and visualizing the 95% limits of agreement. The figures display that for the mastery score all points are within the 95% limits of agreement and for the sum score one point is outside the 95% limits of agreement. It also displays that the mean sum score is, despite one outlier, between 60 and 100.

Table 5. Overview of the intra-rater reliability statistics.

	Mean difference	SD	Range difference scores	Frequency difference score zero (%)	Lower LoA (95%)	Upper LoA (95%)	Observations within LoA (%)	Outliers outside LoA (95%)
Mastery score	−2.7	9.1	−20–15	68.2	−20.5	15.0	100.0	None
Error scores (summed)							93.2	–
Accuracy	1.0	10.3	−20–11	63.6	−19.2	21.2	90.9	−20;−20
Repetition	−6.1	21.6	−100–0	86.4	−48.4	36.2	95.5	−100;−20
Dismiss	0.7	3.1	0–14	95.5	−5.3	6.6	95.5	14
Times	−2.0	5.3	−20–0	86.4	−12.4	8.5	90.9	−20;−13
Sum score	1.3	4.4	−6–16	27.3	−7.4	10.0	95.5	16
Quadrant scores (summed)							93.2	−
Perceive	−0.1	6.0	−15–10	45.5	−11.9	11.6	95.5	−15
Recall	−0.3	5.5	−15–7	50.0	−11.1	10.6	95.5	−15
Plan	2.8	7.7	−4–34	59.1	−12.4	18.0	95.5	34
Perform	2.1	8.7	−17–21	36.4	−15.0	19.2	86.4	−17;21;21
Sub-quadrant (summed)							92.4	–
Attending	0.5	9.3	−22–22	72.7	−17.7	18.7	86.4	−22;−22;22
Sensing	−2.5	10.1	−44–11	81.8	−22.4	17.4	95.5	−44
Discriminating	4.6	14.6	−16–50	81.8	−24.1	33.2	95.5	50
Recall facts	0.5	8.6	−22–22	81.8	−16.4	17.4	86.4	−22;22;22
Recall schemes	1.5	6.2	−11–22	81.8	−10.6	13.6	90.9	−11;22
Recall procedures	−2.5	11.7	−44–11	68.2	−25.5	20.5	95.5	−44
Mapping	0.0	0.0	0–0	100.0	Total agreement		100.0	None
Programming	2.1	12.9	−22–34	63.6	−23.1	27.4	95.5	34
Evaluating	6.6	17.4	−11–67	72.7	−27.5	40.6	90.9	44;67
Initiating	4.6	10.5	0–33	81.8	−16.0	25.1	90.9	33;33
Continuing	−1.5	13.6	−45–34	63.6	−28.1	25.1	90.9	−45;34
Controlling	3.5	9.9	−22–23	54.5	−16.0	22.9	90.9	23;23

Internal consistency

Based on 65 PRPP-Assessment scores on stage 2 (excluding matches), Cronbach’s alpha was 0.92. In 11 PRPP-Assessments, all 35 descriptors were scored, which yielded a Cronbach’s alpha of 0.89.

Inter-rater reliability

For the PRPP-Assessment Stage 1 Mastery Score, ICC_agreement was 0.646 with an ICC_consistency of 0.734. Based on the overall high performance of children (Figure 2), limited variation was observed between cases on the scores of stage 2 of the PRPP-Assessment, which is also confirmed by the SD of the variance components (Table 6). This limited variability resulted in ICCs for Stage 2 Sum score and individual quadrants below the expected 0.6 (Table 6). Consequently, the SD variance component of the raters can be used as an indicator for the inter-rater reliability: a low score indicates a low variation between scores and therefore, high inter-rater reliability [54].

Table 6. Overview of the inter-rater reliability statistics.

	ICCAgreement	ICCconsistency	Mean (SD)	Median (range)	Mean Difference* (SD)	Difference median (range)	SD Variance component Children**	SD Variance component Raters**
Stage 1 mastery score	0.646	0.734	60 (27)	64 (0–100)	4 (26)	0 (−57–60)	22	9
Stage 2 sum score	0.316	0.372	86 (11)	88 (45–100)	2 (15)	1 (−45–55)	6	4
Perceive quadrant	0.325	0.333	90 (13)	100 (52–100)	1 (16)	0 (−48–33)	7	2
Recall quadrant	0.225	0.286	91 (11)	93 (44–100)	0 (16)	0 (−44–56)	5	5
Plan quadrant	0.269	0.317	79 (18)	85 (33–100)	5 (24)	4 (−59–67)	10	7
Perform quadrant	0.220	0.253	82 (13)	83 (33–100)	3 (19)	2 (−50–67)	6	5

*: The difference was calculated for each activity with the score of the first rater as a ‘reference value’. Based on those differences per activity, the mean difference (and SD) was calculated.

**: Standard deviation was calculated by rooting the variance components which were used to calculate the ICC.

Convergent validity

Hypotheses 1 and 2: Rank-correlation between COPM and PRPP Stage 1

Overall, the mean COPM performance parent-rated was 7.1 (SD 2.3) and the mean PRPP-Assessment mastery score (clinician rated) was 60% (SD 25). For the 20 child-rated activities, the mean COPM performance child-rated was 8.1 (SD 2.1) with a mean COPM performance parent-rated of 7.9 (SD 2.3) and a mean PRPP-Assessment mastery score (clinician rated) of 67.0% (SD 21). Rank-correlation between COPM-performance score (parent-rated) and PRPP stage 1 mastery score resulted in a Kappa of 0.062. The correlation between the COPM-performance score (child rated) and the PRPP resulted in a Kappa of −0.045. As result, both of the hypotheses were rejected.

Hypothesis 3:

Child-recognition by Parents in the PRPP-Assessment Report

Eleven out of thirteen parents (85%) said they recognized their child in the PRPP-Assessment report. Therefore, the hypothesis was accepted. One parent stated partial recognition and one parent reported no recognition (did not agree with the information contained in the report). The parents who stated partial recognition didn’t elaborate on the statement. The parents who stated no recognition were surprised that no major problems were observed in the Quadrant Perceive while the child had a visual impairment. They felt that great clinician knowledge about the child and his/her disabilities would have made a difference in the outcome of the PRPP-report.

Structural validity

Hypothesis 4:

Item fit Statistics

Before examining fit, the overall analysis is presented in the Wright map in Figure 3. It shows that all children scored relatively high on the PRPP-Assessment, that items and tasks varied in difficulty, and all raters conducted the assessment within expectations. The PRPP items applied most effectively by the children were recalling facts about objects, body parts and task environment (“recognizes”), and to recall procedures about objects (“uses objects”). PRPP items focusing on evaluating task performance, which require a higher order metacognitive function, were less efficient (“questions,” “analyses,” “judges”). The least efficient was applied cognitive strategy behaviors for ongoing programming and controlling of motor responses during task performance (“coordinates,” “calibrates,” “flows,” “adjusts”).

Figure 3. Wright map for the many-facets Rasch analysis. PRPP Items: PRPP-Assessment phase 2 descriptors of each quadrant, a higher score means the least efficient or harder item (id obstacles = identifies obstacles, contextplace = contextualizes to place, contexttime = contextualizes to time, contextduratio = contextualizes to duration); Patient measures: each number represents participant, a higher score means better ability; Rater measures: each letter represents a rater, a higher score means more severe rater; Task measures: task categories (see Supplementry Appendix 1 for activities), a higher score means a more difficult task.

Figure displaying the hierarchy of items, patient measures, rater measures and task measures illustrating the difficulty (items), ability (patient in this case child) or severity (rater and task) of facets. It shows that items are situated between −2 and +2 logits, patients are situated between 0 and + 3 logits, and raters and tasks are situated between −1 and +1 logits.

Structural validity was evaluated by examining “fit” of the 35 PRPP descriptors to the Rasch mathematical model, and all 35 items showed acceptable fit. Table 7 shows that all items demonstrated infit and outfit mean square and z-scores within the expected range, indicating congruence with the underlying Rasch model. Therefore, the hypothesis was accepted.

Table 7. Item fit statistics and measures.

	PRPP-Item	Measure	Infit statistics		Outfit statistics
Quadrant	(most difficult on top)	(logits)	MeanSquare	Z-score	MeanSquare	Z-score
Perform	COORDINATES	1.67	0.93	−0.71	1.00	0.04
Plan	CALIBRATES	1.46	1.01	0.10	1.01	0.11
Plan	ANALYSES	0.98	1.01	0.15	1.01	0.13
Perform	FLOWS	0.98	1.13	1.79	1.22	1.85
Plan	QUESTIONS	0.93	0.89	−1.53	0.86	−1.24
Plan	ADJUSTS	0.83	1.05	0.68	1.12	0.97
Plan	JUDGES	0.63	1.01	0.10	1.12	0.85
Plan	CHOOSES	0.39	1.09	0.97	1.15	0.86
Perceive	MODULATES	0.37	1.00	0.00	1.29	1.51
Plan	SEQUENCES	0.28	0.85	−1.60	0.71	−1.53
Recall	RECALL STEPS	0.20	1.01	0.09	0.85	−0.67
Plan	ID OBSTACLES	0.16	0.99	−0.05	1.34	1.43
Perform	TIMES	0.16	1.13	1.14	1.13	0.61
Perceive	MATCHES	0.12	0.96	−0.04	0.77	−0.27
Plan	ORGANISES	0.11	0.93	−0.57	1.33	1.34
Recall	CTX PLACE	0.08	1.01	0.10	0.88	−0.44
Recall	USES BODY	−0.01	0.95	−0.38	0.80	−0.75
Perform	CONTINUES	−0.01	0.90	−0.73	0.93	−0.20
Recall	LABELS	−0.04	1.03	0.28	0.86	−0.45
Perceive	MONITORS	−0.05	0.96	−0.24	0.82	−0.60
Perform	PERSISTS	−0.17	1.04	0.32	1.06	0.30
Perceive	SEARCHES	−0.19	1.06	0.41	0.83	−0.49
Recall	CTX DURATION	−0.19	1.09	0.64	0.94	−0.09
Perceive	LOCATES	−0.22	0.98	−0.08	0.74	−0.83
Recall	CTX TIME	−0.25	0.98	−0.08	0.67	−1.04
Perceive	MAINTAINS	−0.35	0.85	−0.79	0.59	−1.28
Perceive	NOTICES	−0.39	1.11	0.63	1.26	0.78
Perform	STARTS	−0.65	1.11	0.52	0.98	0.10
Perform	STOPS	−0.68	1.05	0.29	1.15	0.46
Perceive	DISCRIMINATES	−0.74	0.93	−0.18	0.74	−0.46
Plan	KNOWS GOAL	−0.78	0.99	0.07	0.83	−0.20
Recall	CATEGORISES	−0.85	1.06	0.29	0.81	−0.23
Perceive	REGULATES	−0.93	1.07	0.32	1.67	1.22
Recall	USES OBJECTS	−1.08	1.27	0.79	1.28	0.64
Perceive	RECOGNISES	−1.81	1.00	0.23	0.55	−0.27

Item Reliability: 0.95; Separation Index: 4.56. ID = identifies, CTX = contextualizes.

Hypothesis 5:

Expert statements on Hierarchy PRPP Descriptors

Experts stated that due to the anticipated level of motor problems children will be challenged in the Perform Quadrant. They expected that the best functioning Quadrant would be Recall. More than half of the experts expected the descriptors “coordinates,” “calibrates,” “flows” and “uses body” to be more difficult in comparison to the known hierarchy of children [20].

Summarizing all item measures; the Recall Quadrant was indeed the easiest for children with a mitochondrial disorder. The experts mentioned that the Perform Quadrant would be hard for children with a mitochondrial disorder. Comparing the hierarchy output of this study to those of Nott et al. [20], the Perform Quadrant is indeed harder for the current studied population. The two statements on the Quadrants were correct, and three out of the four items were harder for children with a mitochondrial disorder. Only “uses body” did not seem as hard for children with the mitochondrial disorder as expected by the experts. In summary, five out of six statements were correct, so the hypothesis was accepted.

Known group validity

Hypothesis 6:

Distinction in Strategy Application between the Profiles

The graph (interaction plot) of Rasch calibrated mean scores generated for each of the 35 PRPP items for the three functioning profiles is shown in Figure 4. Roughly, two patterns emerged. First, the global low functioning profile shows a less efficient cognitive strategy application than the other two functioning profiles. The other two profiles (mixed and global moderate) demonstrated a similar performance pattern across most PRPP items. Secondly, the difference in efficient cognitive strategy application between the low functioning profile and the other two profiles differs the most on the Perceive and Recall Quadrant. As the difference between groups is visually noticeable, the hypothesis is accepted.

Figure 4. Interaction plot showing the difference between scores in logits obtained on all PRPP-Items for each of the functioning profiles. Context: contextualizes; ID: identifies.

A line graph plotting the absolute measure of each descriptor of the PRPP-Assessment, showing three lines fitting with each of the global functioning profiles. Visually the line of the absolute measure of the group global low functioning profile is lower throughout the four quadrants than the other two profiles. The lines do get closer to each other in the plan and perform quadrant.

Discussion

The aim of this study was to evaluate the reliability and validity of the PRPP-Assessment by parent-provided videos (n = 65) in 13 children with a mitochondrial disorder, as rated by 11 therapists. Intra-rater reliability was sufficient to good. The internal consistency was high, which shows that the items of stage 2 consistently reflect the information processing model in children with a mitochondrial disorder. Also, the ICC_Agreement on the Mastery score (stage 1) showed acceptable inter-rater reliability. As anticipated, the ICCs to determine inter-rater reliability were low due to the lack of variation between cases on stage 2. However, the standard deviation of the variance component of raters was lower than the variance of the children which indicates acceptable inter-rater reliability. Although convergent validity based on comparing COPM and PRPP could not be established, good structural and known group validity was demonstrated, resulting in four out of six accepted hypotheses on construct validity. In addition, Rasch's analysis demonstrated sound goodness-of-fit. Overall, the results support the reliability and validity of the PRPP-Assessment in the heterogenic group of children with a mitochondrial disorder.

Comparing reliability results to existing literature shows similarities and differences. For example, the high internal consistency found in the current study was also observed in other target groups: Nott et al. [21] found an internal consistency of 0.88 in adults with brain injury; Van Keulen-Rouweler et al. [28] found an internal consistency of 0.71 in persons with Parkinson’s disease. For rater reliability, the findings seem in line with previous studies in other target groups, but the comparison is hampered due to different methods in statistics and in using free versus fixed procedural task analysis. For instance, Van Keulen-Rouweler et al. [28] used ICC’s, and found a moderate to good intra-rater reliability (0.6–0.83), although with a wide range. This wide range is also observed in the current study, even though very few data points outside the 95% Limits of the Agreement were found; overall 32 out of 484. In addition, Nott et al. [21] calculated intra-rater reliability for stage 2 Sumscores in adults with brain injury by using Limits of Agreement, as also used in the current study. They found a slightly larger mean percentage difference of −4.52. Although all data points of the study of Nott et al. [21] were within the 95% Limits of Agreement, the differences of the current study are more concentrated around the zero. Focusing on inter-rater reliability, Van Keulen-Rouweler et al. [28] found a relatively low ICC for stage 1 mastery score in persons with Parkinson’s disease (0.36) without the use of pre-set task analyses. Consequently, they recommended using a pre-set procedural task analysis in line with clinical practice and also adopted in this study. Using a pre-set task analysis in the current study led to a more acceptable ICC for stage 1 mastery score (0.65) in line with the study of Steultjens et al. [31], who reported an ICC of 0.63 in persons with dementia. Besides, the ICC_agreement and the ICC_consistency for the quadrants are relatively close together in the current study indicating that there are no systematic errors as the variance due to the raters has little effect on the ICCs, which suggests that the PRPP-Assessment was scored consistently. Therefore, this first study highlighting all types of reliability in the PRPP-Assessment of children can be seen as satisfactory because excellent reliability results are difficult to achieve in highly ecological valid assessments [21].

Content validity represents the degree to which the instrument items are an adequate reflection of the construct to be measured [55]. The PRPP-Assessment focuses on cognitive strategies, which are defined as internally generated thinking processes required to think through task performance [16]. The PRPP-Assessment results in a performance mastery score based on the observed application of cognitive strategies. Effective cognitive strategy use is defined as the simplest and most efficient means of processing information relative to a specific situation [56]. The cognitive strategies (PRPP-Assessment items) showed a very good fit with the Rasch model, and the ordering of items was consistent with the information processing model that underpins the PRPP-Assessment [15,17]. In addition, the consistency with the expert opinion showed that the ordering of items fits with clinical experience. When compared to previous Rasch analysis with children [20], the findings of this study show a different hierarchy, which can easily be explained by the clinical manifestation of a mitochondrial disorder [4,6,57]: the Perform Quadrant, in particular ‘coordinates’, ‘calibrates’ and ‘flows’, was more difficult for children with mitochondrial disorder, whilst the Recall Quadrant was easier. The item hierarchy generated in this study is however similar to other target groups that demonstrate a combination of physical and cognitive disabilities with different severities, for example, Parkinson’s disease [29]. Comparing the PRPP-Assessment to other Rasch calibrated occupation-based assessments, the Child Occupational Self Assessment [58] also demonstrates a good fit with the Rasch model [10]; however, this assessment showed a large ceiling effect, limiting usefulness in clinical practice with heterogenic groups. The PRPP-Assessment showed a good distribution of the items compared to the children, suggesting there is little to no floor- or ceiling effect. This suggests that the PRPP-Assessment measures cognitive functions in a similar pattern of information processing, across varied target groups and that it is valid to use with different levels of functioning.

For the known group validity, the three profile groups defined based on the functional capacities of the children were used [33]. The current analysis suggests that the PRPP-Assessment can map the information processing abilities of children at different functional levels. In addition, it showed that the cognitive strategy pattern of children in the global low functioning profile was different from the cognitive strategy pattern in the other two profiles; low cognitive functioning with a moderate to normal ambulation (‘mixed’) and global moderate functioning. Due to the relatively small sample size, the difference is not robust enough to be statistically significant. However, it is comparable to the study of Sturkenboom et al [29] in people with Parkinson’s disease who found a statistically significant mean difference between two sub-groups. Overall, the findings in the current study show the clinical relevance of the PRPP-Assessment for heterogenic groups as it can measure high functioning and low functioning.

Convergent validity was measured by calculating the rank-correlation between the COPM-performance score and the PRPP-Assessment mastery score and resulted in a very low level of agreement. This finding suggests that the COPM-performance (parent-rated) and PRPP-Assessment mastery score (therapist rated) may in fact be measuring different constructs. Prior to viewing the video footage provided by parents, the researcher discussed the desired level of performance that would be deemed satisfactory by the child or their parents. Therapists used this criterion-referenced approach as outlined by the PRPP scoring instructions [36]. In contrast, parents appeared to use a normative frame of reference when scoring the COPM. Parents used wording as; “he has trouble with participating, so I score every activity a 4,” or “she is not performing this activity as her peers, therefore I can’t score it a good mark.” It appears that the low level of agreement between ratings on the COPM and the PRPP-Assessment may be attributed to parents using a ‘normative’ frame of reference, while therapists used a ‘criterion’ frame of reference that characterizes the PRPP-Assessment. Although the researcher asked ‘how would you score performance taking into account the criterion we just formulated’, the common way in society is to compare children with other children which may have made it hard for parents to score performance against the criterion. Therefore, it is suggested to use more objective measures to establish convergent validity as recently performed by Nott and Chapparo [27].

A strength of using the PRPP-Assessment that was evident in the current study was the self-selection of tasks by parents and/or children and tasks were meaningful. Self-Determination Theory, proven to be important in the meaningfulness of tasks for children with the mitochondrial disorder [8], highlights the internal factors of autonomy and competence that can influence motivation for these tasks. In contrast to other activity-based assessments, tasks selected by parents represent five domains of the ICF-CY and all performance areas of the OPMA. In clinical practice with children with mitochondrial disorders, the Pediatric Evaluation of Disability Inventory (PEDI) [59,60] is often used to gain insight into daily functioning. The PEDI assesses self-care, ambulation and social functioning but does not focus on domestic life, play or other major life areas for children, such as education, and is a norm-referenced test. In the current study the wide-ranging complexity of tasks is presented in the Wright map (Figure 3), and the PRPP-Assessment was shown to be valid when used to assess children performing this wide range of meaningful tasks and activities.

Despite the inclusion of children from different functioning profiles, the research faced a lack of variability within the PRPP-Assessment scores. We anticipated that this could underestimate the reliability values since reliability is the ratio of variability between the subjects to the total variability [40,54]. This was confirmed by another study with low variability and low ICCs [28] whereas studies with the larger variation found moderate to acceptable levels of inter-rater reliability [21,61]. Overall, the children scored relatively high on the PRPP-Assessment, which could be remarkable as children with different levels of functioning were included, from fully dependent to independent functioning in daily life. However, the heterogeneity of the children did not lead to heterogeneity in task performance due to the criterion-referenced basis of the PRPP-Assessment. For example, tasks such as ‘Caring for body parts’ was observed to be one of the most difficult tasks and were performed primarily by children who required high levels of assistance. Whereas more complex tasks such as school or vocational training were performed by children who required minimal assistance and were primarily independent in daily functioning. Criterion-based assessments such as the PRPP-Assessment enable occupational therapists to observe and evaluate cognitive functioning on a wide range of tasks, irrespective of the perceived level of difficulty for typically developing children. Tasks can be calibrated as ‘easy’ or ‘hard’ relative to each individual child, not to the entire sample of children being assessed or in respect to all children with similar diagnoses. Thus, it can be concluded that in this study, tasks were selected and/or criterions were formulated in a way that the outcome of the PRPP-Assessment was quite high. In practice, this could be problematic when the objective of the PRPP-Assessment is to implement and/or measure the effectiveness of child-family-centered interventions. In that case, tasks and/or criterion should be carefully selected to give enough room for improvement. Overall, it is recommended to be careful in task selection and formulating the criterion. In practice, a method could be to question parents on the desired improvement on the activity and specifically within that activity. This should give room for improvement when used for planning interventions or measuring change. Either by choosing a more challenging task or by formulating a stricter criterion while keeping the focus on meaningful occupational performance.

Although a limitation of this study was the relatively small sample of 13 children with a total of 65 activities, it is the first study examining the reliability and validity of the PRPP-Assessment in the complex target group of children with a mitochondrial disorder. Reliability and validity statistics are satisfying, and most parents stated that they recognized their child in the PRPP-report. In previous studies [19], parents also expressed that the PRPP-Assessment was valuable for the care of their child. Unfortunately, in this study, the COPM data couldn’t be collected from all children themselves. Due to the communication difficulties experienced by the children, in-person data collection was necessary. The COVID-19 restrictions, which were introduced during data collection, inhibited in-person contact, which led to very few child perspectives in this research. Therefore, a recommendation for future research is to evaluate the clinical relevance of the PRPP-Assessment from the child’s perspective. In addition, to make the PRPP-Assessment applicable to monitor the course of the disease, and the impact on everyday functioning, more research focusing on the ability to detect the clinically relevant change and overall responsiveness is necessary. In addition, when applying the PRPP assessment to these complex target groups, where setting a personalized criterion is needed, it is important to select tasks that fit with the abilities of the child and are challenging at the same time. The current study adds to the knowledge of psychometric properties and provides encouraging evidence for the reliability and validity of the PRPP-Assessment to assess each individual child within the heterogenic group of children with a mitochondrial disorder with a personally-relevant occupational focus.

In memoriam

Dr. Esther Steultjens contributed extensively during this research, but was no longer with us finishing this article. Her input during the research was extremely valuable.

Acknowledgements

We would like to thank all children, parents and OTs that participated in this study.

Disclosure statement

The authors report no conflict of interest.

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Additional information

Funding

This work was supported by the Dutch Research Council (Netherlands Organization for Scientific Research (NWO)) [project number 023.009.016].