Test-retest reliability of Picture My Participation in children with intellectual disability in South Africa

Abstract

Background

Picture My Participation (PmP) is a promising instrument for measuring the participation in everyday situations of children with intellectual disability (ID), particularly in low- and middle-income countries.

Aim

To explore test-retest reliability of PmP by comparing two repeated measurements of children with ID in an urban context in South Africa.

Methods

A picture-supported interview with 31 children with ID, aged 7–17 years, was conducted twice, two weeks apart. The children rated their participation, operationalised as attendance and involvement, in 20 everyday activities. Analyses were completed for total scores, for the four subcomponents and at item level.

Results

Test-retest agreement at an item level for both attendance and involvement showed slight/fair agreement for most activities (Kappa = 0.01–0.40), and moderate agreement for some activities (Kappa = 0.41–0.60). Moderate agreement was shown for the total scale and at component level (ICC = 0.5–0.75), except for (firstly) attendance of and involvement in ‘Family Activities’ (ICC = 0.26 for attendance, 0.33 for involvement), and (secondly) involvement in ‘Personal Activities’ (ICC = 0.33).

Conclusion

The result indicates that PmP can reliably be used at component level and as a screening tool for intervention planning to identify participation and participation restrictions in children with ID.

Keywords:

• Low- and middle-income countries
• everyday functioning
• picture-supported interview
• cognitive support
• self-rating
• participation

Introduction

The concept and practice of children’s participation in society gained increasing attention since the adoption of the United Nations Convention on the Rights of the Child [1]. However, the concept as defined by the Convention is broad and has come to refer to a range of practices [2]. The International Classification of Functioning, Disability and Health (ICF), developed by the World Health Organisation [3], also introduced the concept of participation as an important indicator of health outcomes. Participation is defined in the ICF as ‘involvement in life situations’ and participation restriction is defined as ‘problems an individual may experience in involvement in life situations’ [3,4]. In their systematic review of the literature, Imms et al. [5] identified two themes, namely attendance and involvement, as being consistent with the concept of participation. According to them, they ‘relate to the objective “being there” and the more subjective “in-the-moment” experience of participation’ [5, p. 33].

Whilst participation has been identified as an important outcome for children, there is a lack of appropriate measurements of this dimension for young children [6,7]. Article 12 of the CRC [1] articulates the right of children to form their own views and express them freely in accordance with their maturity and age. A 90% exclusion rate of children with developmental disabilities and other disabilities was however found in mainstream research [8]. Where information on children with ID was included in research concerning low- and middle-income countries (LMICs), researchers mainly relied on proxy ratings from the adults in the children’s lives [9–11].

Despite South Africa’s long-standing commitment to the protection of children’s rights by endorsing the Convention on the Rights of the Child (CRC) [1] and the African Charter on the Rights and Welfare of the Child [12], the majority of South Africa’s historically marginalised children remain socially excluded and are prevented from developing their full potential [13]. Children with ID have been excluded from voicing their own opinions since they are often dependent on others to express themselves [14]. One explanation is that there is a general lack of assessment in respect of participatory processes for children with ID in South Africa [2]. It has also been suggested that the cognitive difficulties the children experience may make their self-rating less reliable [15]. However, a child’s rights perspective calls for children to be asked about their own experiences and perceptions of participation even though they have an intellectual impairment [16]. Compared to children without disabilities, there is a pattern that children with ID participate less frequently in e.g. recreational, active-physical and skill-based activities such as organised sporting activities with friends or others [17,18]. Instead, children with ID tend to participate more in activities at home. These differences in participation are suggested to be related to physical, cognitive and social skills, but may also be related to differences in the supportive context [17,18].

Participation is a relatively vague construct, although many researchers agree that it describes a person’s involvement in a life situation [3,4]. They also agree that attendance and involvement are two essential aspects of participation and that both should be captured in a measurement of participation [5,19]. Measuring a multi-dimensional construct with a self-rating tool, and especially with persons who have cognitive impairments, generates several challenges concerning consistency in responses that are usually managed by using proxy ratings rather than self-ratings by children and adolescents [6]. However, there is a need for self-rating instruments also for these individuals in order to obtain the child’s perspective on their participation [6,20]. Letting the individuals voice be heard is especially important in clinical settings were responses may be related to the selection of intervention goals. Care provider and child perceptions on participation goals do not always overlap [21] even if ratings from care providers and children tend to be highly correlated [17].

A systematic review of the instruments intended to measure participation by children and adolescents with disabilities in low and middle income countries (LMICs) acknowledge the methodological challenges but conclude that participation may also be measured by self-ratings in children with ID [11]. However, the same systematic reviews show that none of these instruments were originally developed or culturally validated for the everyday contexts of LMICs [7,11]. Only two of the 21 measures evaluated by Rainey et al. [7] were developed in LMICs (China and Taiwan) and both focussed on participation in a clinical setting (hospital and physical therapy) – not in everyday contexts. Rainey et al. [7] furthermore argued that there is a shortage of good quality information regarding the psychometric properties of all 21 instruments.

Picture My Participation

This article reports on the test-retest reliability of a self-rating instrument called Picture My Participation (PmP) that is designed to capture the two participation dimensions of attendance and involvement in children and youth with mild intellectual disability who live in low-resource settings [22]. The instrument has previously been tested for content validity [22], inter rater agreement between care providers and children [17] and structural validity [23]. PmP is manual based structured interview instrument. It comprises of three trial items where the child’s ability to understand the concepts frequency of attending and engagement with help of graphic symbols and the scale anchors illustrated by graphic symbols are tested (Figures 1–3). After the trial items, four sections are following with the purpose to: (1) determine perceived attendance in various activities, using a four-point Likert-type scale (Figure 1), (2) determine perceived involvement in various activities using a four-point Likert-type scale (Figure 2), (3) prioritise activities considered to be the three most important to the child; and (4) determine perceived barriers and facilitators to participation [22]. The first two purposes are the focus of this study whereas the two last purposes are best validated in a clinical intervention context with individual children. The instrument is not intended to be used to obtain a norm based total score since participation is supposed to be individually contextualised based on individual children’s everyday environment in four areas of activities, i.e. subcomponents: ‘Organised activities’, ‘Social interactions’, ‘Family activities’ and ‘Personal activities’ [23]. ‘Organised activities’ includes six items (‘Trips and visits’, ‘Organised leisure’, ‘Cleaning at home’, ‘Health centre’ (visits to), ‘Gathering supplies’ and ‘Shopping’) and encompasses events or pursuits that a group of people are doing together in a structured way. ‘Social activities and taking care of others’ includes seven items (‘Playing with others’, ‘Caring for family’, ‘Spiritual activities’, ‘Celebrations’, ‘Caring for animals/pets’, ‘Social activities’ and ‘Meal preparation’) and encompasses events or pursuits that bring members of the community together. ‘Family life activities’ includes three items (‘Family time’, ‘Family mealtime’ and ‘Quiet leisure’) and encompasses events or pursuits that bring members of the family together. ‘Personal care and development activities’ also includes three items (‘School’, ‘Personal care’ and ‘My own health’) and refers to both basic self-care tasks of bathing, dressing, personal hygiene and grooming, as well more complex tasks related to health and education. The item ‘Paid and unpaid employment’ are not included in any sub-component.

Figure 1. Visual four-point Likert scale for the ratings of attendance. “How often do you participate in…?”

Figure 2. Visual four-point Likert scale for the ratings of involvement. “How involved are you…?

Figure 3. Four examples (one from each subcomponent) of pictures/symbols from Bildstod™ (www.bildstod.se).

Reliability is defined as the degree to which the measurement is free from measurement error [24]. The current study intended to investigate the test-retest reliability of PmP as a tool that is used to assess the two aspects of participation – attendance and involvement – for screening purposes in children with ID in a South African context. Due to learning difficulties, children with ID may show a larger intra-individual variability of scores, compared to an ‘average population’ of children, when aspects of participation are rated [15].

PmP was developed to measure the participation of children with disabilities, especially in LMICs [22]. PmP measures participation in 20 home, social and community activities and is performed as part of a structured interview with children. The items of PmP were selected by reviewing existing participation measures and matching items to the UNCRC [1,25–27]. The content of the 20 items were found to be valid in the LMIC context (South Africa) and for children with ID [22]. PmP used accommodations such as Talking Mats™, which through the use of visual media provide an innovative way of presenting meanings and concepts in an interview [28].

For screening purposes and in a public health or children’s rights context, it may be sufficient to only explore the attendance aspect of participation, measured by ratings of actual performance of an activity [4,29,30]. However, when applied to individuals in a practical setting, the attendance aspect, in combination with the involvement aspect, may be more relevant to explore. These aspects can be measured by studying perceived involvement/engagement in an activity [5,29]. Anonymous et al. [22] conclude that using the PmP methodology to obtain knowledge separately about each of the two aspects of participation (attendance and involvement) was comprehensive for children with ID. The comprehensiveness or content validity of a combined participation measure (combining attendance and involvement) was however not explored. The instrument was also not investigated regarding reliability (i.e. to what extent the ratings of attendance and involvement scores are the same over repeated measurements), or regarding the interrelatedness among items (levels of internal consistency). When evaluating the test-retest agreement for the repeated measures, a possible intra-individual variability has to be considered.

Aim of the study

To explore the test-retest reliability of PmP by comparing two repeated self-ratings with PMP by children with ID in an urban context in South Africa.

Research questions

What is the degree of agreement between two repeated ratings of attendance and involvement respectively when using the PmP for children with mild intellectual disability for the following:

1. Each of the twenty activity items

2. The four subcomponents ‘Organised activities’, ‘Social interactions’, ‘Family activities’ and ‘Personal activities’

3. The total scales

Materials and methods

Design

A quantitative, non-experimental, comparative survey research design was used where 31 children were interviewed. To achieve test-retest reliability of PmP for children with ID, each child was required to complete the interview on two different occasions, scheduled two weeks apart [31,32]. The interviews were conducted by the same interviewers on both occasions.

Ethics

Ethical approval was received from the Faculty of Humanities at University of Pretoria (GW20180301HS) as well as from relevant departments of education and school principals.

Setting

Three schools in one province of South Africa were approached to participate in the study. The schools were government-funded public schools located in an urban area that accepted children diagnosed with mild to moderate intellectual disability [33]. All the schools maintained an average of 640 enrolled learners at the time of the study and used the adapted CAPS (Curriculum Assessment Policy Statement) and the TOC (Technical Occupational Curriculum), with English being the language of teaching and learning.

Participants

The selection criteria for inclusion in the study were as follows:

1. Children had to attend a school that enrols children with ID.

2. Children should score between 40 and 84 on the Kaufman Brief Intelligence Test - Second Edition (KBIT-2) [34].

3. Participants had to be between the ages of 7 and 17.

A total of 80 caregiver consent letters were sent out via the teachers at the respective schools, and 55 consent letters were returned. Of these, nine caregivers did not provide consent for their child to participate in the study. Two children were excluded as they did not meet the selection criteria. A further twelve children with ID were excluded as they did not complete the second interview, either due to absenteeism or not providing assent. A final tally of 31 children with ID participated in this study.

The eventual sample consisted of 31 participants who met the selection criteria for the study and provided assent to take part in the study. The chronological age of the children ranged from 7 to 17 years old (mean = 11.4; SD = 2.43; median = 12) with 77.4% (n = 24) being boys and 22.6% (n = 7) being girls.

Variables and measurements

Kaufman brief intelligence Test - Second edition (KBIT-2)

The KBIT2 [34] is a standardised screening tool used for the identification of a child’s intelligence quotient (IQ) [33,34]. The KBIT-2 was designed to measure verbal and nonverbal intelligence in individuals aged 4–90 years [34,35]. It comprises three subtests: (1) verbal knowledge, (2) matrices and (3) riddles. Two of the subtests are verbal (requiring one-word responses) and one is non-verbal [34].

Picture My participation (PmP)

PmP is a 20-item instrument that was developed to identify the attendance and involvement in home, school and community activities for children with ID in LMICs [23]. It was designed to be completed through the exclusive use of graphic symbols. A graphic symbol that is the closest (most accurate) representation of the core concept of each item was identified online on the website of Bildstod™ (www.bildstod.se). This website contains a variety of symbols from different systems that can be used to create picture-based material for information and communication. The main reasons for using these graphic symbols, rather than for example commercially available symbols, were that they were both freely available and there was a large library of picture support symbols (www.bildstod.se) that could be selected based on cultural preferences [22]. Figure 3 shows four examples (one from each subcomponent) of pictures/symbols from Bildstod™ (www.bildstod.se).

PmP comprises of three trial items that we used to help establish rapport with the child and ensure that understood the requirements of the task. The trial items are followed by four sections that intends to: (1) determine perceived attendance in various activities, using a four-point Likert-type scale, and involvement in the same activities using a three-point Likert-type scale, (2) determine perceived involvement in various activities using a four-point Likert-type scale, (3) prioritise activities considered to be most important to the child; and (4) determine perceived barriers and facilitators to participation [22]. The first two intentions mentioned (perceived attendance and involvement) are the focus of this study.

A specific structured interview approach, Talking Mats™ [36], was used to conduct the interviews with the children with ID. The Talking Mats™ framework is a strategy that facilitates conversations with persons with disabilities, as well as an instrument for facilitating communication with, for instance, children with ID. It comprises of a low-priced textured mat and picture symbols with Velcro attached at the back so that they can be placed on the mat and manipulated [37]. A study conducted by Steward et al. [38] showed that Talking Mats™ was effective in overcoming some of the difficulties that people with intellectual disabilities may experience in verbally based interviews.

Data collection

Kaufman brief intelligence Test - Second edition (KBIT-2)

On the pre-arranged date and time, the children with intellectual disability were taken to a quiet room for the data collection procedures to be completed. The assent procedure was administered for prospective child participants, and for those who provided assent, the KBIT-2 [34] was administered, prior to administration of the PmP instrument. Children scoring an IQ composite between 40 and 84, which indicates mild to moderate intellectual disability, were included in the study. Children who did not meet the criteria were thanked, provided with a token of appreciation and returned to their classroom.

Picture My Participation (PmP)

The participants who complied with the criteria were asked three trial questions to ensure that they understood the instructions. The researcher sat next to the child at a table. On the table was a piece of carpet (63 cm × 44 cm) that was divided into four horizontal columns, with a graphic symbol depicting each point on the Likert scale. For the first section of PmP, the children were asked (with reference to the attendance construct), How often do you participate in daily routines? At the same time, they were shown the Bildstöd symbol of the specific routine, as illustrated on the PmP instrument. The children were required to respond by placing an activity item symbol on the relevant column of the carpet or by pointing to the appropriate choice, thus indicating their selection. For the ratings of attendance, a visual scale that represented the four-point Likert scale was used (4 = Always – basket filled with apples; 3 = Sometimes – basket with three apples; 2 = Seldom – basket with one apple; and 1 = Never – empty basket with no apples) (Figure 1). For the ratings of involvement, a visual scale that represented the three-point Likert scale was used on the piece of carpet (2 = Very involved – basket filled with apples; 1 = Somewhat involved – basket with three apples; and 0 = Not involved – empty basket with no apples) (Figure 2).

Two weeks after the initial administration of the PmP, the process was repeated. Each child was provided a small token of appreciation upon completion of each part of the data collection process. A presentation was subsequently arranged to provide feedback to the Department of Education, the school principal and the teachers involved. A one-page easy-to-read version of the study and its findings was also provided to parents who requested it.

Data analysis and procedural reliability

To ensure that all the interviews were conducted in a similar manner, inter-rater reliability, an independent rater listened to 40% of the recordings to ensure that the questions were asked in the same manner and order for each child, to enhance procedural reliability.

Reliability was explored by analysing the test-retest agreement between the first and second occasions, as well as the children’s ratings of both the attendance and involvement aspects of PmP. The test-retest agreement was analysed item by item for the total scales as well as for the four subcomponents ‘Organised activities’, ‘Social interactions’, ‘Family activities’ and ‘Personal activities’ while linear weighted kappa was used to analyse the test-retest agreement at item level. According to the guidelines suggested by Landis and Koch [39], Kappa coefficients of 0.01 indicate ‘poor’ agreement, 0.01–0.20 indicate ‘slight’ agreement, 0.21–0.40 indicate ‘fair’ agreement, 0.41–0.60 indicate ‘moderate’ agreement, 0.61–0.80 indicate ‘substantial’ agreement, and 0.81–1.00 indicate ‘almost prefect’ agreement. Hence, the terminology ‘poor’, ‘slight’, ‘fair’, ‘moderate’ and ‘substantial’ agreement is used.

For analysis of the test-retest agreement for the total scales and the four subcomponents of the PmP [23], an intra-class correlation coefficient (ICC), two-way random, single measures, absolute agreement (ICC 2.1 A) was used. ICC 2.1 A was used because measurement errors are likely to occur, both in respect of the raters and the children [40]. PmP was designed for use in a routine clinical context by a clinician and is intended to only require a brief introduction to participation and to the methodology, with no specific education or licence needed.

According to Koo and Li [40], ICC values of lower than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability.

Results

Test-retest reliability

The results obtained from the item-by-item analysis of the test-retest agreement for both attendance and involvement, using linear weighted kappa are presented in Table 1.

Table 1. Test-retest agreement at item level (linear weighted Kappa).

	Attendance						Involvement
	Kappa	Asymp SE	Z	p Value	Lower 95% Asymp CI Bound	Upper 95% Asymp CI Bound	Kappa	Asymp SE	Z	p Value	Lower 95% Asym CI Bound	Upper 95% Asym CI Bound
Personal care	0.196	0.164	1.405	0.160	−0.125	0.517	0.105	0.168	0.701	0.483	−0.223	0.433
Family mealtime	0.330	0.187	1.862	0.063	−0.037	0.697	0.178	0.172	1.172	0.241	−0.158	0.515
My own health	0.476	0.160	3.292	0.001	0.163	0.790	0.169	0.149	1.102	0.270	−0.124	0.462
Gathering supplies	0.394	0.136	3.023	0.003	0.127	0.661	0.153	0.118	1.175	0.240	−0.080	0.385
Meal preparation	0.534	0.113	4.454	0.001	0.312	0.755	0.550	0.133	4.134	0.001	0.289	0.810
Cleaning at home	0.191	0.128	1.485	0.138	−0.059	0.442	0.493	0.109	3.563	0.001	0.280	0.707
Caring for family	0.433	0.241	2.899	0.004	−0.039	0.905	0.306	0.203	1.922	0.055	−0.093	0.705
Caring for animals/pets	0.652	0.105	4.615	0.001	0.445	0.858	0.487	0.129	3.206	0.001	0.234	0.741
Family time	0.095	0.163	0.667	0.505	−0.225	0.415	0.162	0.135	1.095	0.274	−0.103	0.427
Celebrations	0.281	0.136	2.213	0.027	0.014	0.548	0.231	0.135	1.709	0.087	−0.032	0.495
Playing with others	0.169	0.186	1.246	0.213	−0.195	0.532	0.510	0.131	3.570	0.001	0.252	0.768
Orginised leisure	0.293	0.108	2.286	0.022	0.082	0.504	0.340	0.174	2.325	0.020	−0.001	0.681
Quiet leisure	0.335	0.120	2.258	0.010	0.099	0.571	0.286	0.154	1.990	0.047	−0.015	0.587
Spiritual activities	0.285	0.170	2.191	0.028	−0.049	0.619	0.375	0.145	2.671	0.008	0.091	0.658
Shopping	0.367	0.127	2.965	0.003	0.119	0.616	0.409	0.162	2.721	0.007	0.092	0.726
Social activities	0.025	0.104	0.233	0.823	−0.179	0.229	0.223	0.138	1.786	0.074	−0.049	0.494
Health centre	0.268	0.131	2.045	0.041	0.012	0.525	0.396	0.161	2.556	0.011	0.081	0.712
School	0.303	0.248	2.641	0.008	−0.183	0.790	0.229	0.191	1.618	0.106	−0.145	0.603
Overnights visits and trips	0.143	0.126	1.310	0.190	−0.104	0.390	0.315	0.155	2.230	0.026	0.012	0.619
Paid/unpaid employment	0.300	0.122	2.348	0.019	0.062	0.538	0.291	0.152	2.181	0.029	−0.008	0.589

The results obtained from the analysis of the test-retest agreement for the total scales as well as for the four subcomponents, using the Intra-class correlation coefficient (2-way random-effects model, average scores), are presented in Table 2.

Table 2. Intraclass Correlation Coefficient (ICC)^b (absolute agreement) for the total scale and per component.

	Attendance							Involvement
Picture My Participation	ICC^a	Lower Bound	Upper Bound	Value	df1	df2	Sig	ICC^a	Lower Bound	Upper Bound	Value	df1	df2	Sig
Total scale	0.64	0.38	0.81	4.52	30	30	0.001	0.63	0.27	0.82	5.68	30	30	0.001
Comp 1. Organised activities	0.57	0.28	0.77	3.67	30	30	0.001	0.60	0.31	0.78	3.95	30	30	0.001
Comp 2. Social interactions	0.54	0.23	0.75	3.32	30	30	0.001	0.53	0.22	0.74	3.27	30	30	0.001
Comp 3. Family activities	0.26	−0.10	0.56	1.68	30	30	0.080	0.33	−0.02	0.61	1.99	30	30	0.032
Comp 4. Personal activities	0.60	0.31	0.78	3.95	30	30	0.001	0.40	0.06	0.66	2.32	30	30	0.012

Single measures. Two-way random effects model where both people effects and measures effects are random.

^aThe estimator is the same, whether the interaction effect is present or not.

^bType C intra class correlation coefficients using a consistency definition-the between-measure variance is excluded from the denominator variance.

Table 1 shows that, according to the recommendations and terminology of Landis and Koch [39], the test-retest agreement at item level for both attendance and involvement in most activities is considered to be slight (0.01–0.20) to fair (0.21–0.40) with lower levels (slight) especially for attendance in family time and social activities. Some outlier items displayed a moderate agreement (0.41–0.60). One item (Caring for animals/pets - attendance) showed substantial agreement (0.61–0.80).

Table 2 which presents the test-retest agreement for the total scale and at a component level, shows moderate agreement (ICC scores 0.5–0.75), as suggested by Koo and Li [40]. The exceptions are attendance and involvement ratings of the component ‘Family Activities’ (with an ICC of 0.26 for attendance and 0.33 for involvement), as well as the involvement rating of the component ‘Personal Activities’ (with an ICC of 0.33).

Discussion

This study looked at the test-retest reliability of the newly developed PmP when using the instrument to assess the two aspects of participation (attendance and involvement) in children with ID for screening purposes in the South African context. Basing assessment results on children’s perceived participation is important because it reflects the child’s perspective of whether the current level of participation warrants intervention or not [41]. According to Portney and Watkins [42], reliability is fundamental to all aspects of clinical research, because without it we cannot have confidence in the data we collect, nor can we draw rational conclusions from that data. The overall results obtained from this study show that the test-retest agreement of the PmP is moderate at scale and sub-component levels and this suitable for screening purposes [23]. When used to facilitate goal selection and goal settying in intervention planning PmP has to be supplemented by more detailed information regarding subcomponets and activities seleted for intervention.

To obtain reliable data accommodations had to be made to adapt the structured interview procedure to the participants’ context. Picture support symbols (www.bildstod.se) and the Talking Mats™ were used to determine the participants’ view of participation and especially to see whether their perception of involvement in activities changed between test occasion 1 and test occasion 2. As stated in the manual, before administering actual instrument items, three test items were administered to ensure that the child understood concepts, items and scales. Moderate reliability values for the subcomponents ‘Organised Activities’ and ‘Social Interaction’ indicated stability from occasion 1 to occasion 2 for both attendance and involvement at a component level. Reliability values for the subcomponent ‘Personal Activities’ indicated that participants’ perception of their attendance is more stable than their perception of involvement. It may be related to that all items in the subcomponent, i.e. personal care, taking care of your own health as well as school in children are activities that you must do but not necessarily perceive that you are highly involved in. Probably involvement is more sensitive to respondents’ perceptions than attendance and this more sensitive to momentary perceptions. Also for the subcomponent ‘Family Activities’, relatively poor reliability values were found at the component level. It may indicate that children’s perception of their participation in these activities varied within a short period of time, especially regarding involvement. Variations in attending may not necessarily relate to perceptions but may also be dependent on actual variations in opportunities, e.g. to family time within a two week period.

It may also be dependent on the quality of the aspect rated. According to the definition of participation provided by Imms et al. [5], the attendance aspect can be measured as the frequency of attending a certain activity/situation, while the involvement aspect can be measured as the level of perceived involvement/engagement when actually attending the activity. Attendance is considered a prerequisite for involvement (to be involved requires one to be present in the situation to some degree). The attendance aspect is always more concrete/visual than the involvement aspect and this facilitates measurement of attendance in comparison to involvement. For example, you can see if someone is attending an activity, but you cannot as easily observe if the person is involved/engaged. The variations in reliability and ICC may indicate that items as well as scales may have different sensitivity to time dependent contextual variations, e.g. family time, or variations in to what degree activities are self-selected or not, e.g. school, personal care.

For screening purposes, the Cosmin checklist manual [24] states that in order to evaluate reliability, an instrument should be administered twice within an interval of two weeks with the rater not being aware of the scores obtained in the first administration. These conditions were all met in the current study design. Test conditions remained consistent from test occasion 1 to test occasion 2. The same tool was used at each interval and the same administrators used the same scripted instructions in the same setting. Using the criteria from the Cosmin checklist [24], it can be postulated that the moderate reliability values are not due to errors in the study’s design. The use of the Likert scale with children who have intellectual difficulties is a concern, as linguistic and cognitive difficulties experienced by individuals with ID may make it difficult to assess their perceptions when the content is complex or abstract [43]. To meet this threat to validity, the three test items were used to ensure that the children understood concepts, items and scales. When using the PmP, Anonymous et al. [22] found that children with ID in South Africa used the entire Likert scale to describe attendance. The pre-test using items and scales, provided with the pictures representing each level of the Likert scale may have contributed to participants’ responses. Anonymous et al. [22] noted that similar training items are not provided when determining involvement and thus recommended that, in order to obtain more valid data for involvement, children have to be trained. When using the manual in a clinical setting this is done [2]. They also had to be introduced to children in a way that accounted for their cognitive and linguistic difficulties, especially in respect of abstract concepts [22].

The test-retest agreement at the item level for both attendance and involvement was found to be slight to fair, with some items displaying a moderate agreement. The most straightforward interpretation of these results was that PmP is not a very reliable measure, at item level, of participation for children with mild intellectual disability. Even when considering the possible larger intra-individual variability of children with ID compared to an ‘average-population’ child [15], caution needs to be paramount when using PmP to evaluate interventions that focus on a single everyday activity. The test-retest agreements for the total scale and for the sub-components, except for the attendance and involvement ratings of ‘Family Activities’, are more reliable. This reliability increases when taking a possible larger intra-individual variability into account [15]. However, before using PmP as an intervention outcome measure of participation by children with ID, the possible impact of intra-individual variability due to specific learning difficulties has to be explored.

As of today, PmP can primarily be used in clinical settings as a possible guideline or screening tool to obtain knowledge about the child’s own perceptions of attendance and involvement before entry into intervention programmes aimed at enhancing child participation in everyday activities [5,6,26]. The PmP tool will be one possible way to give children with ID the opportunity to express their opinion regarding possible problems in different areas of their everyday life. A previous study in Taiwan indicate that PmP can be used as a basis for involving children in discussions with care providers and professionals about selection of goals for participation interventions [22]. An initial use of PMP can be followed by a more in-depth analysis of the environmental and person requirements for enhancing participation in the areas identified by children as important to participate more in e.g. as described by Anaby et al. [44] in using the PREP methodology.

In South Africa, children with cognitive and developmental difficulties who require intervention are often identified late, due to poor access to services at all levels of health care. Access to services is limited because of human resource challenges, more specifically the inadequate supply of key health professionals to provide early intervention services in the public and private sectors [45]. Professionals in South Africa, especially in the public sector, are not dedicated to working exclusively in the area of early intervention. This problem is due to the supply-and-demand gap of health professionals. Staff shortages have also resulted in professionals working in different clinical areas and therefore not being dedicated to early childhood intervention services.

PmP can reliably be used as a screening tool to identify children with poor participation levels based on their own and care providers’ perceptions. It can also support decisions to enter children into intervention programmes. Professionals and paraprofessionals can be trained to conduct PmP in their settings and then refer children who require intervention. The instrument could also be used by educators as part of their continuous assessments within the school system. The rather moderate reliability of subcomponents indicates that PMP can primarily be used in the planning phase of interventions, while more specific measures that are based on targeted activities are necessary to evaluate interventions on an individual level. This is especially true for the component family activities that exhibited non-significant p-values for two items when attendance was rated.

The use of PmP as a tool to evaluate intervention requires further research within the South African context. It is also recommended that a larger sample size be acquired in future studies.

Acknowledgments

The contribution of Master’s student Colette Lesego Buthelezi to this project is acknowledged.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Funding from the National Research Foundation (NRF) of South Africa and Swedish Foundation for International Cooperation in Research and Higher Education (STINT) is hereby acknowledged. The content of this paper is solely that of the authors and does not necessarily represent the official view of the funders.