Pilot Study on the Use of a Video-Based Metric Goal Attainment Assessment

ABSTRACT

The scientific investigation of developmental neurorehabilitation has been negatively impacted by a lack of outcome measures sensitive to interventional effect, and is currently in need of assessment measures that can be used during social distancing protocols. Here we report on the usability and feasibility from a pilot test of a novel assessment methodology. This preliminary data demonstrate the usability and feasibility of this assessment system, suggests that it is sensitive to change, and supports the further development and testing of this assessment. The assessment methodology, here presented, generates a percent change score based on the metric measurement of individualized goals before and after intervention, thereby facilitating comparisons between groups and across goals. Moreover, it can be conducted remotely and introduces the novel element of goal appropriacy assessment such that goal selection, as well as goal attainment, are assessed via metric measurement.

KEYWORDS:

• assessment
• telehealth
• pediatric neurorehabilitation

Background

Pediatric rehabilitation struggles with a lack of appropriate assessments sensitive to interventional changes.^1–4 The limited availability of outcome measures is one factor that has negatively impacted the community’s ability to conduct scientific investigation into many pediatric rehabilitation treatments,⁵ and has contributed to there being very few evidenced-based therapies currently in use in pediatric rehabilitation.⁶ Perhaps the most cited reason why current standardized assessments fail to capture interventional effects is that they are designed to measure and compare children with disabilities to ‘typical-developmental trajectories’.^3,5,7 In other words, they are designed to compare children to age-matched peers, presuming that pediatric rehabilitation treatments, if efficacious, would move children closer to age-matched peers or ‘typical’ trajectories of development.⁷ However, it has been well documented that children with disabilities do not follow typical trajectories of development, and in fact, children with disabilities usually develop skills at different rates and ages, even in different orders, in comparison to age-matched peers.^1,3,5 This makes the logic of using many standardized assessments based on ‘typical’ developmental trajectories problematic.

Some attempts to address this issue have included creating new outcome measures. For example, the Gross Motor Functional Measures is a standardized assessment designed to measure gross motor changes in children with Cerebral Palsy (Russell et al., 1989); the Pediatric Evaluation of Disability Inventory is a self-report tool designed to measure functional skills and levels of independence across skill categories⁸; and the Assisting Hand Assessment is a video-scored assessment for understanding the use of the paretic arm and hand as an ‘assisting hand’ in children with hemiparetic Cerebral Palsy.⁹ Each of these tools has specific criteria for both administration and scoring but are designed to address very limited outcome constructs, and are usually validated within diagnostic categories, which limits their generalizability across diagnoses. This type of construct specific approach is in stark contrast to how most pediatric rehabilitation facilities and therapists work, where children are treated with common interventions and common treatment goals across a myriad of diagnostic categories. An assessment methodology that could measure interventional effect across diagnostic categories, disability levels, and treatment types would be a valuable addition to the field.

An approach, developed in the mental health field, as a means to assess idiosyncratic trajectories and patient-specific goals, is Goal Attainment Scaling.¹⁰ This approach has been adapted by the rehabilitation community to allow therapists to measure treatment efficacy for patient-specific goals along an ordinal scale of expected change. A score of zero represents the achievement of the expected change, a score of +1 or 2 is given for gains above what was predicted by the treatment team and −1 or 2 for change less than what was expected by the treatment team. While this system has demonstrated sensitivity to change for the treatment of a variety of pediatric disorders,^11–13 multiple limitations exist. Perhaps the most significant limitation involves the use of an ordinal scale of measurement, followed by a lack of guidance about operationally choosing and or defining measurement points within the scale.¹⁴ In essence the process is always hampered by potential validity problems with goal choices and development.¹⁵ The subjectivity involved in defining treatment outcome expectation and goal selection makes comparisons between the treatment response of different individuals or groups challenging at best.^16,17

The following reports on the feasibility and usability of an alternative goal-based assessment that allows, like goal attainment scaling, for the assessment of individualized goals along idiosyncratic developmental trajectories, but uses a metric (interval or ratio variable) scale of measurement. The system does not base the measurement metric on the treatment team’s expectations, but rather on the objective and metric measurement of specific goal achievement specifications. It uses the treatment teams expected levels of change, alternatively, as a measure of the appropriacy of goal selection. This is done by comparing expected change to the metrically measured change. Both of these additions to a goal attainment assessment methodology can be important contributions to the pediatric rehabilitation community. These contributions allow for the measurement of individualized outcomes via higher levels of measurement and might help therapists alter and better individualize treatments by identifying when treatment goals are either too easy (accomplished quickly) or too hard.

Since the rehabilitation community adopted goal-based treatment and assessment there has been a significant accumulation of literature on how families and patients feel about goal development,¹⁸ goal development theory and process,^19–21 and how goal-setting influences outcomes.²² Overall, both therapists and families repeatedly communicate that treatment selection should be based on individual goals and needs. Despite this, and the fact that goals are developed for the purpose of directing treatment, there has been very limited research on if the goals identified are appropriately selected. In other words, there is no current measurement to help therapists understand if the goals they have set are 1) being targeted by a given treatment and if they are 2) appropriate for an individual child. There is almost no guidance for identifying what each child’s level of need is or what treatment goals should be for a given condition or skill level; rather, therapists are meant to rely on a clinical intuition based on experience. Occupational therapists are taught, for example, that they are to create treatment tasks that are a ‘just right challenge’,²³ meaning tasks that challenge the child toward the next level of developmental skill but not so challenging as to negatively impact the child. Therapists are encouraged to find this level through trial and error. We are proposing an assessment system designed so that the attainment of treatment goals provides insight into this process by measuring goal appropriacy while also simultaneously assessing responsiveness to treatment by metrically measuring goals with measurement variables that allow comparison across goals and children. This data could then be used to better ‘shape’ treatment activities to truly match each child’s specific needs. Moreover, this assessment system is based on recordings of directed interactions with children making it a viable candidate methodology for telehealth adaptation, much needed in the current environment.

We tested our goal appropriacy metric on a series of children with a wide range of ages, diagnoses, disability levels, and goals, all of whom were treated with an intensive therapeutic protocol. Intensive treatments were targeted toward each child’s goals. The goal attainment tool is scored via video recordings of treatments. The goal appropriacy tool compares this change with the therapists’ expected to change. We felt that examining these tools via intensive therapeutic treatment bursts provided a unique research opportunity because, if efficacious, treatment goals should be changing rapidly. We hypothesized that this assessment system would be sensitive to treatment change regardless of the child’s age, diagnosis, disability level, or goals, would allow meaningful and valid comparisons between treatment groups and goals, and would be able to distinguish between goals that were appropriately set, that were set too low (too easy) and those that were set too high (too challenging).

Methods

Participants

The collection and use of this data were approved by Virginia Tech’s Institutional Review Board, and all families provided consent for their participation. There were 9 participants with an average age of 48 months (s.d. = 34.43) and a range of 22–144 months of age. Commonly, they were all diagnosed with global developmental delay, with etiologies of cerebrovascular, toxic, and genetic injury. The children varied in their degree of disability with a range of initial assessment scores of: Peabody from 134 to 336 (s.d. = 112.5), GMFM from 65 to 101 (s.d = 16.4). All participants were consented and the use of the gathered data was approved by Virginia Tech’s Institutional Review Board. Demographic data, presentation, treatment hours, and participant background are listed in Table 1.

Table 1. Standard of care, presentation, background and treatment hours

Child	Standard of Care	Age	Sex	Presentation	Background	Total Tx Hours
A	Physical therapy (PT)1 hour 3x a month Occupational therapy (OT)1 hour 2x a month Speech therapy 1 hour 2x a month	28 m	m	Diagnosed with Kernicterus the child presented with global motor impairments. The child has choreoathetoid movements, dystonia, and limitations in visual motor control. Upon initial assessment the child could not sit independently, maintain head control, or communicate.	White child of a middle-class family with parental and extended family caregiving structure.	37.5
B	PT 1x a week OT 2x a week Speech Therapy 2x a week Hippotherapy 2x a week Feeding therapy 1x a week Aqua therapy 1x a week.	4y	m	Diagnosed with Dystonic Cerebral Palsy secondary to stroke the child presented with global motor impairments, lateralized deficits, limited communication, and cognitive delays. The child could not crawl, walk or transition between sitting positions. He engaged in avoidance behaviors such as turning away and often ignored directions.	White child of a middle-class family with parental and extended family caregiving structure.	117.5
Child C	PT 1 hour 2–3x a week OT 1 hour 2x month of Speech therapy 2x a month	27 m	f	Diagnosed with ventriculomegaly of unknown etiology the child presented with global motor, visual, and cognitive impairments. Upon initial assessment the child had not attained independent static or dynamic sitting balance, nor could she maintain head control. She neither spoke nor engaged in self-feeding. The child did not crawl, scoot, or pivot toward objects of interest but could stand when assisted.	White child of a middle-class family with parental and extended family and professional caregiving structure.	54
Child D	PT 1x a week + 1x every 2 weeks OT 1x a week Speech Therapy 1x every 2 weeks Aqua therapy 1x every 2 weeks	22 m	m	Diagnosed with GDD secondary to stroke the child presented with global motor, visual, communication and cognitive impairments. He had limited awareness of his environment with minimal volitional movement productions. He had no communication skills, and could sit independently for brief periods. Child had no mobility and limited ability to transition between positions.	White child of a middle-class family with parental and extended family caregiving structure.	60
Child E	PT 1x a week OT 1x a week Speech Therapy 1x a week	4y	f	Diagnosed with Microcephaly/Cerebral Hypoplasia secondary to a CASK-mutation the child presented with a limited ability to follow directions, and tendency to respond to verbal prompts by echoing the last words or phrases. She had behavioral outbursts and difficulty with social interaction and struggled to be still or focused. She had full mobility and grasp but poor coordination, environmental maneuvering, and motor planning.	White child of a middle-class family with parental and extended family caregiving structure.	40
Child F	OT 1x a week PT 1 x a week Speech Therapy 1 x a week Specialized instruction 1x a week Feeding Therapy 1 x a week	24 m	f	Diagnosed with microcephaly and hypotonia secondary to CASK-mutation the child presented with limited mobility, and was unable to creep or crawl forward. She was able to attain sitting balance, but struggled to maintain it. She had limited grasp and no language. She was very limited in her ability to follow directions.	White child of a middle-class family with parental and extended family caregiving structure.	40
Chld G	PT 7x in 9 weeks OT 7x in 9 weeks Speech therapy 1 hour 1x a week.	7y	f	Diagnosed with microcephaly and hypotonia secondary to CASK-mutation the child presented with no spontaneous speech, she had difficulty following directions, and tended to choose the last option presented to her. She had full mobility and grasp but struggled with environmental maneuvering.	White child of a middle-class family with parental and extended family caregiving structure.	40
Child H	PT 40 minutes 1x a week OT 20 minutes 2x a week	12	f	Diagnosed with microcephaly secondary to CASK-mutation the child presented with full mobility, but difficulty with coordination, planning and environmental maneuvering. She had trouble following directions and had no spontaneous communication, neither signs nor language. She had difficulty sitting still.	White child of a middle-class family with parental and extended family caregiving structure.	80
Child I	PT 4 hours a week OT 1 hour a week Speech Therapy 1 hour a week	36 m	m	Diagnosed with microcephaly and cerebellar hypoplasia secondary to a CASK:c.2018 C > T (p.Pro673Leu) (NM_003688.3) mutation; (Clinvar accession no. RCV000443443.1) the child presented with motor, visual and cognitive impairments. The child had GMFCS level of 5 and could neither speak, maintain head control nor self-feed. The child could sit with assistance, could roll to both sides and had achieved a modified ‘commando crawl’.	White child of a middle-class family with parental and extended family and professional caregiving structure.	77

Intervention

All children received 3–4 hours of intensive neuromotor intervention 5 days a week for 3–4 consecutive weeks (See Table 1 for total treatment hours). The intervention delivered was an adapted version of ACQUIRE Therapy (See: DeLuca et al., 2007 for a manualized version of the protocol) which is an operant conditioning-based intervention structured around movement-based playful and functional activities which are progressively refined and shaped toward targeted movements and behavior. Importantly, operant conditioning is designed to maximize the learning of skills and therefore goal development. The targeted skills associated with the child’s goals are identified, and broken down into lower-level tasks, for instance a dynamic sitting goal would be broken down into supported sitting, followed by propped sitting, followed by independent sitting, finishing with the goal of dynamic sitting. These tasks are then taught and progressed, through the use of highly motivating toys and activities, toward the final goal and reinforced via the operant conditioning specific timing schedules for reinforcement.²⁴ The specific activities engaged in during treatment and the timeline of the activites were child dependant based on the child’s starting skill level and identified goals. The component parts of each targeted behavior were progressed and shaped through the selection of play activity, positioning, and demand level. For example, when a targeted goal was increased duration of independent sitting for a child that had no sitting balance, treatment activities working on sitting would start with a child receiving reinforcements for fully supported sitting, then, progressively reinforcement would only be given if the child was able to maintain sitting when supports were lessened and then removed, then, finally the child would be rewarded for increasing durations of independent sitting.

Operant conditioning is based on decades of scientific experimentation presented in the psychology and education literature^25–27 for typically developing children, but has not been routinely adopted in therapeutic efforts in pediatric rehabilitation. DeLuca, et al.^28–31 have developed operant conditioning paradigms for pediatric rehabilitation that are definitional to ACQUIRE Therapy, which has demonstrated efficacy through scientific investigation for children with cerebral palsy. Specifically, the MR 3 Cycle as seen in Figure 1 ²⁸ provides a pictorial representation of this process. However, it relies heavily on a clinical intuition for the development of a treatment plan, the progression process, and importantly, goal selection, which is problematic because this is a common failure point.³² Therefore, this has driven our attempt to here define an assessment methodology that provides data about the appropriacy of goals selection.

Figure 1. The MR3 cycle

Standardized Assessments

Children were assessed before and after each intervention with the following standardized assessments: Bayley Scales of Infant Development³³; Pediatric Evaluation of Disability Inventory⁸; Peabody Developmental Motor Scales³⁴; and The Gross Motor Function Measure³⁵ where appropriate, as based on the treatment team’s assessment of appropriacy. (Full assessment data available on request)

Goal Development

Families communicated a list of goals to the treatment team, which consisted of therapists and researchers. In the case of this pilot study the children included were unable to make their goals known to the treatment team for various reasons including age, disability level, and communication deficits, but for future use, whenever possible the patients should be included in the goal identification process. The treatment team chose 3 goals from among that list to be used for this pilot study. The goals were defined and the metrics used to measure the attainment of these goals were operationalized by the treatment team (See Table 2). The treatment team then determined how much change they projected for each goal as a result of the intervention (See Table 5). Goals were discussed with each family and were based on the family’s near and long-term goals for the child. These goals were then operationalized by the treatment team and treatment activities were chosen and guided by these goals.

Table 2. Participant goal definition and metric measurement specification

Child	Goals	Measurement Specifications
A	Improved: Sitting balance 1) -Duration 2)-Frequency 3) Communication skills	Sitting balance was defined as the maintenance of a seated posture without the assistance of a person or supportive device, allowing for arm propping. We measured sitting duration as the longest interval of uninterrupted sitting balance in 5 minute analysis windows. We measured sitting frequency as the total number of seconds of sitting balance in 20 minute analysis windows.Communication was defined as the production of a word or sign to communicate an idea or feeling, to indicate a request or engage interaction. We measured the number of successful productions in 5 minute analysis windows.
B	Improved: 1) Reaching success 2)Task engagement 3)Communication	Successful reaching was defined as contact with the targeted object using the involved hand. We measured the ratio of successful reaches per attempts in 10 minute analysis windows. Task engagement was defined as attention to and involvement in the task demand. We measured the ratio of time spent on task and time spent engaged in avoidance behaviors within 10 minute analysis windows. Communication was defined as the production of a clear word or sign such that the child’s intention is successfully translated to another person. Repetitive statements were not counted, nor were words not intended to communicate e.g. a patterned utterance. We measured communication as the count of communications in 10 minute analysis windows.
C	Improved: 1)Postural control 2)Sitting balance 3)Targeted reach	Postural control is defined as the ability to maintain the head and torso in an upright posture wherein deviations greater than 10 degrees, not associated with visual tracking or attention shifts. We counted the number of these deviations within 5 minute analysis windows. Sitting balance was defined as maintenance of sitting balance with arm propping allowed. We measured sitting duration in 5 minute analysis windows. Targeted reach is defined as contact with target, repeated patting touches were not counted. We measured a count of successful reaches in 5 minute analysis windows.
D	Improved: 1)Reaching success 2)Postural range 3)Time spent on task	Successful reaching was defined as contact with the target. We measured the count of successful reaches in 10 minute analysis windows. Postural range is measured as the ratio of head only, vs head and torso rotations toward the affected side in 10 minute analysis windows. Time on task is measured as a ratio of time on task vs. time in avoidance behaviors within 10 minute analysis windows.
E	Improved: 1)Communication 2)Ability to follow directions 3)Time on task	Communication was defined as the use of words to express a thought or feeling. Echolalic language was not counted as communication. We measured the count of communications in 10 minute analysis windows. Following directions was defined as the completion of a stated direction. The direction was considered completed as long as the child maintained the intention to execute the behavior. Assistance, physical or attentional, was allowed to support the completion of the child’s intention. We measured a count of completed 1 step tasks within 20 minute analysis windows. Time on task was defined as the instances wherein the child was engaged in the task at hand rather than avoidance behaviors. We measured the ratio of time on task vs time spent on avoidance behaviors in 10 minute analysis windows.
F	Improved: 1)Ability to follow directions to “give” and “put in” 2)Use of the signs for “more” and “all done” 3)Postural control	Following directions was defined as the completion of a stated direction. The direction was considered completed as long as the child maintained the intention to execute the behavior. Assistance, physical or attentional, was allowed to support the completion of the child’s intention. We measured the ratio of completed tasks vs. requests in a 20 minute window. The use of the specified signs was defined as the presence of the gestures assigned to ‘more’ and ‘all done’. We measured a count of the signs in 20 minute analysis windows. Postural control was defined as the maintenance of sitting balance without a deviation greater than 15 degrees in any direction not associated with attention shift or visual tracking. We measured a count of deviations in 10 minute analysis windows.
G	Improved: 1)Use of 2 or more word phrases 2)Use of descriptive words 3)Ability to follow multi step directions.	2 word phrases was defined as any communicative phrase consisting of 2 or more words. We measured a count of 2 word phrases in 20 minute analysis windows. Descriptive words were defined as adjectives, quantifiers or qualifiers. We measured a count of descriptive words in 20 minute analysis windows, Following multistep directions was defined as the completion of a stated muti-step direction. The direction was considered completed as long as both steps were attempted and the child maintained the intention to execute both behaviors. Assistance, physical or attentional, was allowed to support the completion of the child’s intention. We measured a count of completed multistep directions in 20 minute analysis windows.
H	Improved: 1)Ability to follow directions 2) Time on task. 3)Use of signs for “more” and “all done”	Following directions was defined as the completion of a stated direction. The direction was considered completed as long as the child maintained the intention to execute the behavior. Assistance, physical or attentional, was allowed to support the completion of the child’s intention. We measured the ratio of tasks completed vs. requests in 20 minute analysis windows. Time on task was defined as the instances wherein the child was engaged in the task at hand rather than avoidance behaviors. We measured a ratio of time spent on task vs. time engaged in avoidance behaviors in 10 minute analysis windows. Use of signs was defined as the use of defined gestures to communicate. We measured a count of the use of signs in 10 minute analysis windows.
I	Improved: Sitting balance 1)-Duration 2) -Frequency 3) Postural control	Sitting balance was defined as the maintenance of a seated posture without the assistance of a person or supportive device, but allowing for arm propping and allowing stand by assistance by the treatment provider, however the child’s torso cannot be resting on an object or person. Duration was measured as the longest interval of uninterrupted sitting balance in 5 minute analysis windows, and frequency was measured as the total number of seconds in sitting balance in 20 minute analysis windows. Postural control was defined as maintenance of torso and head at midline with deviations of no more than 20 degrees. Deviations from midline associated with attentional shifts or visual tracking will not be counted. Only those instances where the child is seated in a way that will allow deviations of greater than 20 degrees are viable for analysis. We measured the count of deviations in 5 minute analysis windows.

Video Analysis

Videos from the first and last weeks of treatment were recorded using either tablet devices, or GoPro™ cameras. These videos were divided into as many analysis windows of specified length based on goal measurement specification (see Table 2) as possible from available video using Microsoft’s™ Photo Editing software. Therefore, a video from the first day of treatment with a total length of 75 minutes would result in three 20 minute videos, with the remaining 15 minutes of video being eliminated for analysis. This process would be repeated for all videos available from the first and last three days of treatment. These video windows become the pool from which videos are randomly selected for analysis of pre and post-treatment goal attainment. The analysis windows within this pool were assigned numbers within a range and were randomly sampled for analysis, using a random number generator to select numbers within this range, the analysis window with the number nearest to the randomly generated number was selected for analysis. Videos were sampled from the pool of possible videos at a rate of 25–75% of the total number of available videos. Video windows wherein tasks associated with goal attainment were absent or limited to less than 50% of the video were excluded. For instance, if 5 minutes of a 10 minute video involve therapeutic activities that have the child in a supine position, it would be excluded for analysis for a sitting goal. All video analysis was completed by the first author, who served as a member of the treatment team for 4 of the 9 children. Future reliability and validity studies will explore the need for multiple video coders. For the purposes of the development of this methodology the first author was not blinded to a video’s status as pre or , but future use and research should include assessor blinding and should probe for differences in the reliability and validity of assessors associated with the treatment team and not-associated.

Data Processing Pipeline

Based on the definitions and goal specification developed by the treatment team, the video assessor, in this pilot study the first author was the sole video assessor and is trained in neuromotor assessment, collects the counts and ratio measurements associated with goal attainment from the pre and post-treatment videos. In order to calculate the change score the participants’ pre-treatment goal measurement score is subtracted from their post-treatment score to get a difference value, that value is then divided by the pre-treatment score and multiplied by 100 to get the percent change score. For the purposes of this publication, we will use Child A to demonstrate the data collection and processing pipeline for one goal.

Results

Percent Change

Table 4 shows the percent change obtained for each goal and for standardized assessment. On average children had a percent change score of 216 (s.d = 189) across all goals. The range across goals was sometimes dramatic with the least change being 5 and the most being 1770%, however this range can be dramatically reduced with modifications discussed below. Table 4 also shows percent change on standardized assessments where on average children had a percent change score of 66 (s.d = 98). Child H was behaviorally unable to obtain a score on a standardized assessment prior to treatment.

Table 3. Flowchart of data processing pipeline

Goal Definition and Measurement Specification: Goal 3- Communication was defined as the production of a word or sign to communicate an idea or feeling, to indicate a request or engage interaction. We measured the number of successful productions in 5 minute analysis windows.

Video Analysis: Available video from the first and last three days of treatment were broken into 5 minute windows. The first three days had the least number of analysis windows in the video pool with a total of 5 available videos, so 3 videos were selected for a maximum sampling rate of 60%, the last three days had 10 available videos with a sampling rate of 30%. The Random number generator program generated the numbers 342, 269, and 556 for videos for the pretreatment condition (1st 3 days) and 743, 364, and 503 for post treatment condition. These were closest to videos numbered 332, 215, and 521 from pretreatment pool and numbers 651, 316, and 503 from the posttreatment pool. For the pretreatment condition the scoring was as follows: 332: 0 communications in five minutes 215: 0 communications in five minutes 521: 2 communications in five minutes Average Pretreatment Communications.66 per five minutes. For the post treatment condition the scoring was as follows: 651: 0 communications in five minutes 316: 4 communications in five minutes 503:0 communications in five minutes Average Post treatment Communications 1.33 per five minutes.

Calculate Percent Change: Percent Change formula: Post value-Pre value/Pre value x 100 Goal 1% Change for Child A: Step 1: 1.33-.66 = 0.67 Step 2: 0.67/.66 = 1.015 Step 3: 1.015 × 100 = 101.5%

Calculate Goal Appropriacy Metric: Predicted Percent Change: 30% Actual Percent Change: 101% Difference: 71% We have set our cutoff for appropriacy at 50%, 71%>50% therefore this goal was deemed inappropriate.

Table 4. Percent change on goals and standardized assessments (Values rounded up to nearest whole number for ease of reporting. Where values are negative, but indicate an improvement on a goal, absolute value was used)

	Goal 1	Goal 2	Goal 3	Avg. Across Goals	Standardized Assessment
Child A	169	20	102	97	115
Child B	116	50	798	321	43
Child C	52	67	650	256	11
Child D	1770	96	100	655	6
Child E	141	58	40	80	6
Child F	23	650	20	231	26
Child G	73	100	14	62	29
Child H	130	5	400	178	NA
Child I	11	136	−40	62	291

Table 5. Goal appropriacy data

	Expected Change	Actual Change	Difference
Child A	Goal 1: 100% Goal 2: 50% Goal 3: 30%	Goal 1: 169% Goal 2: 20% Goal 3: 102%	Goal 1: 69%+ Goal 2: 30%* Goal 3: 72%+
Child B	Goal 1: 75% Goal 2: 75% Goal 3: 100%	Goal 1: 116% Goal 2: 50% Goal 3: 798%	Goal 1: 41%* Goal 2: 25%* Goal 3: 689%+
Child C	Goal 1: 100% Goal 2: 50% Goal 3: 50%	Goal 1: 53% Goal 2: 67% Goal 3: 650%	Goal 1: 47%* Goal 2: 17%* Goal 3: 600%+
Child D	Goal 1: 100% Goal 2: 50% Goal 3: 75%	Goal 1: 1770% Goal 2: 96% Goal 3: 100%	Goal 1: 1670%+ Goal 2: 46%* Goal 3: 25%*
Child E	Goal 1: 200% Goal 2: 100% Goal 3: 100%	Goal 1: 141% Goal 2: 58% Goal 3: 40%	Goal 1: 59%- Goal 2: 42%* Goal 3: 60%-
Child F	Goal 1: 100% Goal 2: 100% Goal 3: 100%	Goal 1: 23% Goal 2: 650% Goal 3: 20%	Goal 1: 77%- Goal 2: 550%+ Goal 3: 80%-
Child G	Goal 1: 200% Goal 2: 100% Goal 3: 100%	Goal 1: 73% Goal 2: 100% Goal 3: 14%	Goal 1: 127%- Goal 2: 0* Goal 3: 86%-
Child H	Goal 1: 100% Goal 2: 100% Goal 3: 100%	Goal 1: 130% Goal 2: 5% Goal 3: 400%	Goal 1: 30%* Goal 2: 95%- Goal 3: 300%+
Child I	Goal 1: 100% Goal 2: 100% Goal 3: 100%	Goal 1: 11% Goal 2: 136% Goal 3: −40%	Goal 1: 89%- Goal 2: 36%* Goal 3: 140%-

Goal Specific Assessment Compared to Standardized Assessments

Figure 2 pictorially represents the average percentage change on goals compared to average percentage change on standardized assessments. Figure 2 illustrates the way that goal specific assessment can capture large functional changes across children with different diagnostic categories, disability levels, goals, and ages, using a common metric- percent change, and can then compare that change to change on standardized assessment using the same percent change calculation (For percent change calculations see Appendix A). Moreover, in Figure 2, by comparing percent change on standardized assessment to the change captured by goal-specific assessment, one can deduce that for some children, goal-specific assessment may be picking up on change not otherwise captured by standardized assessments.

Figure 2. Percent change on goals and standardized assessments

Goal Appropriacy Metric

We determined that goal appropriacy would be defined as those goals where the difference fell between 0 and 50%. All goals that were deemed appropriate are marked with a *, those where the change was greater than expected are marked with a + and those where change was less are marked with a – in Table 2. There were 11 appropriate goals and 16 inappropriate goals.

Discussion

Metric measurements for all goals were identified and these measurements were sensitive to interventional effect (See Table 1). Coding of video and the determination of measurements was straightforward and unambiguous. The use of a percent change score to capture treatment effects allowed for comparisons between goals, across children, and in comparison to standardized assessments (See Figure 2 and Table 1). We were able to identify a wide range of treatment responses that appeared to go beyond those captured by standardized assessments (See Figure 2). For example, eight of the nine children showed larger gains on goals than on standardized assessments. Importantly, the development of this assessment methodology made it clear that using a ratio metric measurement has many advantages. In those instances where goals were specified as ratio measurements it was possible to use shorter analysis windows of common duration, drastically streamlining the process. Ratio data is quantitative data that allows the completion of the task demand to be a function of opportunity. Measuring the completed task relative to demand makes it possible to more accurately gauge the child’s rate of performance in shorter windows and allows commonly sized windows, regardless of goal, because the measurement is a function of opportunity rather than time.

Moreover, this data makes clear that this novel tool for gathering usable data on the appropriateness of goal setting is feasible and usable (See Table 2). The goal appropriacy data facilitated an assessment of the relative appropriacy of the goals selected. Out of 27 goals across 9 children, 11 goals were identified as appropriate. Those goals were primarily sitting balance and postural stability goals, with the remainder split between staying on task and following one-step directions. Of the inappropriate goals, for 7 the child showed greater gains than expected, indicating that the goals could have been more challenging, and for 9 the children showed less gains than predicted, indicating that the goals could have been reconceptualized so that activities were targeted toward more fundamental tasks in the progressive shaping of that particular behavior. For instance, for Goal 3, Child A, used in the pipeline (Table 3), found to be inappropriate, communication could have been broken down into engaging in co-attentional behaviors with an other, seeking or engaging another for interaction, and the use of gestures to attempt communication, as opposed to a goal focused on the use of structured signs and verbal communications. The goal appropriacy data here presented provides therapists and researchers with opportunities to explore goal appropriacy trends such as these in their own data. Overall this study presents a methodology that can inform therapists in goal-setting and measuring treatment success. The process can also aid therapists when making decisions about how to change treatment processes by informing them if children are being appropriately challenged.

Limitations

This pilot study is limited by a somewhat small sample size, but further research on this assessment methodology will involve a larger population. The development of the range for goal appropriacy (from 0 to 50% difference) was somewhat arbitrary, and further refinement of the change estimation process is proposed below. Sampling rate specifications were likewise somewhat arbitrary. Further analysis is necessary to: 1) determine the minimum duration a video window need to be to capture enough data to make a fair assessment of goal attainment, and if a standard window size can be used for all goals 2) how many windows are selected out of the pool of available windows and 3) how many windows must there be in this pool at minimum to ensure proper random sampling practices. Finally, because this original protocol doesn’t exclude starting values of zero, the calculation of change scores required the adoption of the common solution of giving the child a minimal starting value other than zero, such as.1. However, this practice dilutes the interpretability of the change scores here reported, and suggested refinements are discussed below. We acknowledge there is a lack of diversity in our research populations and are currently pursuing research into what barriers might exist to prevent more diverse study populations.

Future Directions

Goal and Measurement Specification

In ACQUIRE therapy, as is the case in many common interventions, tasks become progressively more difficult and are chosen to make goal attainment more difficult over the course of treatment as part of the process of shaping the behavior toward greater proficiency. This created a challenge when assessing early and late interventional performance on these tasks. It is therefore suggested that for progressive interventions goal specification should include a difficulty offset based on the therapist’s intended progression of task difficulty. For instance, in the goal specifications reaching below shoulder level would be assigned 0 difficulty offset, but reaching at shoulder level or over the head would have a preassigned difficulty offset or handicap. Using this system 5 reaches at or below shoulder level during the first week wouldn’t count the same as five reaches over the head the last week.

Change Score

When a child scores a zero at baseline, a percent change score can’t be calculated because one can’t have a zero in the numerator. A common way to address this is to assign the child a minimal score of .01 or 1 in lieu of zero, however, using this approach led to enormous percent change numbers, diluting the meaningfulness of the change scores and the difference scores on goal appropriacy. Revising the goal estimation protocol to exclude starting values of zero will help maintain the interpretability of the goal appropriacy data and may even enforce the appropriacy of goal selection. We propose revising this protocol so that a goal for which a child has a starting value of zero is considered an inappropriate goal not properly scaled to allow for successive shaping. In such a case the original goal should be reconsidered and broken down into more achievable component goals that can be shaped toward the original goal and giving a starting attainment value greater than zero.

Goal Appropriacy Tool

The extremely wide range of difference scores (0–1670) likely indicates that the tool needs further refinement. These large difference scores result from some children with baseline scores of zero which inflated their change scores (see above) and from therapists rarely expecting change greater than 100%. We propose a protocol modification wherein expected change is no longer reported in terms of expected percent change, but, in keeping with the other components of this protocol, is reported as a metric expectancy measurement grounded in a baseline assessment of goal attainment that must be other than zero. Therefore, in future versions of this protocol a baseline measurement of goal attainment will be determined and therapists will report their expected change in terms of change from baseline, e.g. “child will go from 10 seconds of independent sitting at baseline to 30 seconds at post-treatment assessment”. The metric estimate is then converted to a percent change and compared to actual goal change as per the current protocol.

Telehealth

Because this is a video-based assessment that could potentially be applied to videos of directed parent–child play, this methodology may be of particular use during post-COVID-19 times wherein limiting in-person assessment may be advantageous. Further research is underway to explore the potential for this assessment methodology to be used in telehealth settings.

Conclusion

In conclusion, based on the usability and feasibility data here presented, we believe that developing and applying the above protocol modifications, followed by fully powered reliability and validity studies would be warranted as this assessment system has the potential to be a highly valuable and informative addition to the assessment toolbox, supporting the development of more appropriate goals, the measurement of goal appropriacy, and the comparison of interventional effect between populations, as well as between and across goals.

Disclosure Of Interest

The authors have no conflicts of interest to disclose.