Does strategy instruction on the Rey-Osterrieth Complex Figure task lead to transferred performance improvement on the Modified Taylor Complex Figure task? A randomized controlled trial in school-aged children

Abstract

Objective: Providing children with organizational strategy instruction on the Rey Osterrieth Complex Figure (ROCF) has previously been found to improve organizational and accuracy performance on this task. It is unknown whether strategy instruction on the ROCF would also transfer to performance improvement on copying and the recall of another complex figure. Methods: Participants were 98 typically developing children (aged 9.5–12.6 years, M = 10.6). Children completed the ROCF (copy and recall) as a pretest. Approximately a month later, they were randomized to complete the ROCF with strategy instruction in the form of a stepwise administration of the ROCF or again in the standard format. All children then copied and recalled the Modified Taylor Complex Figure (MTCF). All productions were assessed in terms of organization, accuracy and completion time. Results: Organization scores for the MTCF did not differ for the two groups for the copy production, but did differ for the recall production, indicating transfer. Accuracy and completion times did not differ between groups. Performance on all measures, except copy accuracy, improved between pretest ROCF and posttest MTCF production for both groups, suggesting practice effects. Conclusion: Findings indicate that transfer of strategy instruction from one complex figure to another is only present for organization of recalled information. The increase in RCF-OSS scores did not lead to a higher accuracy or a faster copy or recall.

Keywords:

• Complex figure
• strategy instruction
• transfer
• organization
• children

The Rey-Osterrieth Complex Figure (ROCF) (Osterrieth, 1944; Rey, 1941) is a neuropsychological instrument widely used in clinical practice as well as in research (Lezak, Howieson, Bigler, & Tranel, 2012). In the ROCF task, a complex geometrical figure first has to be copied and then immediately reproduced from memory as accurately as possible in an unannounced recall trial (Osterrieth, 1944; Rey, 1941; Strauss, Sherman, & Spreen, 2006). A second, longer delay recall condition (e.g. 20 minutes after the first recall condition) can also be used in this task. That condition was not used in the present study, and therefore, all references to ‘recall’ throughout the manuscript are related to the immediate recall condition.

While the standard procedure of the ROCF task is as described above, the task can also be administered in a stepwise manner. More specifically, the complex figure can be divided into logical steps that highlight the figure’s hierarchical organizational framework (Chen, Cermak, Murray, & Henderson, 1999; Frisk, Jakobson, Knight, & Robertson, 2005; Kirkwood, Weiler, Bernstein, Forbes, & Waber, 2001; Waber et al., 1994). Studies have shown that this stepwise instruction for the ROCF improved copy and immediate recall performance in terms of organization and accuracy (see below for more information on these different scores) on the ROCF, in children aged 6–17 years with and without learning disabilities (Chen et al., 1999; Frisk et al., 2005; Kirkwood et al., 2001; Waber et al., 1994). This stepwise administration format is often used to disentangle the sources of poor task performance: if children perform better with the stepwise administration format, this is likely a consequence of the stepwise administration supporting the application of metacognitive processes involved in the complex figure task, such as selecting strategies, planning, and organizing. Administering the figure in a stepwise manner can also be considered strategy training. More specifically, by presenting the figure in steps, children are trained to use an organizational strategy which they are not always able to produce on their own. Of particular interest when examining the effects of strategy training is the transfer of performance improvement. More specifically, children have to be able to apply what they learned to a different context, for the learned material to be valuable for their development (Klahr & Chen, 2011). Transfer of performance after the strategy instruction on the ROCF to a different task or context would provide evidence for the usefulness of a stepwise administration of complex material in training healthy, typically developing children to use strategies. In adults, strategy training in the form of stepwise administration of the ROCF has been found to lead to significant improvements in performance not only on the trained task, but also on another complex figure (Chen, Hartman, Galarza, & DeLuca, 2012). In children, it remains to be determined whether the improvements in their performance caused by the strategy instruction on the ROCF are also transferable e.g. to other complex figures.

The present study investigated the transfer of performance improvement from one complex figure to another comparable, though still unique figure. The Modified Taylor Complex Figure (MTCF, see Figure 1) (Hubley, 1996, 1998) has been found to be comparable to the ROCF with regard to its accuracy scores (see below), and its relationship with other tests (e.g. other measures of visual spatial abilities) (Hubley, 1996, 1998; Hubley & Jassal, 2006; Hubley & Tremblay, 2002; Yamashita, 2006). Standard instructions for both the ROCF and the MTCF are the same (Hubley, 1996; Strauss et al., 2006). Importantly, the MTCF is composed in a similar manner as the ROCF in terms of the global and essential elements (i.e. a large rectangle/square, horizontal, and vertical centerlines). In addition to these global elements, both figures contain a number of externally attached elements and internal components. Given the structural resemblance of the ROCF and the MTCF, the stepwise strategy instruction as used in previous studies could be applied for both figures. However, it remains to be determined whether children are able to transfer the strategy instruction provided for the ROCF to the MTCF, thereby enhancing their performance on both figures.

Figure 1. Modified Complex Taylor Figure.

Notes: Copyright © 1996, 1998 Anita M. Hubley. All rights reserved. Reproduced by permission from Anita M. Hubley.

As described above, the stepwise strategy instruction for the ROCF is mainly directed at supporting organizational (i.e. executive) processes by sequentially highlighting essential organizational elements. To assess the success of the strategy instruction in improving complex figure task performance, different task parameters can be investigated: organization, accuracy, and completion time. The organization of the complex figure can be quantified by assessing the order in which children copy and/or recall the various elements of the complex figure, whereby starting with the more global elements (e.g. large rectangle/square, horizontal, and vertical centerlines) reflects a better organizational process. To assess the accuracy of the reproduced complex figure, the similarity of the reproduction with the original figure, irrespective of the order in which the figure’s elements were drawn, is examined. Previous studies have found that a better organization is strongly related to a better accuracy as well (Davies, Field, Andersen, & Pestell, 2011; Waber et al., 1994). Finally, the success of the instruction can be inferred by examining completion time of the complex figure task. The completion time reflects the amount of time a child needs to complete the copy and/or recall of the complex figure. The influence of strategy instruction on complex figure task completion time has not yet been investigated. However, highlighting the organizational framework of the complex figure might enable children to reproduce the complex figure more efficiently (i.e. in less time).

In sum, with the present study, we investigated whether a stepwise strategy instruction on the ROCF would lead to transferred improvement on the MTCF. It was predicted that children presented with the stepwise instruction would perform better on the MTCF in terms of (1) better organization, and as a consequence of this improved organization, (2) higher accuracy, and (3) lower completion times during copy and recall than children who were administered ROCF in the standard manner.

Method

Participants and sampling

In total, 221 children aged ten to twelve years were invited to participate. All children had previously participated in a four year longitudinal study of Maastricht University into cognitive development (Martens, 2012). During those four years, all children were consistently administered the same test batteries, making all children comparable in their history of experience with neuropsychological assessments. The ROCF had been administered to all children before, i.e. more than 3 years before the present study took place. None of the children was previously administered the MTCF and none of them had received the stepwise instruction before. Caregivers of 141 children agreed to their child’s participation in this follow-up, corresponding to a response rate of 64%. In the initial study (Martens, 2012), the children had already been screened based on exclusion criteria, i.e. not speaking Dutch fluently, the presence of neurological disorders (e.g. epilepsy) or the use of medication (e.g. antihistamines). For the present study, 19 children (13%) who had been diagnosed with a neuropsychiatric or neurodevelopmental disorder (e.g. dyslexia, attention deficit hyperactivity disorder) or were not attending regular education were excluded. Furthermore, for 15 other children the proposed tasks could not be administered, e.g. due to scholastic absence at the agreed upon testing dates.

Children were randomly assigned to either the stepwise instruction group (n = 54) or the standard administration group (n = 53). For nine children in the stepwise instruction group, technical problems during the assessment compromised the reliability of their data. Therefore, the final sample consisted of 98 children (45 stepwise instruction group) aged 9.48 to 12.63 years (M = 10.60, SD = .70). An overview of the characteristics of the participants per group can be found in Table 1. No significant differences were found between the two groups in terms of age, sex, level of parental education, estimated IQ, or number of days between pretest and posttest. Level of parental education ranged from primary school (1) to university degree (8) (De Bie, 1987; Kalff et al., 2001), on a scaling comparable to the International Standard Classification of Education (UNESCO, 2011). IQ of the children was estimated by combining the Vocabulary and Block Design subtests of the most recent Dutch translation of the Wechsler Intelligence Scales for children – third edition (Kort et al., 2005; Wechsler, 1991) into one IQ score (Sattler, 1992, 2001).

Table 1. Characteristics of the participants per instruction group.

	Stepwise instruction (n = 45)	Standard administration (n = 53)	df	F/χ	p
Age, M (SD)	10.65 (.66)	10.56 (.74)	1.96	.43	.52
Male, N (%)	19 (42)	26 (49)	1	.46	.50
LPE, M (SD)	5.80 (1.24)	5.90 (1.54)	1.96	.13	.72
IQ, M (SD)	101.07 (12.84)	103.34 (13.10)	1.96	.75	.39
Days, M (SD)	28.69 (10.11)	28.45 (7.39)	1.96	.02	.89

Notes:

M = mean, SD = standard deviation, LPE = Highest level of parental education, IQ = Estimated IQ score, Days = number of days between pretest and posttest.

Procedure

The ethics committee of the Faculty of Psychology and Neuroscience of Maastricht University approved the research. After consent of the caregiver was obtained, well-trained research assistants administered the tests reported in the present article as part of a larger test battery. All tests were performed in the same order for each child in a stimulus free room at the schools of the participating children. Testing took place in two sessions. An overview of the testing procedure is given in Figure 2.

Figure 2. Overview of the assessment procedure.

Children’s pretest organizational abilities were assessed in the first session using the standard ROCF administration, (Rey, 1941; Strauss et al., 2006). In the second session, children in the standard administration group were administered the ROCF in the standard manner. For the stepwise instruction group, the parts of the ROCF were presented in three hierarchically organized steps (see Figure 3) (Chen et al., 1999; Kirkwood et al., 2001; Waber et al., 1994). For each step, the children were first asked to point out the parts shown in that step in the complete ROCF and then to copy these elements. If they pointed the elements out wrong, the examiner would point to the correct elements. No feedback was given if they did not copy the elements correctly. In both instruction/administration formats, instructions did not include any mention of a timely completion of the drawings. After they had finished copying the ROCF in three consecutive steps, children were asked to reproduce the figure in an unannounced recall condition. Since this second administration of the ROCF only functioned as a means to introduce a repeated performance of the ROCF with or without strategy instruction, these drawings were not assessed in terms of organization, accuracy or completion time. To evaluate the effects of the strategy instruction, the MTCF, administered in the standard way (Strauss et al., 2006) in the second session, was used as the posttest measure.

Figure 3. Stepwise instruction format for the Rey-Osterrieth Complex Figure.

Notes: The complete Rey-Osterrieth Complex Figure as presented in step 3 was derived from Le Test de Copie d’Une Figure Complexe (Osterrieth, 1944); in the public domain. Elements added in step 1, 2, and 3 were colored red, blue, and green, respectively.

Materials and scoring

For the standard administration of the ROCF and the MTCF, children were presented with the complete figure on an A4 sized paper. During the stepwise instruction, three cards, each depicting parts of the elements of the ROCF, were consecutively shown (see Figure 3). As the stepwise instruction progressed, new elements were added in a different color. Three outcome measures were included for the copy as well as the recall of the complex figures: organizational level, the accuracy score and completion time.

Organization

The organization of the ROCF and the MTCF was assessed with the Rey Complex Figure Organizational Strategy Score (RCF-OSS) (Anderson, Anderson, & Garth, 2001). The RCF-OSS is a process-oriented scoring approach to the ROCF that takes into account the sequence in which various elements of the complex figure are drawn. The RCF-OSS has been shown to be a valid method to measure organizational skills in children, as indicated by the significant linear relationships with other measures that rely on organization skills (Anderson et al., 2001). Furthermore, it has good inter-rater reliability (Anderson et al., 2001; Davies et al., 2011; Martens, Hurks, & Jolles, 2014) and an acceptable one-week intra-rater reliability (Anderson et al., 2001) when used to assess ROCF performance.

During the assessment sessions, the research assistant who was administering the complex figure task kept track of the order in which the child drew the lines by numbering them. The first author (C.R.) scored all complex figures with the RCF-OSS, blinded to group membership. By analyzing the order in which the elements of the complex figures are drawn, each drawing was rated on a seven-point scale, where a higher number represents a higher level of organization: i.e. level 1 = unrecognizable drawing or substitution, level 2 = poor organization, level 3 = random organization, level 4 = piecemeal/fragmented organization, level 5 = part-configural organization, level 6 = conceptual organization, and level 7 = excellent organization (for a description of the scoring criteria for each level, see Anderson et al., 2001). In the RCF-OSS, the elements that contribute most to the organizational level are the base rectangle and the vertical and horizontal midlines, since they play an important role in the correct placement of the internal sections and outside attachments (for definitions, see Anderson et al., 2001). The RCF-OSS was originally developed as a scoring system for the ROCF. However, given the structural resemblance of the ROCF and the MTCF (as described in the introduction), we believe that the RCF-OSS can easily be applied to the MTCF, without significant alterations of the scoring instructions set for scoring the ROCF. Our version of the RCF-OSS, adapted to the MTCF, is presented in Appendix 1. Changes to the original RCF-OSS pertained only to definitions of elements of the figure (e.g. the main element of the ROCF is a rectangle, the main element of the MTCF is a square) and not to the scoring criteria for the various organizational levels. Given that this was the first time that the RCF-OSS was applied to the MTCF, inter-rater reliability of the scoring was assessed. To that extent, a well-trained early-career neuropsychologist scored 20% of the figures with the RCF-OSS. These scores were compared to the scores previously given by the first author (C.R.). Guidelines characterize a reliability coefficient from 0 to .20 as slight, from .21 to .40 as fair, from .41 to .60 as moderate, from .61 to .80 as substantial, and over .81 as excellent agreement (Shrout & Fleiss, 1979). Using weighted kappa analyses, moderate inter-rater reliability of the RCF-OSS was found for the copy condition of the ROCF, κ_w= .50, p = .002, and the MTCF, κ_w= .55, p = .002. Substantial agreement was reached in the recall condition of the ROCF, κ_w= .80, p < .001, and the MTCF, κ_w= .74, p < .001. Even though the inter-rater reliabilities were similar for the RCF-OSS for the ROCF and the MTCF, it was surprising that the agreement in the copy condition was only moderate, in contrast to previous studies who found higher inter-rater reliability (Anderson et al., 2001; Davies et al., 2011; Martens et al., 2014). Following up on these findings, we examined the origins of the differential ratings. Recommendations for future use and some additions to the scoring system will be discussed in the discussion section.

Accuracy

The accuracy scoring system was used to evaluate the accuracy of copy and recall ROCF and MTCF performance (Hubley, 1996, 1998; Strauss et al., 2006; Taylor, 1959). A score of 0, .5, 1, or 2 is given for the accuracy and correctness of placement of 18 elements of the figure, independent of the order in which they were drawn. This score is given irrespective of the order in which the elements are drawn and leads to a sum score ranging between 0 and 36 points, with a higher score reflecting a higher similarity of the drawn figure (either copied or recalled) to the original ROCF or MTCF. Inter-rater reliability has previously been found to be excellent (Loring, Martin, Meador, & Lee, 1990; Shorr, Delis, & Massman, 1992). In the present study, accuracy scoring was performed by two of the authors (C.R. and P.H.) and three well-trained, early career neuropsychologist, who each scored a separate set of complex figures.

Completion time

Completion time for all figures was recorded by starting the timer as soon as the task instruction was completed and stopping the timer when the child indicated to have finished drawing the figure.

Statistical analysis

To investigate the degree to which the accuracy scores, organizational levels, and completion times on the copy and recalled reproduction of the ROCF and the MTCF are associated with each other (thus potentially measuring the same underlying constructs), bivariate correlation analyses were performed for the standard administration group. The influence of strategy instruction on performance on a complex figures task was examined using repeated measures mixed ANOVA, with pretest (ROCF) and posttest (MTCF) scores (i.e. RCF-OSS, accuracy, and completion time for copy and recall) as within-subject factor and administration group (i.e. standard vs. stepwise) between-subjects factor. Partial η squared (${\eta }_p^2$) was computed as measure for effect size. One outlier (defined as a score of more than three standard deviations above or below the variable mean) was identified and subsequently excluded from all analyses. Analyses were carried out with IBM SPSS Statistics 24, α was set at .05.

Results

Bivariate correlation analyses

For the standard administration group, organizational levels were consistently correlated across conditions (i.e. copy and recall) and sessions (i.e. pretest and posttest). Similarly, copy and recall as well as pretest and posttest were correlated for the two other complex figure outcome measures, i.e. accuracy score and completion time. Within one session, organizational level and accuracy score are positively correlated (with the exception of the posttest copy condition). No relations were found between organizational level and completion time or between accuracy and completion time. Given these findings, influence of the strategy instruction, which was mainly directed at improving organization level, was more likely to occur on the RCF-OSS and the accuracy score but not for completion time. An overview of the correlations for the standard instruction group can be found in Table 2.

Table 2. Correlations between complex figure outcomes (i.e. RCF-OSS, accuracy score, and completion time) during pretest and posttest for the standard administration group (n = 45).

			Pretest						Posttest
			Copy			Recall			Copy			Recall
			RCF-OSS	Accuracy score	Completion time	RCF-OSS	Accuracy score	Completion time	RCF-OSS	Accuracy score	Completion time	RCF-OSS	Accuracy score	Completion time
Pretest	Copy	RCF-OSS	–
		Accuracy score	.421^**	–
		Completion time	−.109	.067	–
	Recall	RCF-OSS	.609^***	.512^***	−.026	–
		Accuracy score	.465^**	.550^***	.058	.717^***	–
		Completion time	.208	.228	.521^***	.202	.301^*	–
Posttest	Copy	RCF-OSS	.381^**	.187	−.199	.314^*	.207	−.065	–
		Accuracy score	.173	.500^***	.331^*	.368^**	.517^***	.449^**	.151	–
		Completion time	−.031	−.045	.649^***	−.147	−.192	.504^***	−.150	.136	–
	Recall	RCF-OSS	.276^*	.387^**	−.086	.515^***	.348^*	.167	.444^**	.408^**	−.213	–
		Accuracy score	.275^*	.359^**	.193	.513^***	.583^***	.369^**	.246	.528^***	−.135	.613^***	–
		Completion time	.028	.067	.570^***	−.103	.009	.590^***	−.169	.383^**	.611^***	−.050	.103	–

Notes:

RCF-OSS = Rey Complex Figure Organizational Strategy Score.

^* p < .05.

^** p < .01.

^*** p < .001.

Influence of strategy instruction on complex figure task performance

An interaction between administration group and time of measurement (i.e. pretest and posttest) was found for the organization of the recall of the complex figure F(1, 95) = 9.10, p = .003, ${\eta }_p^2$= .09 (see Figure 4). Simple effects analyses revealed that improvement from pretest to posttest was larger for the stepwise instruction group (p < .001, ${\eta }_p^2$= .46) than for the standard administration group (p = .003, ${\eta }_p^2$= .16). Consequently, average organizational level differed between groups at posttest, F(1, 44) = 7.74, p = .007, ${\eta }_p^2$= .08. Descriptive details for the scores are displayed in Table 3. No interaction between administration group and time of measurement was found for copy organization, F(1, 94) = .01, p = .93, ${\eta }_p^2$= .00, copy accuracy, F(1, 94) = 3.25, p = .075, ${\eta }_p^2$= .03, recall accuracy, F(1, 95) = 1.26, p = .27, ${\eta }_p^2$= .01, copy completion time, F(1, 95) = .51, p = .478, ${\eta }_p^2$= .005, or recall completion time, F(1, 95) = 1.524, p = .220, ${\eta }_p^2$= .016.

Figure 4. Graphic representation of the influence of strategy instruction on organizational recall performance.

Notes: Organizational level was assessed with the Rey Complex Figure Organizational Strategy Score (Anderson et al., 2001). Scores could range van 1 to 7. The difference between standard and the stepwise administration was significant at posttest.

Table 3. Complex figure task performance for the stepwise instruction and the standard administration group.

		Copy		Recall
		Pretest	Posttest	Pretest	Posttest
		M (SD) [range]	M (SD) [range]	M (SD) [range]	M (SD) [range]
RCF-OSS	Stepwise	4.38^1 (.78)	4.80^1 (.50)	3.22^2 (1.36)	4.60^2 (.99)
		[3–6]	[3–6]	[2–6]	[2–6]
	Standard	4.15^1 (.94)	4.54^1 (.75)	3.42^2 (1.26)	3.96^2 (1.24)
		[2–6]	[3–6]	[1–6]	[2–5]
Accuracy	Stepwise	25.09 (3.96)	26.79 (3.47)	12.57^1 (5.93)	15.18^1 (5.22)
		[16.00–33.00]	[19.00–33.50]	[3.50–23.00]	[4.00–25.00]
	Standard	24.93 (6.00)	24.68 (4.11)	12.67^1 (6.12)	13.89^1 (5.92)
		[10.50–35.00]	[17.00–32.00]	[2.50–26.00]	[2.50–29.00]
Completion time (seconds)	Stepwise	223.27^1 (74.87)	137.20^1 (41.83)	130.96^1 (47.80)	90.71^1 (28.81)
		[62–421]	[72–255]	[48–243]	[46–185]
	Standard	228.60^1 (81.59)	151.71^1 (54.97)	154.65^1 (59.55)	102.96^1 (35.00)
		[121–478]	[72–310]	[54–285]	[36–183]

Notes:

The stepwise group consisted of 45 children. In the standard administration group, 53 children participated, of which one was removed from subsequent analyses due to its outlying scores.

Abbreviations: M = mean, SD = standard deviation, RCF-OSS = Rey Complex Figure Organizational Scoring System (see Anderson et al., 2001).

¹ Main effect of time of measurement: better performance at posttest than at pretest across administration groups.

² Interaction between administration group and time of measurement: more improvement from pretest to posttest for stepwise administration group than for standard administration group.

There was a main effect of time for copy organization, F(1, 95) = 18.35, p < .001, ${\eta }_p^2$= .16, recall accuracy, F(1, 95) = 9.57, p = .003, ${\eta }_p^2$= .09, copy completion time, F(1, 95) = 160.04, p < .001, ${\eta }_p^2$= .628, and recall completion time, F(1, 95) = 98,28, p < .001, ${\eta }_p^2$= .508, indicating that performance on these parameters improved over time across administration groups. Finally, there was a main effect of group on recall completion time, F(1, 95) = 5.26, p = .024, ${\eta }_p^2$= .052, indicating that the stepwise instruction group recalled the figures faster than the standard administration group across pretest and posttest.

Discussion

The present study investigated whether a stepwise strategy instruction on one complex figure would lead to transferred improvement on another complex figure. Results show that transfer of performance improvement from one complex figure to another, as elicited by a stepwise strategy instruction, occurs only for recall organization. In a relatively ‘easy’ task such as copying the complex figure, children show spontaneous improvement in organization of the figure irrespective of whether they received a stepwise instruction or not (which will be discussed in more detail below). In a more demanding task, such as recall of the complex figure, spontaneous improvement in organization of information also occurs, but improvement is greater when strategy instruction is provided.

These findings are in line with previous studies into effects of strategy instruction on performance on the same complex figure, indicating that stepwise strategy instruction for the ROCF has a greater impact on recall organization than on copy organization (Frisk et al., 2005; Kirkwood et al., 2001; Waber et al., 1994). According to Waber et al. (1994), the stepwise complex figure instruction aids children to apprehend the organizational framework of the complex figure by removing distracting details, thereby improving the encoding of the major organizational elements. Findings of the present study suggest that children can transfer this strategy to a task condition where the distracting details are already minimal, namely in the recall condition, since children are unlikely to remember all the details (Akshoomoff & Stiles, 1995). During the copy condition, the task at hand (i.e. to copy the figure as accurately as possible) makes it necessary to also pay attention to elements that are not pivotal for the organization of the figure. This required focus on details seems to prevent the transfer of the stepwise strategy.

Since children’s organizational level was found to be correlated positively with complex figure accuracy score but not with completion time, it seemed more likely that the positive effect of the strategy instruction on recall organization would also positively influence recall accuracy (but not completion time). This expectation was only partly met: the positive influence of strategy instruction on recall organization neither led to a higher accuracy of recalled information or to more efficient production of the drawing. These findings can be explained by a utilization deficiency (Miller, 1994), meaning that the children were not able to effectively apply the strategy to enhance recall accuracy on a parallel figure. Future research should take into account the developmental stage of children when applying strategy instruction, since this might influence the instruction’s effectiveness. Additionally, the lack of correlation between complex figure organization and task completion time should be further examined. Emphasizing accuracy as well as speed during the task instruction might lead to differential results.

Limitations and future directions

As mentioned above, results showed that children were able to spontaneously (i.e. even without strategy instruction) adapt their organizational approach to a complex figure during copy and recall, thereby achieving a higher organizational level during posttest. Similarly, at posttest, recall accuracy scores were higher and copy and recall completion times were lower than at pretest. The findings are in line with results by Waber and colleagues (1994), indicating improvement in children’s ROCF organization with repeated exposure to the task. Our study adds to these findings by showing that this improvement even occurs when a different complex figure is used as outcome measure. It is possible that the children learned from previous complex figure task performance (i.e. either ROCF administration during pretest or posttest) and spontaneously adapted their posttest (organizational) strategy accordingly. In research as well as in clinical contexts, executive function tasks, such as copying and recalling complex figures, are often administered repeatedly. Therefore, it is important that researchers and clinicians take the potential practice effects into account. In the present study, the mean interval between pretest and posttest assessment was 28 days, which is likely to be shorter than the standard interval between repeated assessments in clinical practice. Future research could determine whether the practice effects diminish or disappear when the time period between the first and second administration is increased.

Alternatively, improvements in the standard administration group may not have been caused by practice effects but by a test effect, with the MTCF being easier to organize than the ROCF. Given the comparability of geometrical properties and main organizational elements of the two figures (Hubley & Jassal, 2006; Hubley & Tremblay, 2002), we expect this (if at all) to only minimally affect our results. Unfortunately, due to the design of the present study, it is not possible to differentiate practice effects from test effects because all participants received the same test (i.e. ROCF) at pretest and the same test (i.e. MTCF) at posttest. Future research could supplement our findings by making sure that half of the children first perform the ROCF followed by the MTCF, while the other half of the children receive the tasks in the opposite order (i.e. first the MTCF, then the ROCF). Alternatively, practice effects could be assessed by asking children to reproduce the same complex figure multiple times, either with or without strategy instruction, and assess whether task performance improves. However, since the main topic of interest in the present study was the transfer effect of strategy instruction, this was not assessed.

Lastly, by investigating the transfer of performance improvement from one complex figure to another, we were only able to examine near transfer effects, i.e. transfer to a task that is structurally similar to the task the instruction was based on. Future studies should investigate whether strategy instruction can also lead to far transfer effects, i.e. transfer to other tasks assessing children’s organizational abilities that are structurally dissimilar to the instructed tasks.

Notes on the use of the RCF-OSS

Inter-rater reliability of RCF-OSS levels was different for the copy and the recall conditions of the complex figure, with the copy condition showing only moderate inter-rater reliability. To explore this further, we post hoc examined the figures scored by the two raters again. Disagreement occurred on both the ROCF and the MTCF, and mainly originated from differences in interpretation of incorrect drawings (e.g. whether a certain line could be considered a centerline or not), definitions used for ‘piecemeal approach,’ or what should be considered as ‘connecting’ lines (see Figure 5 for examples). To enhance agreement between raters in the future, the following specifications could be added to the RCF-OSS criteria: (1) A line (horizontal or vertical) does not have to be straight to be considered a centerline; (2) Fragments have to consist of subunits of at least two figure-elements to be considered as drawn in a piecemeal manner; (3) Lines can be considered as ‘connecting’ when they are less than 3 mm apart. After creating these criteria, the first author again scored all complex figures with the RCF-OSS, while being blind to the score given on the first trial and blind to the administration group. Intra-rater reliability was substantial for the copy condition (ROCF: κ_w= .77, p < .001; MTCF: κ_w= .75, p < .001) and excellent for the recall condition (ROCF: κ_w= .90, p < .001; MTCF: κ_w= .83, p < .001). In total, the scoring of 17.5% of all figures changed when taking into account the additional criteria described above. Of the adapted figures, four were adjusted by two RCF-OSS levels, while all other changes pertained to one RCF-OSS level. Results of the main analyses did not change when scores from the first or second rating were used, adding to the robustness of our findings. In sum, even though the main criteria for the RCF-OSS are well defined, they still leave room for interpretation. By defining the criteria more strictly, as suggested above, reliability of results are likely to improve. Future studies should take the above mentioned differences between raters in interpretation of criteria into account and should consider using the additional criteria.

Figure 5. Examples of children’s drawings with commonly seen points of discussion regarding the ReyComplex Figure Organizational Strategy Score (RCF-OSS). Notes: (a) Centerline or not? Lines 14 and 28 together are here seen as the vertical centerline, but this might lead to discussion given the curving of line 28. (ROCF copy, level 5). (b) Piecemeal or not? Drawing has characteristics (e.g. left corner) of piecemeal but is rated level 5 due to presence of 2 complete centerlines. (ROCF, level 5). (c) Connecting or not? Lines 36 and 10 were not considered to be connecting (i.e. form a complete horizontal centerline together), since they were more than 3mm apart. (ROCF copy, level 4).

As a final remark: given the large overlap in important organizational units between the ROCF and the MTCF, the RCF-OSS proved to be applicable to both figures. However, the use of the RCF-OSS for the MTCF needs to be validated further. This could be achieved in a similar way as the validation of the RCF-OSS for the ROCF, by comparing the RCF-OSS levels of the MTCF to other cognitive measures (Anderson et al., 2001).

Disclosure statement

No potential conflict of interest was reported by the authors.

Appendix 1. Adapted definitions of complex figure elements for the Modified Taylor Complex Figure

Table

Square	Refers to large square and is a configural component of the figure
Centerlines	Refers to vertical and horizontal bisectors of square. Centerlines are also configural components of figure, and separately drawn portions must connect
Contour	Refers to outline of figure. This may (or may not) include total outline, such as arrows, semi-circle including two dots, triangle, or small rectangle on top of the figure
Diagonals	Refers to diagonals of square. Diagonals do not have to be completed as single whole lines and sections of each diagonal do not have to connect
Outside attachments	Refers to all sections of figure external to square, includes arrows; large triangle attached on left side of the figure; small rectangle on top of the figure; semi-circle with two dots; triangle on the right
Internal sections	Refers to all internal sections of large square which could be divided into half, quarters or eights. This includes: short horizontal and short diagonal line in upper right quadrant; triangle on the ride with two vertical lines; horizontal line in lower right quadrant; wavy line with two short lines; horizontal and vertical lines in upper left quadrant; circle in upper left quadrant; four horizontal lines within triangle on the left
Alignment	Refers to an ‘attempt’ (i.e. perfect execution not necessary) to align or connect outside attachments and internal sections with centerlines. Alignment of diagonals refers to an ‘attempt’ to connect sections of each diagonal at midpoint junction of vertical and horizontal centerlines

Note: The original definitions of the complex figure elements and instructions on how to use them to quantify organizational strategy performance were developed and first published by Anderson et al., 2001, p. 86–87. When applying the original instructions to assess performance on the Modified Taylor Complex Figure (Hubley, 1996), the word rectangle has to be substituted by the word square, and the adapted definitions have to be used for the other elements.