Promises and pitfalls of machine scoring of the Index of Productive Syntax

ABSTRACT

The AC-IPSyn computerised system for scoring the Index of Productive Syntax (IPSyn) was evaluated. Twenty language samples, ten taken at 30 months and ten of the same children at 42 months, were each scored for the IPSyn by hand and by AC-IPSyn. Point differences and point-to-point reliability were examined at the levels of the total, subscale, and individual structure scores. Points missed and erroneously given at each level were also analysed. The difference in total scores between manual and AC-IPSyn scoring was relatively small; point-to-point agreement was lower than reported elsewhere. Age differences were also found. AC-IPSyn accuracy varied by subscale and structure, with results suggesting that AC-IPSyn be used at this point in conjunction with hand scoring of more error-prone and low frequency structures. The relatively small total point difference masked the lower reliability revealed by other measures, demonstrating the importance of detailed comparisons of manual and machine scoring.

KEYWORDS:

• Assessment
• IPSyn
• computer analysis
• child language
• morphosyntax

Acknowledgments

We are grateful to Hollis S. Scarborough for her invaluable input and to Susan Ellis Weismer for making these data available through the CHILDES database. Thanks are also due to our research assistant, Rebecca Ragusa, for all her efforts. Portions of this paper were presented as a poster at the ASHA Convention, November, 2014.

Declarations of interest

This research was partially supported by a Hofstra University Faculty Research and Development Grant, 2010–2011, and a Hofstra University Presidential Research Award, 2010–2011.