Decomposing social-emotional skill rubrics: a methodological approach to examining acquiescence in rubrics’ ratings

ABSTRACT

Rubrics are popular tools to assess students’ social-emotional skills. However, little research has been devoted to investigating whether rubrics’ performance level descriptions validly represent the intended skills. In this study, we addressed the validity of Self-management rubrics by examining the presence of construct-irrelevant variance in their scores. To this end, we decomposed rubrics into single items, and those indicating low versus high levels of a construct were used to compute an acquiescence index to control for response sets. The sample consisted of 947 middle-school students (54% female; M_age = 13.8; SD = .89). This innovative methodological approach allowed us to correct rubrics’ scores for acquiescence and demonstrate its biasing effects on rubrics’ structural psychometric properties. After correction, reliabilities were preserved, and the underlying factor structure was more unidimensional. The opportunities and implications of the use of decomposed rubrics for the process of rubric development are broadly discussed.

KEYWORDS:

• Rubrics
• validity
• acquiescence
• social-emotional skills

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Statistical analyses of the underrepresentation of the construct in decomposed rubrics can be available upon request to the first author.

Additional information

Funding

The work was supported by the Instituto Ayrton Senna.

Notes on contributors

Gina Pancorbo

Gina Pancorbo is currently the Research, Policy, and Advocacy Coordinator at Education International, an organization that represents more than 32 million teachers worldwide. Her areas of expertise include early childhood education, access to knowledge, and educational equality. She earned her PhD in Psychology at Ghent University, Belgium, with a dissertation entitled ”Social-Emotional Rubrics for Formative Assessment: Psychometric Properties and Developmental Trends During Adolescence.” She holds a Master's degree in Organizational and Social Psychology from the University of Brasilia, Brazil, with specialization in psychological assessment. Prior to her current role, she worked as researcher for the Institute Ayrton Senna chair at Ghent University and as educational specialist for UNESCO in Lima and Brazil.

Ricardo Primi

Ricardo Primi is a Psychologist and an Associate Professor at Universidade São Francisco (USF) in Campinas, Brazil. He earned his Ph.D. in School Psychology and Human Development from the University of São Paulo, with part of his research conducted at Yale University. He is one of the lead research scientist for EduLab21 at the Ayrton Senna Institute (IAS) and serves on the Advisory Committee on Statistics and Psychometrics at the National Institute for Educational Studies and Research ”Anísio Teixeira” (INEP). Ricardo was a Visiting Professor in the Psychology Department at the University of Toledo, Ohio, USA, supported by a Fulbright/CAPES scholarship. In 2020, he was a visiting scholar at the Institute of Personality and Social Research at the University of California, Berkeley, with a CAPES scholarship, and also at the Lemann Center within the Graduate School of Education at Stanford University.

Oliver P. John

Oliver P. John, PhD, is Professor of Psychology and Research Psychologist at IPSR at the University of California, Berkeley. His research focuses on personality structure and development, emotion regulation, and socio-emotional skills in children and youth; his Big Five Inventory (BFI) and Emotion Regulation Questionnaire (ERQ) have each been translated into more than 30 languages. Dr. John has won numerous awards, including the Senior Career Contributions to Personality Psychology and the Theoretical Innovation Prize from the SPSP, and the Distinguished Teaching Award in the Social Sciences at UC Berkeley.

Daniel Santos

Daniel D. Santos is a Professor of Economics at the University of São Paulo- Riberão Preto with a PhD in Economics from THe University of Chicago. He is currently the coordinator of the Laboratory for Research in Education and Social Economics (LEPES), and former vice-president of the Brazilian Econometric Society (2009-2011). His research focuses on the early childhood development and the social and emotional development in educational contexts, with emphasis on the construction of new measurement tools, impact evaluation of promising interventions, and longitudinal studies that document the human developmental trajectories.

Filip De Fruyt

Filip De Fruyt is senior full professor of Differential Psychology and Personality Assessment at Ghent University in Belgium. He is specialized in assessing and building psychometric models on how individuals differ from each other and how that affects their functioning in daily life and work. He currently holds the Institute Ayrton Senna chair at Ghent University. He is the Past President of the European Association of Personality Psychology (EAPP) and (has)serve(d)s on the editorial board of a broad series of top-tier academic journals. He is a Fellow of the Society of Industrial and Organizational Psychology (SIOP).