A Framework for Measuring Undergraduate Learning and Growth

In Short

• The Postsecondary Value Commission, funded by the Bill & Melinda Gates Foundation and organized by the Institute for Higher Education Policy, commissioned our research group at the University of California, Irvine to articulate a measurement framework based on an Andrew W. Mellon Foundation funded state-of-the-art undergraduate measurement project designed to track undergraduate experiences, trajectories, and outcomes.

• Our measurement framework includes six dimensions: (a) cognitive ability and intellectual dispositions, (b) development of identity and adaptive life-course agency, (c) self-regulation skills, (d) social capital, (e) civic engagement, and (f) mental health and psychological flourishing. We discuss these and how they are measured.

• We believe that the value of postsecondary education is derived from its relationship not only to a narrow set of skills related to occupational training, but also to broad aspects of human development that include cognitive, psychological, social, and civic characteristics.

Measurement of outcomes and value in undergraduate education is a normative undertaking that is influenced by the historical context. For example, the pace of economic change and occupational disruption has led to an increased emphasis on generic competencies (such as critical thinking and collaborative problem solving) over more narrow vocational skills. In recent years, the Black Lives Matter movement, growing environmental challenges, a public health crisis, and pronounced political polarization have also shaped the normative assumptions of stakeholders about what should be measured in higher education.

The recent work of the Postsecondary Value Commission, funded by the Bill & Melinda Gates Foundation and organized by the Institute for Higher Education Policy, illustrates well how social factors affect the scope of undergraduate measurement. The commission included two former U.S. secretaries of education, Margaret Spellings and Ted Mitchell, the current director of the U.S. Institute for Education Sciences, Mark Schneider, and other prominent members. “As we continue to wrestle with the dual challenges of COVID-19 and longstanding systemic racism,” the commission cochairs wrote in the forward to the report, “we must embrace the fact that postsecondary value is about both earning a decent wage and building a stronger and fairer democracy” (Postsecondary Value Commission, 2021, p. 1).

To support the goal of broadening consideration of the purposes of undergraduate education, the commission reached out to a diverse set of researchers to expand their efforts. This included outreach to our group at the University of California, Irvine (UCI), which had received funding in spring 2019 from the Andrew W. Mellon Foundation to develop a state-of-the-art undergraduate measurement system to track undergraduate experiences, trajectories, and outcomes. The Mellon Foundation was particularly interested in examining the value of liberal arts education (broadly conceived), but also recognized the need to improve higher education measurement to address that issue.

The UCI project goals were to: (a), develop and disseminate new measures of undergraduate education, (b), inform institutional improvement efforts and advance educational equity, and (c), identify value and improve social scientific understanding of educational processes. Our efforts have involved an interdisciplinary team of more than a dozen faculty and consultations with several expert panels. For our project, we have integrated diverse forms of data (including surveys, performance assessments, administrative records, learning management system data, and experiential sampling) alongside information on students’ longitudinal trajectories to define near- and long-term measures of the multifaceted benefits students derive from college attendance.

The project began tracking more than 1,200 freshmen and junior undergraduates in fall 2019, a second cohort of 1,200 freshmen and juniors in fall 2020, and a third cohort in fall 2021. As part of our work, we articulated a holistic framework for measuring undergraduate student growth and development that was commissioned as a supplementary research report by the Postsecondary Value Commission (Arum et al., 2021). The commission incorporated this into their larger report in a section that emphasized the noneconomic outcomes of postsecondary education.

We provide a summary of our measurement framework in this article. We believe that the value of postsecondary education is derived from its relationship not only to a narrow set of skills related to occupational training, but also to broad aspects of human development that include cognitive, psychological, social, and civic characteristics. Our measurement of holistic aspects of human development in 21st-century higher education systems manifests a deep commitment to equity and inclusion: we believe that college should provide equitable economic returns as well as broad benefits to cognitive growth and human development for all students, including first-generation college-going students and students from underrepresented minority backgrounds (including Black, Latinx, Indigenous, and Asian American and Pacific Islander students).

We focus our discussion on six dimensions of conceptualizing and measuring the value of college education through longitudinal observation of undergraduate student experiences, attitudes, and behaviors. The six dimensions include: (a), measuring cognitive ability and intellectual dispositions, (b), development of identity and adaptive life-course agency, (c), self-regulation skills, (d), social capital, (e), civic engagement, and (f), mental health and psychological flourishing.

Measuring Cognitive Ability and Intellectual Dispositions

In today’s complex global economy, a combination of flexible thinking and content knowledge is necessary to respond effectively to real-life situations encountered in work and civic life. In this age of excessive (mis)information, institutional grades and subsequent wages alone do not capture the full potential of student development nor can they serve as a basis to make accurate inferences regarding overall growth occurring in programs of study. In other words, good grades and high paychecks are inadequate indicators of student success.

A primary competency educators and employers alike hope graduates possess is critical thinking. While continued academic debate remains whether such a competency is transferable across domains, the increasing importance placed on a capacity to reason under uncertainty and to solve problems in complex and new circumstances is unequivocal.

To this end, several studies show that a diversity of experiences and challenging and engaging coursework are key aspects in the development of critical thinking and associated dispositions (e.g., Pascarella et al., 2013). In addition to assessing change in student critical thinking ability with the HEIghten instrument (Liu et al., 2014, 2016), we developed and fielded in partnership with Educational Testing Service a suite of performance assessments on perspective-taking, confirmation bias, and an innovative collaborative problem-solving task that students completed in groups. We also assessed change in online civic reasoning ability with an instrument developed by Sam Wineburg at Stanford (Breakstone et al., 2021).

Cognitive performance is one component of a larger and more holistic approach to measuring development that occurs in college. Other intellectual virtues and character traits that position one to apply themselves critically (such as curiosity, open-mindedness, trust in reason, and the desire to seek truth) are also relevant to undergraduate development. These dispositions belong to a cluster of traits that encourage the continued exercise of reflective cognition in academic and nonacademic areas. With the longitudinal nature of the current project, testing the relations between intellectual abilities, dispositions, and student preferences across assessment formats and designs is now viable. Ultimately, critical thinking, adaptability, and problem-solving aptitude are examined as crucial to success in a fast-paced and ever-evolving digital age.

Developing Adaptive Life-Course Agency

In individualistic societies with substantial social inequality, such as the United States, youth are encouraged to play a substantial role as active agents in shaping their own paths into and through adulthood. However, many youths from underrepresented minority or lower socioeconomic status backgrounds may not have the opportunities and resources to pursue and navigate such choices. By encountering important challenges (e.g., making a choice of major, calibrating educational aspirations, and preparing for career entry) and interacting with people from diverse cultural, socioeconomic, familial, and career backgrounds, college attendance can provide students with the opportunity to acquire and hone critical self-regulatory competencies and a broadened worldview. In turn, students become more effective adaptive agents in shaping their own life trajectories and adjusting to new realities in the social and material world. For upwardly mobile youth, higher educational institutions may provide the opportunity to develop and hone the skills needed to become more effective agents in their own lives. For students from racially or economically privileged backgrounds, higher education can increase their awareness of the world around them and help them develop the agency to navigate a diverse world and promote a more equitable and just society (Heckhausen et al., 2019).

Higher education institutions provide an ideal context in which students develop skills in adaptive agency and self-regulated motivation, including expectations and values associated with life goals, goal engagement, disengagement, and goal adjustment (Eccles & Wigfield, 2002; Heckhausen et al., 2019). These skills are essential because they help students navigate the uncertainties and discontinuities rampant in modern globalized societies.

Motivational and lifespan developmental researchers have developed measures of individual development of self-regulation skills, coping skills, control strategies, expectancies, beliefs, values, and academic and occupational identity and attitudes. Specifically, we assess students’ values following the conceptual framework of situated expectancy-value theory (EVT; Eccles & Wigfield, 2020) with regard to the following academic activities and goals: Getting a college degree (e.g., gets you a high paying job, is a life-changing experience, challenges your beliefs about society), studying a particular major (e.g., best opportunity to get into career I want, interests me most, fits my abilities best), taking a particular course (e.g., useful in everyday life, interesting, intellectually challenging, important in terms of your values and identities, useful in terms of your long-term goals), and getting a particular job (make lots of money, little chance of being laid off, be your own boss, learn new things and new skills, opportunities to be helpful to others).

Moreover, also in the context of situated EVT (S-EVT; Eccles & Wigfield, 2020), we assess students’ expectations about their performance goal(s) in their most difficult and their most important current course. Using the conceptual framework of the motivational theory of life-span development (MTLD, Heckhausen et al., 2019) and its multiscale Optimization in Primary and Secondary Control assessment instrument (Heckhausen et al., 1998), specific strategies of striving toward this goal (i.e., increase effort and time invested in this course; try harder; get help from classmates, friends, or counselors; avoid distractions; expect success) and their readiness for adjusting their goal (e.g., become more realistic in your aspirations, drop or withdraw from the course, remind yourself that other students in this course are struggling too, tell yourself it is not your fault if you are struggling in this course) are assessed.

Based on the theoretical propositions of the S-EVT (Eccles & Wigfield, 2020) and the MTLD (Heckhausen et al., 2019) and the associated assessment instruments, we have developed and collected detailed longitudinal assessments of educational and occupational goals, self-perceptions of competence and values, self-reports of engagement with goals and flexibility to adjust goals if necessary, perceptions of academic-related experiences, and perceptions of their impact on subsequent educational, occupational, and civic engagement planning and choices. These measures will be used to model academic and occupational identity and goal development as a function of educational and social experiences over the college years.

The relationship of these proximal measures with longer-term outcomes, such as later general life satisfaction, adaptive capacity, vocational success, lifetime productivity, familial success and satisfaction, civic engagement, and general mental and physical health have not been adequately studied. Lifespan developmental theorists assume that a successful passage through the college years and into early adulthood for students from 18 to 30 years of age sets the stage for a more successful passage through adulthood along many dimensions. Gathering a comprehensive set of measures of the impact of college attendance on the developmental processes associated with the development and honing of adaptive agency will put higher educational policy makers in a much stronger position to understand the full benefits of a college education to both the individual and the larger society in which these individuals live.

Self-Regulated Learning Skills

The ability to set goals, plan, organize, and monitor one’s own behavior is highly relevant for being an active and efficient agent in one’s own life. A subset of these skills, self-regulated learning skills, are particularly important for academic thriving and lifelong learning. Several well-established theoretical models describe self-regulated learning as a cyclical process with different phases and subprocesses (Boekaerts, 2011; Pintrich, 2000; Zimmerman, 2002).

Three central phases of a successful self-regulated learning process are: a phase of forethought and planning, a phase of performing a task while monitoring and controlling the progress, and an evaluation phase with reflections on performance outcomes and potential adaptation of the goals and strategies for future tasks. Regulation is needed in all three phases. Metacognitive and cognitive skills are needed to plan and monitor the learning process and to use adequate learning strategies. Also, students need to regulate their motivation and emotions that can foster or inhibit the learning process, and they need to organize their external resources (e.g., choosing their study environment and seeking help if it is needed). Postsecondary education is a highly relevant setting to develop such self-regulated learning skills by observing and learning from others who successfully use these skills and by interacting with demanding course and learning material that requires self-regulated learning.

Self-regulated learning is an immense research field in higher education. Most of the measurement instruments in this field use students’ self-reports to assess self-regulation skills, for example, the Motivated Strategies for Learning Questionnaire (Pintrich et al., 1993) and the Learning and Study Strategies Inventory (Weinstein et al., 1987). A common critique of these instruments is the gap between reported skills and the objective use and application of these real-life learning situations. We address this critique in two ways. In student surveys, items are worded as specifically as possible and ask about behaviors in specific course-related situations. An example item from the survey is: “In the past seven days, how often did you set goals to help managing your studying time for your most difficult course?”

Further, self-reported survey data are combined with data from the learning management system about students’ actual learning behavior in corresponding courses. Measures were created for each course students were taking in each quarter. These measures include the total number of logins and the total time spent on the course to quantify the total amount of study activities on a course in the campus-wide learning management system Canvas. Measures to operationalize time-sensitive self-regulation behaviors include, for example, measures of the time students submit assignments before a deadline and measures of the regularity of study activities across the academic quarter.

Social Capital

Colleges and universities can foster social capital to facilitate social mobility. The characteristics of students’ social ties and the structural features of social networks are strongly related to a range of positive outcomes, including degree attainment and career success.

At the institutional level, colleges and universities provide access to professional opportunities, affiliated organizations, and alumni communities that support social relationship formation and expand social networks. These relationships are an essential component of success in higher education (Tinto, 1993). Students gain social capital when they form ties with others who are not only willing to directly employ their own resources for the student, but also leverage their social connections and the resources of others in their extended network (Stanton-Salazar, 2011).

Students tend to gravitate toward others who are similar (McPherson et al., 2001), but social network diversity allows access to different kinds of information and previously unfamiliar groups and contexts, which may open new opportunities, encourage innovation, and foster communication skills and adaptability (Burt, 2000; Granovetter, 1973). Diversity in college environments can promote tolerance of differences and appreciation of multiculturalism that supports a cohesive society. Students who have low social capital due to their socioeconomic circumstances may especially benefit from the social scaffolding that colleges and universities provide, enabling students to deepen ties and expand networks with faculty, mentors, peers, and professionals.

The field of social network analysis has produced a powerful suite of tools capable of empirically capturing social capital and the effects of relationships in higher education (Biancani & McFarland, 2013; Kezar, 2014). The measures employed are diverse; multiple types of student networks can be captured, including friendships, roommate relationships, and academic collaborations. We employ several survey items that allow researchers to investigate the links between student success and social networks. For instance, students are asked to identify their closest friends at college and provide information about each friend. In combination with student record data, the project will be able to model the career outcomes of students who report having friends of higher socioeconomic status, as well as assess the degree to which the university supports the formation of demographically diverse social networks through classroom interactions.

Additional measures capture factors that may also be salient to their university social networks, such as the subjective value students attribute to their relationships and their perceived social mobility. More generally, the project aims to advance research that will clarify how social capital is accumulate by students and elucidate the mechanisms underlying network change and stability that are responsible for academic success and social advancement.

Civic Engagement

For generations, education has been one of the most powerful and reliable predictors of civic participation. Institutions of higher education may be one of the most important socializers of civic behavior in society, as early adulthood is a crucial period for the development of lifelong civic attitudes and practices (Flanagan & Levine, 2010).

Colleges and universities foster civic engagement through a multitude of overlapping pathways, four of which are especially relevant. First, an institution’s civic ideology (often manifested in its mission) may be transmitted to students and internalized. Second, college courses can increase students’ knowledge of social issues, as well as familiarity with the political system. Third, institutions of higher education facilitate civic engagement through direct involvement in community organizations. Last, student-led organizations and extracurricular activities can promote democratic qualities, such as leadership, agency, and cooperative organizing skills.

Given the complex relationship between higher education and civic engagement, substantial work remains in developing research strategies capable of capturing the civic value of colleges and universities. We are working to advance understanding of civic attitudes, which play mediating roles between college experiences and civic behavior. The project’s weekly surveys cover a wide range of student experiences that will allow the researchers to identify mechanisms underlying civic engagement. For instance, in collaboration with Sam Wineburg at Stanford University, the study employs new measures regarding civic attitudes and reasoning that present students with sample news and media clips, then asks them to evaluate the reliability of the information. Also, the project conducted multiple surveys during the outbreak of COVID-19. Students were asked about their perceptions of social responsibility related to social distancing, which may be related to reported trustworthiness of sources of information and subjective value of civic involvement.

Mental Health and Psychological Flourishing

Although often implicit, improved psychological well-being is an important benefit of a college education. Traditionally, psychological well-being was equated with mental health. However, the rise of positive psychology has also highlighted the importance of psychological flourishing. In that literature, psychological flourishing is associated with the Aristotelian concept of eudaimonia. This is distinct from hedonia, the pursuit of pleasure, enjoyment, comfort, and absence of distress, which is often caricatured as the sole goal of undergraduates.

To understand eudaimonia, Huta and Ryan (2010) suggested and assessed four elements: (a) seeking to pursue excellence, (b) seeking to do what you believe in, (c) seeking to use the best in yourself, and (d) seeking to develop a skill, learn, or gain insight into something. These elements align well with the more abstract goals of higher education.

Of course, neither mental health nor psychological flourishing are part of most college curricula. Instead, the ways that the college experience furthers these goals is diffuse. For example, challenging courses and the experience of coping with new living situations both create psychological stress and provide students with the opportunity, when done well, to learn to manage and overcome that stress and flourish.

Ideally, improvements in mental health and psychological flourishing should persist beyond graduation. Although research into longer-term effects is challenging, there are promising results. For example, based on two decades of data from the British National Child Development Survey, Chevalier and Feinstein (2006) concluded that “education reduces the risk of poor mental health. The impact is observed for all ages and at all points of the distribution. Education also reduces the risk of becoming depressed” (p. 3). What is less clear from studies such as these is what aspects of higher education promote these benefits.

Our project includes measures to assess how a college education leads to quantifiable increases in both mental health and flourishing during and after college and to improved functioning in society after college. The project includes repeated, traditional survey questions as well as more intensive Experiential Sampling Method techniques designed to track students’ activities and growth in the four areas of eudaimonia assessed by Huta and Ryan (2010) outlined above. In addition, the project also includes frequent assessments of perceived stress, mental health, and general life satisfaction. The various instruments involved in this study also provide information about other, possibly mediating factors (e.g., the degree of engagement, environmental mastery, social connectedness, and substance abuse).

Identifying and Measuring the Value of Postsecondary Education

The areas highlighted above and measures developed for the current project provide a path forward to identifying the value of postsecondary education for individuals and a mechanism for colleges and universities to respond to student needs and support student success. It is hoped that such efforts will provide a starting point and catalyst for a broader institutional movement to expand, iterate, and deepen undergraduate measurement focused on improving conditions for student outcomes.

We also recognize that the social and behavioral sciences are not well positioned to identify what society ought to value and produce in higher education. Once those goals and outcomes are agreed on, however, educators will have the ability to deploy measurement tools to identify mechanisms that increase the likelihood of desired outcomes.

This project is premised on an assumption that the value of postsecondary education should include the development of general and specialized cognitive competencies, intellectual dispositions, identity formation, self-direction, self-regulation, productive social relationships, civic engagement, well-being, flourishing, and the attainment of self-defined goals. We believe that all students regardless of social background have the right to aspire to postsecondary value along these dimensions and that these aspects of human development are linked to the increased likelihood of living healthy and productive lives. Improved measurement of undergraduate education can help demonstrate the extent to which students attain this value from college and can simultaneously serve to inform institutional efforts to improve postsecondary success and equity. Educators all have an obligation to do this work for all their students.

In the future, we have plans to develop our project in multiple ways and interested parties can track our progress on the project’s web site (http://education.uci.edu/next-gen-ug-success-project.html). First, we have expanded our study in fall 2021 by bringing on a new cohort of freshmen and juniors into the study as well as beginning to track the postgraduate outcomes of juniors from our original fall 2019 cohort. Second, we will be publishing and presenting findings from ongoing analyses. Third, we will be working to identify external partners that are interested in utilizing or helping to disseminate our measures to improve institutional performance in undergraduate education. Finally, we will be archiving deidentified data from our study at the University of Michigan’s Inter-University Consortium for Political and Social Research for use by other researchers to improve understanding of undergraduate education. We expect the project to develop and expand in these and other ways and welcome inquiries and suggestions about our work. 

Additional information

Notes on contributors

Richard Arum

Richard Arum is Dean of the School of Education at University of California, Irvine. He served as Senior Fellow at the Bill & Melinda Gates Foundation (2013–2015), and Director of Education Research at the Social Science Research Council (2006–2013).

Jacquelynne S. Eccles

Jacquelynne Eccles is a Professor in the University of California, Irvine School of Education and is interested in studying achievement motivation. Her work has focused mostly on educational and occupational choices. She is a mother of two and grandmother of six.

Jutta Heckhausen

Jutta Heckhausen is a Professor in the Department of Psychological Science, University of California, Irvine (PhD, 1995, University of Strathclyde, Glasgow, Great Britain). Her research addresses the role of individual agents and their motivation in life-span development.

Gabe Avakian Orona

Gabe Avakian Orona is a PhD candidate in the School of Education at the University of California, Irvine. His research focuses on measurement and assessment in higher education. Broadly, his interest is in designing and analyzing assessments consonant with the aims of higher education and using these to evaluate student learning outcomes.

Luise von Keyserlingk

Luise von Keyserlingk is a Postdoctoral Researcher and Project Manager of the Next Generation Undergraduate Success Measurement Project in the School of Education at the University of California, Irvine. Her research focuses on students’ motivation, educational choices and trajectories, and students’ mental health and well-being.

Christopher M. Wegemer

Christopher M. Wegemer is a PhD candidate in the School of Education at the University of California, Irvine. His research centers on youth civic engagement, social networks, and educational research–practice partnerships.

Charles E. (Ted) Wright

Charles E. (Ted) Wright is the Associate Dean heading the Campuswide Honors program at the University of California, Irvine. In this role, his focus is on increasing the diversity of the honors cohort and ensuring that students maximize their college experience.

Katsumi Yamaguchi-Pedroza

Katsumi Yamaguchi-Pedroza is the Lab Manager of the Next Generation Undergraduate Success Measurement Project at the University of California, Irvine. Her research interests are related to intergroup relations, identity, and stereotyping.