Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

ABSTRACT

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.

KEYWORDS:

• Longitudinal studies
• mixed methods
• program evaluation
• international development
• evaluation methods
• interview methods
• research design

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Mastercard Foundation under a grant with the University of Minnesota, with co-Principal Investigators David Chapman and Joan DeJaeghere.

Notes on contributors

R. Bamattre

R. Bamattre is a PhD candidate in Education at the University of Minnesota. His research interests include political sociology of education, alternative schools, quantitative methods, and program evaluation. His dissertation is a mixed methods study on the national policy and educational outcomes of community schools in Zambia [ORCID 0000-0003-0739-9985].

B. Schowengerdt

B. Schowengerdt is a PhD student in Education at the University of Minnesota. Her research interests include teaching/learning in higher education, academic mobilities in sub-Saharan Africa, faculty professional development, and program evaluation. Her past work involved supporting research on institutional partnerships and youth employability in Nigeria, Ghana, Kenya, and South Africa as well as teaching in Rwanda.

A. Nikoi

A. Nikoi is the Project Director for the Learn, Earn, Save Initiative Learning Partnership and a Post-Doctoral Research Associate at the University of Minnesota. Her research interests center on the role of non-formal and vocational education in youth development and empowerment. Her previous work experiences have been at the intersection of higher education and community development with a focus on youth and childhood well-being in East and West Africa.

J. DeJaeghere

J. DeJaeghere is Professor of Comparative and International Development Education in the Department of Organizational Leadership, Policy, and Development at the University of Minnesota. Her scholarship and professional practice are concerned with inequalities in education focusing on how poverty, and gender, ethnic, and caste relations affect educational participation and future civic engagement, livelihoods, and well-being. She served as the co-principal investigator (with David Chapman) of The Mastercard Foundation-sponsored Learn, Earn, Save project (2012–18), which assessed the impact of livelihood programs on the lives of disadvantaged youth in East Africa [ORCID 0000-0002-6084-8447].