![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
This article presents an independent large-scale experimental evaluation of two online goal-setting interventions. Both interventions are based on promising findings from the field of social psychology. Approximately 1,400 first-year undergraduate students at a large Canadian university were randomly assigned to complete one of two online goal-setting treatments or a control task. In addition, half of treated participants were offered the opportunity to receive follow-up goal-oriented reminders through e-mail or text messages to test a cost-effective method for increasing the saliency of treatment. Across all treatment groups, we observed no evidence of an effect on grade point average, course credits, or second-year persistence. Our estimates are precise enough to discern a 7% standardized performance effect at a 5% significance level. Our results hold by subsample, for various outcome variables, and across a number of specifications.
Notes
1 Authors’ calculations using data from Ontario’s Ministry of Advanced Education and Skills Development (http://www.iaccess.gov.on.ca/OsapRatesWeb/enterapp/home.xhtml). Looking ahead to our attempt to replicate an experiment conducted at McGill University (Montreal, Quebec) with an experiment at the University of Toronto (U of T), the completion rates at McGill and U of T’s main St. George campus are 85% and 80%, respectively (http://www.macleans.ca/education/canadian-universities-with-the-highest-and-lowest-graduation-rates/). At the Mississauga campus of U of T, the site of our experiment, the six-year graduation rate is 70%.
2 A recursive process occurs when a psychological belief affects performance, which then affects the belief and performance further, continuing the cycle further in a reinforcing manner (Cohen, Garcia, Purdie-Vaughns, Apfel, & Brzustoski, Citation2009). For example, low self-confidence can cause poor performance in school, leading to even lower self-confidence and lowering performance further.
3 We outline below how our intervention differs from that in Morisano et al. (Citation2010).
4 More than 400 studies find a correlation between goal-setting and task performance (Locke & Latham, Citation1990, Citation2002, Citation2007).
5 See Morisano and Shore (Citation2010) for a detailed overview of conditions related to successful goal setting (pp. 253).
6 As one might expect, the probability of goal attainment declines as goal difficulty progressively exceeds individual ability (Bandura, Citation1977; Perrone, Civiletto, Webb, & Fitch, Citation2004; Schunk, Citation1991), as perceived obstacles present too great a challenge to attainment (Lent, Brown, & Hackett, Citation2000).
7 See Smith (Citation1998) for an overview of the benefits associated with expressive writing.
8 Summary statistics for the St. George campus reflect the authors’ calculations using a sample of students taking first-year introductory economics at St. George during the 2016–2017 academic year.
9 In addition to having a GPA below 3.0, participants in Morisano et al. (Citation2010) underwent a brief phone interview, designed for screening and assessing feelings of academic difficulty.
10 The intervention in Morisano et al. (Citation2010) was most effective among native English-speaking students who are self-nominated as academically struggling and had pre-treatment GPAs lower than 3.0.
11 We proxy for ethnic minority status with Canadian citizenship.
12 Negative effects on low-confidence students can be undone by communicating information via “self-persuasion” methods, in which participants generate some information themselves instead of passively reading material provided by the researchers.
13 In addition, treatment effects for the top quantile of students are not statistically different from zero at the 5% significance level. The bottom quantile of students had a high school average of 77% and the top quantile of students had an average of 88%.
14 We perform a conservative correction, assuming that we test 72 hypotheses (nine treatment effects for each the seven primary subgroups above and the full sample), which results in a significance level of .0007. The p value for the largest negative effect is .008, more than an order of magnitude higher than the corrected significance level. Performing the correction using all hypotheses tested would result in an even lower significance threshold.
15 Moreover, to explore whether treatment caused students to change the composition of their classes, we estimated the effect of treatment on dropping economics (a course in which all students are initially enrolled) and on average course GPA in the following semester. The effect of treatment on economics enrollment is −0.007 with a standard error of 0.021. The effect of treatment on average course GPA is −0.005 with a standard error of 0.012. Standard errors were clustered by student identification numbers. These estimates are not statistically different from zero.
16 The Self Authoring modules can be found at https://www.selfauthoring.com/.
17 While our sample does not allow us to identify a subset of students that would correspond to the self-nominated academically struggling students in Morisano et al. (Citation2010), we note that our estimates are precise enough to discern a 7% standardized effect on grades. The reported effect in the original study is 70% of a standard deviation on GPA. Assuming that this is the real effect for the students in our sample who are similar to those in Morisano et al. (Citation2010) and that the effect is zero for everyone else, these students would have to compose less than 10% of our total sample in order for us to not detect a significant effect on performance.
18 For examples of Canadian rankings, see Maclean’s Education Hub (http://www.macleans.ca/education-hub/), and for examples of international rankings, see Times Higher Education World University Rankings (https://www.timeshighereducation.com/).
19 Faculty members who teach courses in the economics department at UTM also teach at the St. George campus in both the graduate and undergraduate programs and quite often have offices on both campuses.
20 Another strand of research, however, argues that neurobiological constraints do not necessarily make goal-setting interventions unfeasible. Students who have clear goals may strengthen goal-directed behavior in the face of conflicting temptations, as such temptations make long-run goals more salient and further trigger goal-pursuant behavior (Schippers et al., Citation2015). Research also suggests that teaching adolescents to perform mental contrasting and to specify implementation intentions can be effective strategies for strengthening goal-pursuant behavior (see, for example, Duckworth, Grant, Loew, Oettingen, & Gollwitzer, Citation2010).
21 In related work (Oreopoulos & Petronijevic, Citation2018), we show that a more intensive text-messaging campaign is also unable to cause an improvement in academic outcomes, highlighting the difficulty with using virtual coaching to cause and sustain productive study habits.
22 In the current study, all experimental materials were generously provided by Jordan Peterson (Morisano et al., Citation2010; Peterson & Mar, Citation2013; Schippers et al., Citation2015) and were approved by a multidisciplinary ethics board that deemed the study to be low risk.