Assessing Effectiveness of Mnemonics for Tertiary Students in a Hybrid Introductory Statistics Course

ABSTRACT

Mnemonics (memory aids) are often viewed as useful in helping students recall information, and thereby possibly reducing stress and freeing up more cognitive resources for higher-order thinking. However, there has been little research on statistics mnemonics, especially for large classes. This article reports on the results of a study conducted during two consecutive fall semesters at a large U.S. university. In 2014, a large sample (n = 1487) of college students were asked about the usefulness of a set of 19 published statistics mnemonics presented in class, and in 2015, the students (n = 1468) were presented 12 mnemonics related to inference and then asked whether or not they used mnemonics on that exam. This article discusses how students assess the usefulness of mnemonics and evaluates the relationship between using mnemonics and reducing anxiety. Additionally, the relationship between mnemonic usage and learning outcomes achievement will be discussed, along with this study's limitations and implications for teaching.

KEYWORDS:

• Anxiety
• Hybrid
• Mnemonic
• Memory
• Online
• Statistics education
• Study aids

1. Introduction

1.1. Potential Benefits of Mnemonics

Mnemonic, a word derived from the Greek word mnemonikos (“of memory”), is a technique used to assist memory dating back to 477 BCE (Yates 1966). A mnemonic can be classified by its form (e.g., an acronym based on initial letters of the target material) and by its function (e.g., recalling a fact versus recalling a process), and Bellezza (1981) offers further refinements in classification. We refer readers to Manalo (2002) and Bellezza (1981) for a review of mnemonics in educational settings in general.

In the field of cognitive psychology, mnemonic techniques are considered to be strategies for encoding new information in memory in such a way that they can be more easily retrieved. Among the most studied techniques are those involving imagery or verbal mnemonics (Cook 1989), such as using the first letters of a set of words to form an acronym or phrase or using the words to make up a story. The materials typically used in controlled laboratory experiments are lists of words, but more applied studies have been conducted with K–12 students using classroom materials. It has long been known that memory performance is best when information is encoded in a meaningful or organized manner (Bower et al. 1969; Craik and Lockhart 1972), for example, by connecting it to pre-existing knowledge structures. Some mnemonic techniques take advantage of the benefits of meaningful and organized encoding and supplement them by setting up an organized retrieval structure in which each retrieval cue is stored with a specific piece of information to be remembered. To be maximally effective, these cues must be memorable and have a good probability of reminding the individual of the target information.

The unprompted use of mnemonic techniques among college students has been documented in both laboratory and classroom settings. Immediately after learning a list of words for later recall, the most commonly reported mnemonic techniques were first-letter mnemonics and sentence mnemonics (Boltwood and Blick 1970). In a retrospective study of undergraduate psychology students who had recently studied for final exams, 30% of students reported using mnemonic techniques of some sort, with most being verbal mnemonics (Gruneberg 1973).

The efficacy of first-letter based mnemonics depends on the nature of the materials to be learned (Cook 1989). When sets of unrelated words were learned, first-letter mnemonics did not enhance performance relative to study with no particular instruction; however, when learning sets of related words, first-letter mnemonics did enhance performance. Sentence mnemonics helped later recall and recognition of concrete nouns (Bower and Winzenz 1970). Both ready-made and self-produced sentence mnemonics were helpful in several studies, but the evidence is mixed with regard to which is more effective (Cook 1989). Sentence mnemonics were particularly robust across delays of several days or weeks (Boltwood and Blick 1970). In a more applied study, instruction of algebraic concepts using ready-made sentence mnemonics involving rhymes improved student performance over standard instruction both in an immediate test and after a two-week delay (Machida and Carlson 1984).

Statistics mnemonics are consistent with the Guidelines for Assessment and Instruction in Statistics Education (GAISE) College Report (ASA 2005; GAISE College Report ASA Revision Committee 2016) regarding reduced emphasis on memorization because, as Lesser (2011a) noted, the mnemonics are not “intended to replace higher-order thinking or conceptual understanding, but merely to stimulate students to take into account more possibilities or to recall a low-level fact more readily in order to have more cognitive resources available to apply to something more important and conceptual” (p. 152). Lesser added that mnemonics might allow students to “keep up better in real time with class discussion by being able to apply to the conversation the mental energy or time they would have had to spend trying to search their mind (or book) to recall what a term meant” (p. 151). Mnemonics may also help students perform better and/or with less anxiety on exams. Also, as Stalder and Olson (2011) noted, mnemonics are easily added to existing material and thus require less instructor effort relative to other interventions. Stalder (2005) also noted motivational effects for his introductory psychology students: “Students generally reported that acronyms increased their motivation to begin studying, that it was helpful to solve review sheet acronyms [i.e., to review their notes to fill in what each letter stood for] on their own, and that they would try to use acronyms or other mnemonics on their own in future courses” (p. 226).

That said, the value of mnemonics should not be overstated. Just as analogies can eventually hit limitations (see Martin 2003, and its associated letters), so can mnemonics. For example, there are many limitations of the famous PEMDAS (“Please Excuse My Dear Aunt Sally”) mnemonic used to help mathematics students recall the correct order of operations or procedure precedence in a mathematical expression. For example, students may think multiplication is of greater rather than equal hierarchy than division and incorrectly evaluate the expression 8 ÷ 4 × 2. Also, Rogers (2013) noted that parentheses do not always imply grouping, and a vinculum, absolute value symbol, or square root symbol can function as a grouping symbol without using parentheses. Lee and Messner (2000) noted that operations involving additive inverses or towered exponents may not be clear. And Urbina-Lilback (2016) noted that the particular phrase “Please Excuse My Dear Aunt Sally” may be completely unfamiliar or irrelevant in the culture of learners from other countries. Another possible limitation is that some mnemonics may not be easy to recall or implement, although the latter may sometimes be addressed by the very way a mnemonic is written (e.g., the modified representation of SOHCAHTOA in Reilly 2014). The former may be addressed by having the mnemonic be so explicitly tied to the content that it cannot be forgotten, such as the acronym mnemonics for mean, median, and mode that use those very words (see Appendix A in the supplementary materials).

1.2. Prior Research

1.2.1. Identifying Statistics Mnemonics

Certain mnemonics in mathematics (e.g., FOIL, PEMDAS, and SOHCAHTOA) seem to be far more widely known than any mnemonics in statistics. It seems that until the current decade, there was no attempt to compile (much less catalogue) a comprehensive collection of mnemonics that could be used in statistics courses. However, the papers of Hunt (2010) and Lesser (2011a) collectively identify more than 30, with Lesser (2011b) offering a couple of new ones and refinements for old ones. These mnemonics serve a variety of roles, including helping students recall facts, formulas, definitions, assumptions, or processes. Lesser (2011a) gave a taxonomy for classifying mnemonics by function (e.g., helping to recall a fact vs. helping to implement a process) and by form (letter-based, image, and rhyme/jingle). A subset of these mnemonics appears in Appendix A and was used in the current study. The first two authors subsequently encountered another 11 statistics mnemonics in Table 1 of Stalder and Olson (2011), bringing the number of published distinct statistics mnemonics to around 50.

1.2.2. Assessing Effectiveness of Statistics Mnemonics

There have been only a few studies on individual statistics-related mnemonics, and these have generally had limitations. For example, Lakin et al. (2007) found that students were better at remembering and elaborating on the steps of the scientific method if their instructor discussed the mnemonic acronym HOMER more frequently—that is, as a structure for the entire course rather than just when transitioning from one step of the scientific method to the next. Those authors, however, acknowledge the limitations that students were not randomly assigned to sections of the course and there were two different instructors as well. VanVoorhis (2002) sang mnemonic statistics jingles in one section of psychology statistics and read definitions of the same terms in another section she also taught. The section that was sung to performed significantly better (t₆₉ = 2.01, p < 0.05) on test items related to the content of the definitions/jingles and had a significant correlation (r₃₁ = .37, p = 0.04) between performance and self-rated familiarity with the jingle. While the sections were shown to have statistically equivalent grade point averages (GPAs), randomization was not used to assign students to the sections, and it is possible sections varied in some important way other than GPA.

Stalder and Olson (2011) gave anonymous semester-end surveys to n = 61 undergraduates from an introductory psychology statistics course at a university in the midwestern United States. The surveys assessed degree of recall and perceived helpfulness for each of 11 listed mnemonics, finding significant positive results (i.e., ratings significantly higher than the midpoint of a 7-point scale) on both criteria for 8 of the 11 mnemonics.

Mocko (2012) discussed how and why her university took the multi-section introductory statistics course and created a separate section for students with learning disabilities, a population for which mnemonics has been identified by researchers (e.g., Manalo et al. 2000) as often helpful for mathematics students. During the Spring 2013 semester, she conducted a pilot study (Mocko et al. 2013) at a large research university in the southern United States on an (otherwise traditional) introductory statistics class specifically designated for students with learning disabilities (e.g., ADHD, reading comprehension, dyslexia, and hearing impairment) registered with the campus' disability resource center. During two class periods of the pilot study, students were given a 15-min pretest on statistical concepts (spanning descriptive statistics, graphs, and inference), then given a review sheet to study for 10 min, then given a memory cleansing activity (e.g., a unrelated word search, maze, or cryptogram), and then a 15-min posttest on the same statistical concepts. The randomized review sheets included mnemonics drawn from Hunt (2010) and Lesser (2011a). The findings were limited due to the small size of the class. The class had only 26 students, of which 21 consented to have their data in the study. Due to absences and tardiness as well as the length of the in-class activity, only 17 of the 21 students appeared to complete the questions on the activity. Additionally, of those who consented to have their data used, only 11 out of the 26 responded to the online survey and only 8 consented for their data to be used. From this pilot, we learned that some students did use mnemonics that were provided by the instructor, found in the textbook, created by themselves, or created by a classmate. Because this population was small, the research team decided to try to survey a larger, more general population of students.

1.3. Research Questions

This study aimed to explore benefits and usefulness of using mnemonics in a hybrid college course in introductory statistics. In particular, the research questions are as follows:

1.	Do students use mnemonics and find them helpful in introductory statistics?
2.	Is there an association between student self-reported usage of mnemonics and self-reported reduction in anxiety from taking tests or learning statistics?
3.	Is there a difference in the self-reported and measured learning objectives as measured by exam questions, among those with high versus low self-reported use of mnemonics?

2. Methodology

2.1. Population/Sample

The study was conducted over two consecutive Fall semesters at a large public research university in the southeastern United States where roughly 55% of the undergraduate students are White non-Hispanic, 18% Hispanic/Latino/a, 8% Black non-Hispanic, and 8% Asian-American. Also, roughly 55% of the undergraduate students are female, according to the university website. For the Fall 2014 phase of the study, the sample consisted of 1487 students, of whom 66% were female. Virtually all students in the sample were in the traditional age group for college students: 92% were between 18 and 20 years and 7% were between 21 and 23 years. For the Fall 2015 phase, the sample consisted of 1468 students, of whom 67% were female. Once again, virtually all of the students were in the traditional age group with 93% between 18 and 20 years and 6% between 21 and 23 years. Since both semesters represented only traditionally aged college students (by not exceeding age 24; see, e.g., https://nces.ed.gov/pubs/web/97578e.asp), these results are not generalizable to nontraditional age college students.

During both semesters, new lectures were recorded. Within each semester, students could attend each lecture in person or watch its recording online. For the recorded lecture, the mean number of views or partial views of videos per lecture for Fall 2014 was 1829 and for Fall 2015 was 1974. The students within each term took essentially the same quizzes (questions of similar difficulty were drawn from a quiz bank) and the same exams. These students were divided into sections of 40 students. During the Fall 2014 semester, there were 45 sections taught by 16 different teaching assistants (TAs), and during the Fall 2015 semester there were 46 sections taught by 17 TAs. The students met with a TA once a week to complete an activity. In the Fall of 2014, since the use of the mnemonics by the teaching assistants could have added much variation in the introduction of the mnemonic to the students, the TAs were not taught or encouraged to use them. In 2015, the TAs used the PHANTOM mnemonic (item #16 of Appendix A), to discuss the properties of the significance test. Because of the online nature of the course, the impact of memory usage from the TAs would be minimal.

The students attending the institution may represent higher achieving students than typical college students. The university website states that the middle 50% of entering freshman students in 2014 have SAT scores between 1810 and 2060. The mean SAT score for the Fall 2015 incoming class was 1899 at the university. For the U.S., the median SAT score for the 2014 college-bound seniors was 1490. The students in this class outperformed other college students on this standardized test.

2.2. Course

The study was conducted during a Fall 2014 and a Fall 2015 course offered in a hybrid format. The course has been taught in this format for over a decade. The students could watch the lectures online or in person, but had to attend a weekly on-campus 50-min lab meeting that entailed collecting data and working with Minitab or statistical applets. The course emphasizes concepts, but still covers calculation by hand. The students are provided a formula sheet during the exam. The students can see the formula sheet before the exam, but they are given a fresh one during the exam period. The material covered includes exploratory data analysis, regression, probability, sampling distributions, and one- and two-sample inference. During the Fall 2014 semester, randomization or bootstrap methods were not taught. During the Fall 2015 semester, these topics were briefly introduced taking about 20 min of class time each. Each topic then appeared as one question on a quiz. The course is run using the Canvas course management system. This course is generally a required course for business and social science majors.

2.3. Study Design

The study consisted of two phases, one that occurred in Fall 2014 and then another that occurred in Fall 2015. During the Fall 2014 semester, the instructor (the first author) introduced students to the 19 mnemonics listed in the first part of Appendix A. During the second week of November 2014, students were asked to complete an extra credit survey (whose purpose had been explained in the preceding lecture) about the use of mnemonics. Students were asked to complete an informed consent form as a part of the survey in Canvas. The survey mechanism allowed for anonymous responses, so the instructor would not know who consented to their data being used. All students received extra credit for completing the survey, regardless of whether they gave consent for their data to be used. Another benefit of the untimed online survey is that students were able to give more thoughtful or complete answers to certain questions. For example, in response to the question asking students for any new mnemonics they may have thought of, one student responded by giving the actual URL for a YouTube music video (made by an AP statistics class about hypothesis testing), a piece of information that would not have been available if the survey had been pencil and paper without technology access. In the class, 178 students chose not to take the survey or were already planning on dropping the course. Of those who took the survey, less than 10% of the students did not give this consent and there were only four students who gave consent who were under 18 and therefore excluded by the IRB protocol.

In the Fall 2015 semester, the students were introduced to the original 19 mnemonics, but also 8 additional mnemonics that had not been covered the previous semester. The new mnemonics were either suggested during the Fall 2014 survey or were related to content (e.g., Type I and Type II errors) covered after the completion of the Fall 2014 survey and so could not be included in that year's survey. For Fall 2015, the focus turned to the mnemonics related to inference. The students were asked to complete an extra credit quiz that included nine questions about memory use and an informed consent form. The students were asked if their responses from the survey could be used as well as their responses to select exam questions. The students received credit for completing the survey regardless of their decision to participate in the study. In addition to the survey, mnemonics were linked to 10 questions from the exams – 7 on the second exam and 3 on the last exam. The students were then asked how confident they felt about that set of exam questions. Because the exam questions would later be linked to survey questions, the survey results were not anonymous, but tracked by a university assigned student identification number. These numbers were then deleted once the linkage was made. For the Fall 2015 semester, students who were under 18 or did not consent to have their data used in the study were removed from the study as per IRB protocol. Additionally, students who did not take the survey, did not take the exams at the regularly scheduled times, dropped the course or entered their school identification number incorrectly on their Scantron form during one of the exams were also not included in the study. After all of this was taken into consideration, 1468 students' responses were studied (the starting enrollment for the course was 1911).

2.4. Mnemonics Assessed

For the Fall 2014 phase of the study, all known mnemonics that corresponded to content covered in the particular course were chosen. All chosen mnemonics corresponded to examples from Lesser (2011a), except for three (#2, #11, and #13) that were provided by the first author. The mnemonics that were not used were mnemonics for terms or concepts not discussed in the course. For the Fall 2015 phase of the study, although the first author continued to introduce mnemonics to the class from the previous study, the study concentrated only on the mnemonics that dealt with inference. The mnemonics used for inference included the mnemonics listed in Appendix A (#16, #17, #18, and #19). Additionally, eight other mnemonics were added (listed in Appendix B in the supplementary materials). Some (#20, #21, #22, #23, and #24) of these were suggested by students in the Fall 2014 semester, one (#25) came from Stalder and Olson (2011), another (#26) came from Lesser (2011a), and one (#27) came from the first author.

2.5. Construction of the Surveys

The Fall 2014 survey included 2 demographic questions about age and gender, and 17 questions about their previous/current mnemonic usage, preferences for mnemonics, use of mnemonics to help reduce anxiety, and use to help prepare for exams. Included in these questions were also questions about the students' perceived memory ability. The students were also asked to provide any mnemonics that they had thought of that could be used in an Introduction to Statistics course. The remaining 19 questions asked students about their usage and perceived helpfulness of that particular mnemonic. An earlier version of the survey had been used in a pilot study in Spring 2013 (Mocko et al. 2013). This survey was adapted and improved upon, mainly in terms of how the questions were organized. In the older version, students were asked “Which mnemonic (mnemonics) did you remember during the exam (even if you don't remember them now)? Please write down the actual mnemonics, if possible, or at least the name or topic of each mnemonic that you remembered.” This question required that the respondent write the mnemonic(s) into a short answer survey box, and it did not get any responses during the pilot. This question was revised in the new survey to improve response rate. This one question was divided into 19 questions that asked specifically about each mnemonic's usage. The other main change on the survey was that the students were also asked how many mnemonic aids they felt helped them get a correct answer on the first exam or on the second exam, rather than just lumping those two separate items together by referring to “the exam.” The survey questions can be found in Appendix C in the supplementary materials.

The Fall 2015 data collection, just like the Fall 2014 data collection, included two demographic questions about age and gender, but it also included an additional question about the students' language spoken outside of school and nine other survey questions. The same survey questions were not used because the authors wanted to focus the students' responses to questions related to how difficult the students felt that the course was and around issues of confidence. In order to assess whether the students had any previous statistical training, the students were also asked if they had been in a course that discussed the p-value. This was done because some courses in K–12 may cover statistical concepts, but the course name might not be “Introduction to Statistics” or a similar title. The survey questions for the Fall 2015 semester can be found in Appendix D in the supplementary materials.

3. Results

On the survey, the students sometimes skipped a question or two, but this was a relatively minor occurrence. For each question, there were 0–5 missing values in the responses. In the following analyses, it is of primary interest to detect any association between items in the survey. Hence, chi-squared association tests are reported for detecting statistical significance and Cramér's V effect size measures are reported to assess the practical significance. Reliabilities were computed for the scale assuming a particular item is dropped. Small variation in these item-deleted reliabilities (computed using Cronbach's α) is desirable since that indicates that no one item significantly affects the scale's internal consistency. For the Fall 2014 survey, the item-deleted reliabilities are all either 0.84 or 0.85 when considering all items in the survey asking about mnemonics and for the Fall 2015 survey, they ranged from 0.62 to 0.69.

3.1. Usage and Helpfulness of Mnemonics

In this subsection, we will address the first research question about usage and helpfulness of mnemonics by first looking at past usage of mnemonics in previous courses and then usage in the statistics course. First, students were asked to state their level of agreement with the statement that they had used a mnemonic in a previous course (Q5) or a mathematics/statistics course (Q6). The students were asked to state if they strongly agreed, somewhat agreed, agreed, neither agreed nor disagreed, disagreed, somewhat disagreed, or strongly disagreed. Out of 1486 students, 69.2% had some level of agreement (includes agree, somewhat agree, and strongly agree) with the statement for both types of previous courses, whereas only 7.5% had some level of disagreement (includes disagree, somewhat disagree, and strongly disagree) about usage in any course and mathematics/statistics courses.

Students were also asked to give their past frequency of use of mnemonics (frequently, occasionally, or not at all). The students reported that they had used mnemonics frequently (22.2%) or occasionally (69.6%) in any of their previous courses, but only 11.4% used frequently or 67.0% occasionally in other mathematics/statistics courses. In Fall 2014, students were asked to describe to what extent they agreed that mnemonics were helpful. Table 1 shows that 71.2% of the students surveyed somewhat agreed, agreed, or strongly agreed that mnemonics in the course were helpful.

Table 1. Question related to mnemonics and helpfulness.

	Strongly disagree	Disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Agree	Strongly agree
“The Memory Aids in this course were helpful.” Responses (n = 1485)	28 (1.9%)	72 (4.8%)	49 (3.3%)	278 (18.7%)	476 (32.1%)	427 (28.8%)	155 (10.4%)

3.2. Self-Reported Anxiety and Use of Mnemonics to Reduce Anxiety

The second research question was about the frequency with which students used mnemonics to reduce test anxiety as well as the anxiety of learning statistics. To address this question, the research team examined questions that related to mnemonic usage and anxiety in learning statistics (Q10) as well as mnemonic usage and anxiety in test taking (Q8). Table 2 reports these results. In summary, 43.8% of students agreed (somewhat agree, agree and strongly agree combined) that test anxiety was reduced, and 41.2% reported that general anxiety about learning statistics was reduced.

Table 2. Survey responses to questions about mnemonics and anxiety.

	Strongly disagree	Disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Agree	Strongly agree
I have found mnemonics (memory aids) helped reduce my test anxiety in this course. (n = 1487, reliability = 0.85)	79 (0.53%)	186 (12.5%)	117 (7.9%)	453 (30.5%)	350 (23.5%)	235 (15.8%)	67 (4.5%)
I find that using mnemonics (memory aids) helped reduce my anxiety learning statistics in this course. (n = 1487, reliability = 0.85)	104 (7.0%)	227 (15.3%)	145 (9.7%)	396 (26.7%)	380 (25.6%)	188 (12.7%)	45 (3.0%)

Additionally, the relationship between test anxiety and the anxiety about learning statistics is reported in Table 3. The expected cell count was less than 5 in multiple cells, so the three levels of agree responses were combined and the three levels of disagree responses were combined. The association was significant: X²(4) = 1046.2, p < 0.0001, with 1485 respondents. Cramér's V was equal to 0.594, and this indicates a very strong effect (Cohen 1988, p. 49). This significant association indicates that students who found the mnemonics more helpful in reducing test anxiety also found mnemonics more helpful in reducing anxiety about learning statistics.

Table 3. Test anxiety (row) versus anxiety learning statistics (column).

	Anxiety learning statistics
Test anxiety	Agree (that it reduced anxiety learning statistics)	Neither agree nor disagree	Disagree (that it reduced anxiety learning statistics)
Agree (that it reduced test anxiety)	507	101	43
Neither agree nor disagree	70	261	121
Disagree (that it reduced test anxiety)	36	34	312

3.3. Learning Objectives and Use of Mnemonics

To answer research question 3, first the Fall 2014 survey results were used to explore the students' self-reported use of mnemonics on the exam, and then the results of the Fall 2015 were used to explore the students' self-reported usage of mnemonics while answering specific test items. On the Fall 2014 survey, a question asked the students whether they remembered any of the mnemonics while they were taking the test. The distribution of responses to this question can be found in Table 4. In short, only 14.3% of students said that they did not use any mnemonics to help during the exam.

Table 4. Number of mnemonics that students reported remembering when they took the test.

	None	Parts of a mnemonic	1 or 2 complete mnemonics	Bits and pieces of a few mnemonics	3 or 4 complete mnemonics	More than 4 complete mnemonics
Number of mnemonics during exam, Responses (n = 1486)	212 (14.3%)	217 (14.6%)	542 (36.5%)	295 (19.9%)	173 (11.6%)	47 (3.2%)

Students were also asked whether they felt that the mnemonics helped them to get any of the questions correct on the two exams that had already occurred during the semester. The results for this can be found in Table 5. From these self-reported results, about 80% used a mnemonic on each exam.

Table 5. Number of questions students reported answering correctly due to a mnemonic.

	0 questions	1 question	2 questions	3 questions	4 or more questions
Correct for first exam (n = 1487)	281 (18.9%)	348 (23.4%)	504 (33.9%)	205 (13.8%)	149 (10.0%)
Correct for second exam (n = 1483)	335 (22.6%)	361 (24.3%)	401 (27.0%)	197 (13.3%)	189 (12.7%)

In Fall 2015, students were asked whether they had used a mnemonic while working out problems during the exam period. Appendix E in the supplementary materials lists the exam questions. Some questions (Exam 2 question #1, Exam 2 question #3, Exam 3 question #1) were adapted from the ARTIST database (https://apps3.cehd.umn.edu/artist/), while others were written by the authors. After each question that was linked to a mnemonic, the student was asked the following question:

For the above question, which statement best describes your experience?

1. I used a mnemonic and it was very helpful.

2. I used a mnemonic and it was somewhat helpful.

3. I remembered a mnemonic, but I did not use it.

4. I did not remember a mnemonic, but I would have used it had I remembered it.

5. I did not remember a mnemonic, and I would not have used it even if I had remembered it.

The student was considered to have used a mnemonic if they chose answer a or b. For each of the 10 questions (7 from Exam 2 and 3 from Exam 3), the percentage of students who used a mnemonic is given as a function of whether their answer was correct or incorrect in Appendix E.

Two of the ten exam questions stand out. Question #1 on Exam 3 asked students about the definitions of Type I and Type II error. For this question, 79.3% of the students who used a mnemonic got it correct, compared to 73.1% of students who did not use a mnemonic.

The other question was question #11 on Exam 2. This question was asking for the students to apply the p-value rule. For this question, 93.2% of the students who used a mnemonic got it correct, compared to 88.5% of students who did not use a mnemonic.

The question that had the lowest reported mnemonic usage was question #5 on Exam 3 (i.e., Ex3 Q5). This question asked about the concept of a significance test. For this question, only 66 students self-reported use of a mnemonic.

A chi-squared test was used to examine the association between mnemonic usage and correctness for each question. False discovery rate adjusted p-values (Benjamini and Hochberg 1995) are included in Table 6. These adjusted values can be compared to a 5% significance level to assess evidence for association between mnemonic usage and correctness. Note that the association between mnemonic usage and correctness was significant for all items except the first two.

Table 6. Multiplicity adjusted p-values for chi-squared test of association between mnemonic usage and correctness.

Exam	Question	% Correct with mnemonic	% Correct without mnemonic	p-value (adjusted)	Cramér's V
2	1	83.3%	77.1%	0.1500	0.07
2	3	20.6%	20.0%	0.1500	0.07
2	5	73.9%	69.0%	0.0003	0.12
2	7	63.1%	66.3%	0.0050	0.10
2	9	79.6%	68.1%	0.0000	0.16
2	11	93.2%	88.5%	0.0003	0.13
2	13	92.9%	89.0%	0.0000	0.15
3	1	79.3%	73.1%	0.0000	0.14
3	3	99.3%	99.1%	0.0116	0.10
3	5	48.5%	70.4%	0.0005	0.12

The first two questions from Exam 2 do not show a difference in mnemonic use for students who answered correctly and those who answered incorrectly. All but two of the remaining questions showed that students who answered correctly were more likely to report having used a mnemonic. The two exceptions were questions Ex2 Q7 and Ex3 Q5. For question Ex2 Q7, there was only a small (3.2%) difference. For question Ex3 Q5, there was a much larger difference of 22.2%, but, for this question, the percentage of students who used a mnemonic was quite small (only 4.5%). Cramér's V is also included in the table; this measure reflects a small to moderate level of association (Cohen 1988) for the questions showing statistical significance.

4. Discussion

4.1. Summary/Interpretations

4.1.1. Usage and Helpfulness of Mnemonics

For the sample, 91.8% of the students had used mnemonics frequently or often in any past courses, but only 78.4% had used mnemonics frequently or often in a past mathematics/statistics course. The analysis suggests that there is a practical difference in the use of mnemonics between the different types of courses. Since nearly three-quarters (71.3%) of those surveyed expressed some level of agreement that the mnemonics in the course were helpful, there seems to be compelling evidence that there is a demand or opportunity for mnemonics to be used more widely in mathematics/statistics courses.

4.1.1.1. Why might mnemonics be considered useful?

The students were asked about the amount of memorization required in the course. Out of the class, 44.5% agreed at some level that they had trouble remembering symbols or notations in mathematics/statistics courses.

When the students were asked to choose a completion for the fill-in-the-blank sentence “Statistics/math involves ____ memorization,” 531 (35.7%) chose “a large amount,” 795 (53.5%) chose “a medium amount of,” 153 (10.3%) chose “a small amount of,” and 7 (0.47%) chose “no.” So, a majority of students still felt that there was a large amount of memorization needed in the course even for a course that tried to emphasize concepts.

These responses suggest that mnemonics could be used to help recall information or procedures. For example, when the students were asked whether they found that the mnemonics provided in the course helped them with definitions of a term, steps of a procedure or aspects of a concept, 70.7% agreed (somewhat agreed, agreed, or strongly agreed combined). When the students were asked whether they agreed with the statement that they had specifically spent time studying mnemonics to help with the exam, only 34.5% agreed (somewhat agreed, agreed, or strongly agreed combined).

4.1.1.2. What qualities may lead to a mnemonic being considered helpful?

One of the questions asked the students why they remembered the mnemonic(s) that they did. Students were allowed to mark any or all of the choices that applied. The most endorsed reasons were “Mnemonic was brief” (57.8%) and “Mnemonic was well connected to the content” (43.9%). The mnemonics that had rhyme (37.1%), humor (35.6%), or a visual representation (33.4%) also received support. The reasons with the fewest endorsements were mnemonic was “strange/bizarre” (24.3%) or “the time that I spent studying or reviewing the mnemonics” (13.5%). For students who are already having trouble remembering concepts, remembering a long mnemonic may not be helpful when they might as readily memorize the full statistical concept. As for the second most common reason, if a mnemonic was well connected to the material, it was more obvious why learning that mnemonic would be beneficial—essentially, it was not yet another construct to memorize but more of a tool to remember what they needed to know. This finding was consistent with the notion that the mnemonic itself must be memorable to be useful and the notion that it must provide a strong cue to the target information.

Surprisingly, the option of “Mnemonic was visual” was further down the list—at about 33.4%. Students do not rank visual mnemonics as helpful as a word or phrase. Note, however, that in cognitive studies, imagery-based mnemonics were more effective than verbal mnemonics (Bower and Winzenz 1970). The other least common choices were “The piece of content was something that I was already familiar with,” “Mnemonic was strange/bizarre” and “The time that I spent studying or reviewing mnemonics.”

4.1.1.3. Which mnemonics were perceived as the most useful?

The students were asked to mark the answer choice that best reflected their reported use and usefulness of the mnemonic. The distribution of responses for each of the 19 mnemonics appear in Appendix A. No more than five students skipped any particular item. The mnemonic with the highest rate of reported usage and usefulness (63.7%) was the mnemonic “Population and Parameters both start with p. Sample and Statistics both start with s.” This mnemonic illustrates the attributes of brevity and being well connected to the context. This result is consistent with responses to the survey question that asked about qualities preferred in a mnemonic. The mnemonic with the second highest rate of reported usage and usefulness was “Explanatory variable on the x axis.” Once again, this memory aid has the qualities of brevity and being well connected. Also, notice that both of these mnemonics are of the same type; both leverage the “first letter of a word” to recall a fact. This finding is consistent with the finding that first-letter based mnemonics have frequent spontaneous use relative to other types of mnemonics and that they are effective for learning inter-related pieces of information (Boltwood and Blick 1970; Cook 1989).

The two mnemonics that received the lowest rankings were the graphical image of the mean and the song about the p-value. In the case of the graphical mean, it could be that to make the visual image more helpful, it needed to be better connected by the instructor. However, this result is consistent with the finding that the quality of the mnemonic being visual was not as highly endorsed as brevity and connectedness. As for the p-value song, the instructor played the song for the students over the classroom audio system. During the Fall 2014 semester, the p-value song was heard in both the recorded and live lectures. During the Fall 2015 semester, the instructor played the audio file again, but explained that some people learn more easily with music. The instructor reported that the students seemed more receptive and even clapped after hearing the p-value jingle. The p-value song has more words than the two more highly ranked mnemonic items, so it lacked the brevity that the students seemed to prefer.

It is also of interest to observe whether students created their own mnemonics. Students were encouraged to create and submit their own mnemonics to be shared with the class and asked on the survey to report any that they had used as well. For this question, 114 students (7.7% of those that took the survey) submitted a mnemonic in response to the request. Some of the mnemonics provided by the students were only slight variations of the provided mnemonics in the course. For example, the student submission “RAMP” (residual equals actual minus predicted) is very similar to the mnemonic called “ROMP” (residual equals observed minus predicted) from the list provided to the class.

The first two authors independently reviewed the student-suggested mnemonics and after discussion came to 100% consensus on a list of mnemonics that showed the most promise in helping students remember key concepts and ideas in an introductory statistics course. The most common responses were about a method to describe a histogram and for hypothesis testing based on a p-value, with eight and seven suggestions, respectively. There were also four suggestions for assumptions, confidence intervals, scatterplots or regression as well as the test statistic format. There were also six other areas (remembering symbols, distinguishing Ho and Ha, p-value, mean/median, parameter/population/sample/statistic, and sample size) that received three or fewer suggestions. There were also mnemonics suggested that were unclear (8), inappropriate for class use (5), already on the list provided (5), or not really a mnemonic (3). The mnemonics that showed the most promise and related to inference were included in the Fall 2015 semester and are in Appendix B.

4.1.2. Self-Reported Anxiety and Use of Mnemonics to Reduce Anxiety

Almost half of the class agreed that using mnemonics helped reduce test anxiety (43.8%) or anxiety over learning the material (41.2%). The cross tabulation (Table 3) suggests that there is an association between these two variables. Students who reported that they agreed that mnemonics helped in one area were more inclined to say that they helped in the other area. This is consistent with Stalder and Olson's (2011) finding that students who remembered a high number of the mnemonics had a higher reduction in anxiety ratings. This research finding supports the use of mnemonics to relieve anxiety, particularly for high-stakes assessments.

4.1.3. Learning Objectives and Use of Mnemonics

In the first exam of Fall 2014, 35% of students indicated some level of agreement that they studied mnemonics for the exam, but this figure likely underestimates the true proportion of students who studied the mnemonics. This is because students might have recalled the mnemonic while looking over their notes, but they might not have felt that this was explicitly studying the mnemonics for the exam. Also, only 14.3% of students reported that they had not used any part of a mnemonic during the exams and only about 20% (18.9% for Exam 1 and 22.6% for Exam 2) could not credit any mnemonic for helping them get to a correct answer on the exam. This shows that mnemonics in this study did provide at least a little aid to most students during the exam periods.

In Fall 2015, the students were asked at the time that they completed the question whether they used the mnemonic. The results showed that students did use mnemonics on the exam to answer questions. The question with the highest mnemonic usage was the question that utilized the Type I/Type II error mnemonic(s). There are two mnemonics associated with this concept that were presented to the students. These mnemonics are “A Type I error is a false Positive; and P has one vertical line. A Type II error is a false Negative; and N has two vertical lines” and “For a Type II error, you Fail to Reject Ho when Ho is False—a description with 2 Fs. For a Type I error, you Reject Ho when Ho is True.“

Additionally, out of the 10 questions included on the exam, 8 questions showed statistically significant association between use of a mnemonic and question correctness, although only 6 of these showed higher mnemonic usage for those who answered the question correctly. This gives further evidence that teaching students about mnemonics can help them better answer exam questions and hopefully better understand concepts presented in an introductory statistics course.

4.2. Implications for Instruction

Based on what we observed in this study, we can say that students have probably used mnemonics in their previous courses and even in their mathematics/statistics courses, to a lesser degree. Therefore, when teaching, the instructor can assume that most students have experienced mnemonics positively in another course. This may make some teachers less anxious about trying mnemonics.

Additionally, the most commonly selected response to where the students learned about the mnemonic was the teacher (79.0%). The other sources were selected as follows: themselves (54.2%), other students (36.5%), something online but not on the course management system (22.5%), textbook (14.3%), or other book (6.0%). Therefore, it does appear to be worthwhile to spend valuable class time to present and practice mnemonics. The students are less likely to get this helpful information elsewhere, even in the age of the internet.

Additionally, when considering which mnemonics to present to the class, it may be important to remember that in our study, students preferred mnemonics that were brief and directly related to the content. Moreover, these mnemonics should be repeated in class in order to help students “practice” their use. The students demonstrated their preference for these qualities when they remarked about mnemonics they found the most helpful as well as when asked to identify desirable qualities for mnemonics.

When a student reveals to an instructor extreme test anxiety, or even if the instructor is just discussing anxiety with the entire class, perhaps the instructor can discuss the use of mnemonics as an addition to the student's toolbox of possible resources and approaches. Encouraging students to study some mnemonics for the exam may give the students the edge that they need to help fight the anxiety that they experience when working on a problem. Remembering a mnemonic to help the student begin the problem may be enough to get them started toward successful completion. The instructor can also tell students that there is some evidence that using mnemonics might help them with anxiety during the exam and help them do well on the exam.

Instructors are encouraged to apply and fine-tune the findings from this study by facilitating conversation with and among their own students about memory and mnemonics in order to get a direct sense of what aspects of the content seem hard to remember and why. This local knowledge, together with the suggestion of criteria for classification of mnemonics (e.g., Lesser 2011a), can yield best practices for identifying effective mnemonics for one's student population. To help students with their studying, the instructor can offer access to a list of mnemonics, and let students collectively add to it, perhaps on a wiki page. On the list used in this study, there is sometimes (but not always) a sentence that visually or verbally connects each mnemonic to its topic, and this is a recommendation by Stalder (2005) that the authors support.

4.3. Limitations

While the sample size was quite large, the survey data was collected from the students of only one instructor at one research institution in the southeastern United States. Since the class size was so large, with around 1800 students initially enrolled in the course both semesters, there was limited opportunity for one-on-one interactions between students and teachers. Students do not often get a lot of personal attention from the instructor. Perhaps this characteristic mitigates some of the instructor effect on the results. Additionally, the list of mnemonics was posted on the course website so that even students who may not have watched or attended a lecture still had access to the mnemonics. Because the instruction about mnemonics occurred largely in the online lectures and not in the face-to-face lectures, results from the hybrid course should reasonably translate to an online course. However, the impact of mnemonics in a smaller face-to-face course may be different, due to the increased face-to-face dialogue with the instructor. Additionally, because there were very few students over the age of 24 in the study, generalizations from this study to nontraditional age students should be avoided.

It is possible that asking students about the use of mnemonics could have increased the students' overall use of mnemonics on the exam. However, because these questions did not make reference to any specific mnemonic, it seems unlikely that it provided a strong cue to use the mnemonic that would be most helpful for the particular question, especially considering that so many different mnemonics were presented to the students.

5. Directions for Future Research

In light of the aforementioned limitations, it would be useful to see how the mnemonics would be rated by students at additional institutions, including institutions that have students with different cultural and linguistic backgrounds as well as greater age diversity. Additionally, a study that compared the progress on student learning outcomes between students who were exposed to mnemonics and those who were not would add to the discussion.

It would also be helpful to gather data that goes beyond self-report and directly tracks student performance on items whose content relates to the mnemonics, including information on which mnemonic they used. This information could be helpful in determining whether students are using and remembering the mnemonic correctly. When tracking actual student performance, it would be helpful to know what characteristics of the mnemonics help students reach a further level of statistical reasoning than mnemonics with other characteristics. For students with high anxiety, can mnemonics reduce the level of anxiety to help them reach further success? In the future it would be helpful to also obtain a report of anxiety prior to instruction.

Additionally, we should consider how many and how often mnemonics need to be presented in order to be helpful as well as the appropriate distribution of these presentations. Do mnemonics need to be presented every time that the material related to that content is presented or is there an optimal timing and frequency? Additionally, how should the mnemonic terms be presented—read out loud or as part of a problem solving experience? Jacoby (1978) stated that requiring problem solving rather than just remembering a solution enhances the impact of spaced repetition of tasks on memory. In studies where only one mnemonic is being studied, there is often more flexibility to control and measure this variable. For example, Lakin et al. (2007) explored the effectiveness of the HOMER mnemonic (for the scientific method) in three sections of a course: in one section, it was used as a structure for the course and presented on 71.4% of the course meeting days, while in the other two sections it was used only on “transition days” (23.8%). On the last day of class, the students were asked to state the steps of the scientific method. The group that had seen the topic as the structure of the course performed better than those who only used it on “transition days” (p < 0.01).

Another consideration is the way in which the mnemonic is presented. Mnemonics can be presented by being read out loud by another person, read silently, displayed on a projector, handed out on a worksheet, or connected with an in-class or out of class activity. Additionally, the mnemonic could just be words or a picture. Mnemonics can be used to remember a term or a process. For more examination of taxonomy matters, see Lesser (2011a).

In future work, it would be good to explore further how best to incorporate mnemonics into a beginning statistics course. This research may include how many, how frequently, and what types of mnemonics, as well as the effect of mnemonics on understanding and anxiety. There seems to be some promising evidence that using mnemonics helps students in introductory statistics courses, but there remains much room to investigate how best to implement this resource.

Supplementary Materials

Supplemental data for this article can be accessed on the publisher's website.

Acknowledgments

The authors express appreciation to the JSE Editors and anonymous referees for their very helpful feedback.