When the type of assessment counteracts teaching for understanding

ABSTRACT

According to the model of constructive alignment, learners adjust their learning strategies to the announced assessment (backwash effect). Hence, when teaching for understanding, the assessment method should be aligned with this teaching goal to ensure that learners engage in corresponding learning strategies. A quasi-experimental field study with 81 university students was conducted to test whether “downward misalignment” – such as using and announcing a fact-oriented test when trying to teach for understanding – reduces learners’ use of sophisticated learning strategies, learning motivation, and learning outcomes in terms of understanding. We found that learners in the alignment condition applied more elaboration strategies and revealed better learning outcomes in terms of understanding. The well-aligned learning situation also led to higher perceived competence and less feeling of being under pressure. These findings confirm the backwash effect. For classroom practice, they underline the importance of carefully aligning teaching goals and assessment when teaching for understanding.

KEYWORDS:

• Constructive alignment
• assessment
• learning strategies
• learning motivation
• higher education

Teaching in schools or universities is subdivided into several phases: a planning phase, teaching and learning phase, and evaluation phase. Ideally, teachers choose while planning appropriate teaching goals that aim at understanding and activities that guide learners to meaningful learning processes. For the teaching and learning phase, teachers design their instruction so that students are assisted when engaging in suitable learning strategies that foster understanding. In the evaluation phase, teachers assess the goal achievement, usually by testing learning outcomes. Ideally, constructive alignment takes place (e.g., Biggs & Tang, 2007), meaning that the teaching goals, learning strategies triggered by the teachers’ instructions, and the learning outcomes test correspond to each other.

A teaching session is likely to succeed when learners adopt teaching goals and transform them into motivating learning goals (Schunk & Zimmerman, 2008). Learning goals guide the entire learning process: engaging in learning strategies, self-monitoring understanding, and self-evaluating the learning outcomes. In a well-aligned teaching session, students receive consistent (implicit or explicit) information from different sources, that is, the communicated teaching goals, learning environment, and announced learning outcomes test. For example, students should be consistently directed towards understanding, as this is the teacher’s goal. In the case of misalignment, that is, providing a fact-oriented learning outcomes test when trying to teach for understanding (downward misalignment), teaching goals and the announced learning outcomes test communicate different information on what and how to learn (e.g., Biggs & Tang, 2007). Such misalignment can be regarded as a widespread problem because teaching’s principal goal is to guide learners to deep understanding rather than the superficial acquisition of factual knowledge.

Why, then, do teachers frequently choose misaligned learning outcomes tests? The decision to give a particular knowledge test does usually not just depend on the teaching goals, but on factors such as class size or the resources available for correction also. In some cases, teachers have to choose an assessment required by their supervisors (i.e., standardised test). In some cases, teachers are aware of a misalignment and communicate it to the students. They may downplay the test’s importance, and emphasise that it is important to know much more than what is being tested. However, students essentially rely on the announced learning outcomes test to plan, monitor, and assess their learning processes and learning outcomes (“backwash effect”; Biggs, 2003). In an aligned learning environment, the backwash effect helps students to deduce adequate learning goals and learning strategies. However, in a misaligned learning environment, their learning goals, strategies, and outcomes may come into, at least occasionally, conflict with the teachers’ intentions.

In the present article, we created learning scenarios in which the goal of teaching for understanding is either combined with an appropriately aligned or a misaligned learning outcomes test. We analysed the effects of alignment and downward misalignment on students’ learning motivation, their use of learning strategies, and their learning outcomes.

The model of constructive alignment

According to the model of constructive alignment, teaching goals aligned with both the instructional methods and assessment method support learners in reaching the intended goals (Biggs, 1996; Biggs & Tang, 2007). Such a well-aligned learning environment encourages students to engage actively in beneficial learning processes that foster understanding (Biggs, 1996; Biggs & Tang, 2007). The relation between constructive alignment and learners’ engagement in beneficial learning strategies has been demonstrated empirically by Wang, Su, Cheung, Wong, and Kwong (2012). They took a multi-method approach and combined document analyses of course syllabi with semi-structured interviews with teachers and students to assess the constructive alignment in several university courses. Furthermore, they assessed students’ learning approaches via a questionnaire, and found that learners in well-aligned courses (focusing on understanding) reported applying more learning strategies that foster deeper understanding (e.g., elaborating on the materials to be learnt to understand them in depth). Learners in poorly aligned courses reported greater use of surface learning strategies, that is, focusing on strategies that require a minimal effort for learning while still meeting course requirements (Biggs & Tang, 2007).

To design an adequate assessment, teachers need to carefully examine the exact nature of their teaching goals. The SOLO taxonomy (Structure of the Observed Learning Outcome) is a helpful instrument with which to categorise desired learner actions and to derive suitable teaching and learning goals (Biggs, 2003; Biggs & Tang, 2007). According to the SOLO taxonomy, a learner can pass through five levels of complexity, beginning with no available understanding (prestructural), then acquiring individual pieces of correct and relevant information (unistructural) and accumulating more information, resulting in a quantitative increase in knowledge (multistructural). On the latter level, however, the individual pieces of information are still not well-connected. In the next step, learners relate the individual pieces of information to each other and detect structures and interrelationships (relational). On the final level, learners can reach the highest level of complexity, where they generalise and apply their knowledge to new areas. On that level, learners are able to reflect on critical issues and formulate their own ideas (extended abstract). Biggs and Tang (2007) assumed that a learner who attains a certain level of complexity can also deal with tasks on prior levels. However, students do not necessarily pass through all levels of complexity. Their studying can be limited by the resources available and by the principle of minimal effort.

According to the latter principle, students decide at which degree of complexity they can stop learning without sacrificing significant rewards. Thus, in a poorly aligned learning situation in which teaching goals focus on understanding (on the relational and extended abstract level), but the assessment method primarily asks for factual knowledge (uni- and multistructural level), learners are likely to reduce their efforts to attain higher levels of the taxonomy.

Assessment tasks can focus on either fact-oriented declarative knowledge (unistructural and multistructural) or functioning knowledge on a deep level of understanding (relational and extended abstract) (e.g., Biggs, 2003; Biggs & Tang, 2007). In a declarative knowledge test, students are typically asked to recall information about certain topics on different levels of granularity. Efficient tasks to assess declarative knowledge are short-answer and multiple-choice questions (e.g., “Name the three types of cognitive load.”).

These tasks assess students’ ability to remember facts. Such a test is generally very economical and reliable as well, and objective scores are easy to obtain.

In a functioning knowledge test (relational and extended abstract), it is not enough to remember facts. Students are asked to demonstrate understanding via explanations or critical reflection. A precondition for functioning knowledge is students’ organisation and elaboration of the information provided. Tasks appropriate to assessing functioning knowledge are explanation tasks (e.g., ‘Explain a principle on how to design learning materials based on the Cognitive Load Theory.’). Students also can be asked to apply their knowledge to transfer problems (e.g., ‘Referring to what you have learned about models of human memory, formulate hypotheses to explain a blackout during a written exam.’).

Jensen, McDaniel, Woodward, and Kummer (2014) examined how different question levels fostered students’ conceptual understanding in two introductory biology courses. A teaching style was used aiming at high-level learning goals in both courses. Students in both courses had to take several exams throughout the semester. While one of the groups received exams containing high-level exam questions, the other group received low-level questions.

The low-level questions were primarily recall-oriented. To answer them, students had to remember and reproduce factual knowledge. The high-level questions required that students demonstrate their ability to apply their knowledge, for example, by evaluating or analysing. The repeated assessment with a certain type of questions was expected to create a “test expectancy” for the final exam. The final exam consisted of both low- and high-level questions. The results of this study revealed that learners who took all the exams with high-level questions outperformed the other students in both question types in the final exam.

Misalignment and self-regulation in higher education

Learning in higher education is usually characterised by requiring substantial self- regulation (Wäschle, Allgeier, Lachner, Fink, & Nückles, 2014). Students usually receive little instruction or advice on how to structure and manage their study routine during the semester, or on how to prepare for exams. The process of self-regulated learning is described in several models that distinguish different phases with varying foci (e.g., Boekaerts, 1996; Zimmerman, 2002). Winne and Hadwin (1998) propose a four-stage model, beginning with (1) the task definition. The learners analyse the task requirements, the constraints that must be considered, and the quality standards that will be applied to assess the learning results. Furthermore, learners think about their prior knowledge and abilities related to the tasks. The analyses result in (2) the definition of individual learning goals and standards, which in turn determines the choice of cognitive learning strategies.

Cognitive learning strategies comprise rehearsal, organisation, and elaboration strategies (Weinstein & Mayer, 1986). Rehearsal strategies primarily support the consolidation of information in long-term memory without integrating the new information within prior knowledge. For example, rehearsal strategies are often used by medical students, who must learn enormous amounts of factual knowledge (e.g., Tooth, Tonge, & McManus, 1989). Referring back to the SOLO taxonomy: rehearsal strategies are most suitable for learning goals on the unistructural and multistructural level, as the focus lies on a quantitative increase in new information (Biggs, 2003). The aim of organisation strategies is to restructure information to facilitate the cognitive processing of complex information. A commonly used organisation strategy is concept mapping (Novak, 2010), which helps learners better understand the relations between loose pieces of information. Organisation strategies therefore structure a quantitatively huge amount of information and illustrate underlying conjunctions within a complex topic, thereby facilitating the attainment of learning goals on the relational level. To integrate new information successfully within one’s prior knowledge and understand a given topic deeply, elaboration strategies are helpful, such as connecting new information with personal experiences or gathering examples that extend beyond the given information (Weinstein & Mayer, 1986). By elaborating, learners identify underlying structures within a topic, reflect on critical issues, or apply their new knowledge to other areas. Hence, elaboration strategies also help attain high-level learning goals (relational and extended-abstract).

When (3) using a strategy, learners (4) monitor their comprehension and goal achievement to determine whether the strategies they have chosen support efficient goal achievement. This evaluation influences the learning strategies they choose in subsequent learning phases.

What influence does a misaligned learning arrangement have on the students’ self- regulated learning process? Imagine a teacher outlining the goals of understanding for a seminar (e.g., “You should be able to apply your knowledge about human memory to the design of a learning environment.”). To assess the learning outcomes, she announces a multiple-choice test. Such downward misalignment will have a substantial impact on learners’ planning processes. Based on environmental factors and information about the seminar and assessment, students define their personal learning goals (Winne & Hadwin, 1998). Both the goals the teacher presents and the assessment she announces are crucial external factors that influence how the goal is defined. They must decide how much weight they assign to each factor. However, as the students must pass certain courses to obtain their degree, information about the test is highly valued and will probably have substantial impact on the decision. Actually, concentrating on the aspects being assessed can be seen as an adaptive strategy to achieve good outcomes with moderate effort.

Of course, students are influenced by external factors and their intrinsic values. For example, they are more deeply engaged in productive learning strategies when dealing with contents that correspond to their interests (e.g., Rotgans & Schmidt, 2011). However, irrespective of learners’ motivational orientation, we assume that extrinsic factors (such as the expected type of test) influence learning behaviours. It is also important to note that German universities – where this study took place – recently introduced an assessment system of frequent exams associated with individual courses (there used to be less frequent but more comprehensive exams). This change significantly enhanced perceived stress and the degree to which the university students focused on passing their exams (Sieverding, Schmidt, Obergfell, & Scheiter, 2013). Hence, we assume that the type of expected assessment substantially influences students.

In a synthesis on motivating students to learn, Brophy (2013) asserts that teachers should be “consistent in articulating their expectations” (p. 24). In line with this recommendation, Brophy repeatedly emphasised clear goals. If teachers are inconsistent in terms of their stated goals and teaching methods on the one hand and the announced assessment on the other hand, students might feel insecure and even get confused, hampering the motivation to learn. In addition, Brophy (2013) emphasised that teaching for understanding fosters motivation because the acquired knowledge is considered more useful and applicable when needed. Misalignment in terms of fact-oriented assessments might, however, make it less clear to students whether a course is devoted to understanding the learning contents. Furthermore, if students are no longer pursuing the teachers’ goal of understanding and instead concentrate on passing a fact-oriented exam, they have the “veridical” feeling that they have primarily learned to deal with the exam instead of gaining understanding for later problem solving. This perception compromises their feelings of competence, which is detrimental to motivation (Ryan & Deci, 2000). Overall, there are several arguments why a misaligned learning situation in which the teaching goals of understanding and the assessment of facts do not fit can reduce motivation. However, there was until now no empirical evidence on the effects of a downward misalignment on motivation, which is why we examined this relationship in the present study.

The present study and hypotheses

We assume that constructive alignment is important for guiding students to learning for understanding. However, there is very restricted empirical evidence for this effect (for exceptions see Jensen et al., 2014; Wang et al., 2012). While Jensen and colleagues (Jensen et al., 2014) focused exclusively on learning outcomes, we also examined whether students report using different learning strategies depending on the expected test type. Students’ approach to learning was also considered by Wang and colleagues (Wang et al., 2012), who examined several already existing university courses to see how well teachers were implementing constructive alignment. Unlike them, we conducted an experiment by comparing a misaligned learning situation with a well-aligned one in order to clearly establish the effects of downward misalignment. As self-report questionnaires on learning strategies often lack convincing validity (e.g., Motivated Strategies for Learning Questionnaire (MSLQ), Pintrich, Smith, Garcia, & McKeachie, 1993), we took an interview approach referring to a concrete situation to gather information on which learning strategies students used to prepare for announced assessments.

The aim of the present study was to empirically test the backwash effect in a learning setting in which teaching aims at understanding and teaching goals are set on a higher level than the announced assessment procedure. We designed a quasi-experimental field study in which we compared the learning outcomes of an aligned versus a misaligned learning environment in higher education. We presented the same teaching goals to the students in both conditions: After the learning session, they should be able to explain the theoretical concepts behind different models of human memory and cognitive load theory and to apply their acquired knowledge to optimise a learning environment.

In both groups we announced an in-class assessment to be done in the following week. In the alignment condition, we announced an essay exam. Responding to a question in an essay exam requires deep understanding of the learning contents and, thus, elaboration and organisation strategies. In the downward misalignment condition, we announced a fact-oriented multiple-choice test. Answering factual knowledge questions requires students to recall information and, thus, primarily rehearsal strategies. As we expected a backwash effect, students should engage in different learning strategies. The announcement of an essay exam should increase the use of deep learning strategies (elaboration, organisation). The announcement of a fact-oriented assessment should lead to more rehearsal (“strategy hypothesis”). As elaboration strategies lead to deeper understanding of the learning contents than the use of rehearsal strategies, we expected students in the alignment condition to achieve better learning outcomes (“learning outcomes hypothesis”). Finally, we expected the aligned learning environment to promote enhanced feelings of competence and enhanced learning motivation (“motivation hypothesis”).

Method

Sample and design

Eighty-one undergraduate students from a German university participated in this quasi-experimental study (M_age = 21.90; SD = 2.60). They were all first-year students in educational science. Most were female (64 female, 17 male), which is typical for educational studies in Germany. As part of their studies, they were obliged to attend a course on learning and cognition. In the semester during which we conducted our study, this course was offered twice by the same university teacher. In the process of arranging their class schedule, students were free to choose either of the two parallel classes. Then, each class as a whole was randomly assigned to either the experimental condition or the control condition.

The learning goals, topics, teaching methods, and materials were identical in both courses. The study took place during the second session of both courses. The session covered the topics “models of human memory” and “cognitive load theory”. The teacher presented the main teaching goals (i.e., “Students are able to explain the theoretical concepts and use the acquired knowledge to optimise a learning environment.”) and a didactic lesson plan. To keep the two sessions as identical as possible, we designed a detailed didactic plan and integrated elements that were “immune” to adaptations such as informative videos.

Furthermore, the teacher informed the students about a short test to be written one week later that would cover the topics in the upcoming session. The teacher told the students in one course that a test with the focus on factual knowledge in the following week would assess whether the students had learned the key facts concerning the topic (downward misalignment group). In the other course, she announced an essay exam that would determine whether students had understood the theoretical concepts in depth (alignment group). Five weeks later, all students had to take a final learning outcomes test that covered again the topics of the first lesson. This test was also announced one week beforehand, but this time without giving the students certain information on the format of the questions.

Given that all the students in a particular class were assigned to either the experimental or control condition, our design was quasi-experimental. Dependent variables encompassed the strategies students reported using, their motivation, as well as learning outcomes.

Materials

Treatment

Before the seminar session, which was identical in both experimental conditions, the teacher announced a short test for the following week. In the alignment condition, she announced an essay exam requiring that learners display their understanding of the theoretical concepts presented in class in depth by applying them to a practical problem. In the downward misalignment condition, in contrast, she announced a test with the focus on factual knowledge. She told the students that they should be able to reproduce the topics discussed in the seminar session. In the short test, they had to fill in missing information in an illustration, and they worked on multiple-choice, right-wrong questions (i.e., statement to be judged) and short-answer questions.

Instruments

Test on final learning outcomes

We designed a test with four questions covering the topics of the first session to assess final learning outcomes. We used the SOLO taxonomy as a “theoretical” tool to construct this test. As the first level of the SOLO taxonomy (prestructural) represents no available understanding (Biggs & Tang, 2007), we constructed four questions corresponding to the following four levels. Two questions assessed the learners’ declarative knowledge on the unistructural and multistructural SOLO levels, and two questions assessed their procedural and transfer knowledge on the relational and extended abstract SOLO levels (see Appendix A). A maximum of 10 points could be achieved.

To score each question, we blinded the students’ answers and compared them to an expert’s solution. For the statistical analyses we grouped the two questions corresponding to the unistructural and multistructural SOLO levels as the assessment of declarative knowledge (maximum six points) and the two questions corresponding to the relational and extended abstract SOLO levels as the assessment of functioning knowledge (maximum four points).

Motivation questionnaire

To assess learners’ motivation we used a shortened and adapted version of the Intrinsic Motivation Inventory (IMI; Deci & Ryan, 2006) with the four subscales interest/enjoyment (e.g., “I would describe the seminar tasks as very interesting.”), perceived competence (e.g., “I’m not likely to do very well on the seminar tasks.”), effort/importance (e.g., “I’ll put a lot of effort into the seminar tasks.”) and pressure/tension (e.g., “I feel pressured when thinking about the seminar tasks.”). The learners had to answer the items on a 7-point Likert rating scale ranging from 1 (not at all true) to 7 (very true). We computed an average score for each sub-scale. We assessed the students’ motivation twice: first, after they had taken the essay exam or the multiple-choice test and second, after the final learning outcomes test.

For reliability analysis, we calculated Cronbach’s alpha to assess internal consistency of each sub-scale for both times of assessment. At the first time of assessment, directly after participants had taken the essay exam or the multiple-choice test, internal consistency for the sub-scale pressure/tension was good (α = 0.84), and still acceptable for the sub-scale interest/enjoyment (α = 0.71). With Cronbach’s alpha of 0.50, internal consistency for the sub-scale perceived competence was poor. However, in the field of research on meaningful learning, this value can still be interpreted as acceptable (see Schmitt, 1996). Nevertheless, we had to exclude the sub-scale effort/importance from all subsequent analyses due to an unacceptable internal consistency (α = 0.46).

For the second time of assessment, after the final learning outcomes test, internal consistency for the sub-scale perceived competence was good (α = 0.81), and acceptable for the sub-scale interest/enjoyment (α = 0.77), as well as for the sub-scale pressure/tension (α = 0.70). For the sub-scale effort/importance internal consistency was questionable (α = 0.68). However, this value can still be interpreted as fair in a research design comparing different groups of participants (Ponterotto & Ruckdeschel, 2007).

Students’ reported learning strategies

To gain insights into the application of learning strategies while preparing for the announced short tests, we took a qualitative approach. After they had taken the essay exam or the multiple-choice test, we asked the students to recall their preparation and describe the strategies they had used according to the following instruction: “Please describe how you prepared for the test. Which learning activities did you engage in, and what strategies did you use (e.g., mind-mapping, summarising, repeating, thinking about your own examples, etc.)”. The instruction text was presented on a piece of paper with several blank lines underneath, indicating that the question is to be answered by producing a freely written answer. The examples given to the students in brackets served as hints to clarify the kind of activities and strategies the question is asking for. We put the students’ answers into single statements and then categorised them as typifying a rehearsal, organisation, or elaboration strategy. To categorise the statements we used a coding scheme developed by Nückles, Hübner, and Renkl (2009) and added rehearsal strategies as an additional category. We coded those statements as rehearsal strategies that revealed the aim of storing pieces of information in long-term memory by repetition (e.g., “I read the summary several times.”). Organisational strategies help learners to structure and restructure information to facilitate cognitive processing. Therefore, we coded those statements as organisational strategies focusing on highlighting the main points of a topic and their interrelationships (e.g., “I generated a mind map.”). We coded as elaboration strategies those statements that revealed the integration of new information into one’s prior knowledge to understand a topic deeply, for example by generating illustrations or examples (e.g., “I made an entry in my learning journal about the seminar session and developed my own examples.”). We computed for each student a score for each category of learning strategies. This score represented the number of statements we categorised as belonging to the corresponding category (i.e., elaboration, organisational, or rehearsal strategy). Students’ answers varied in their detailedness and consisted on average of 16 words (SD = 13.33). Statements not including a strategy were not coded.

For example, one student reported the following learning activities: “I had a close look at my seminar notes, again [rehearsal] and wrote another summary on the seminar topic [organisation]. Furthermore, I recapitulated the task we’ve worked on in the seminar [rehearsal].” The words in brackets show how we coded the students’ reported activities as learning strategies. In the example, we computed one point for organisation and two points for rehearsal.

To assess interrater reliability, the answers of 20% of the sample (i.e., 16 students) were coded by two raters with good agreement, elaboration strategies κ = 0.89, p < 0.001, organistional strategies κ = 0.75, p = 0.001, rehearsal strategies κ = 0.63, p < 0.001.

Procedure

The study was conducted within a university course offered by the same teacher to two groups in the same semester. The course requirements included several tests. We designed one of the tests in each of the two groups for the purpose of our study. At the beginning of the semester, both groups had an identical first session with regard to contents and instructional methods. In that session, the teacher announced the first test for the following session, varying the test format depending on the experimental condition. The teacher informed the students that the tests were part of the study requirements, and that they need to be passed in order to fulfil the study requirements. Furthermore, she informed the students that their test results will be used in a scientific study, giving them the opportunity to object participation.

One week later, the students took the announced test in class (30 minutes). After taking this test, they filled in a questionnaire assessing their motivation in learning for the test. We also asked them to recall their activities while preparing for the test and to report the learning strategies they remembered. Five weeks later, all students had to take the final learning outcomes test on the first lesson’s topics. We assessed the students’ motivation again. Once the study had ended, the students were debriefed, and we informed them whether they had passed the tests or not (all students passed).

Results

In our results, we report partial eta-squared to specify the effect sizes and interpret η_p² ≤ 0.06 as a small effect size, η_p² ≤ 0.13 as a moderate effect size, and η_p² ≥ 0.13 as a large effect size (Cohen, 1988). Table 1 displays the means and standard deviations of all dependent variables and control variables for both experimental conditions.

Table 1. Means and standard deviations for the dependent variables and control variables separately for the experimental conditions.

	Experimental condition
	Alignment (n = 49)		Downward misalignment (n = 32)
Dependent Variables	M	SD	M	SD
Learning strategies^a
Elaboration strategies	0.40	0.64	0.07	0.26
Organisation strategies	0.54	0.65	0.43	0.74
Rehearsal strategies	1.10	0.99	1.17	1.10
Final learning outcomes test^b
Overall	4.55	2.63	5.94	2.70
Declarative knowledge	1.93	0.66	1.71	0.82
Functioning knowledge	1.23	0.70	0.80	0.73
Motivation after the announced, different tests (essay and multiple-choice test)^c
Interest	4.70	1.02	4.58	1.09
Perceived competence	4.75	1.09	4.17	1.03
Effort	5.26	1.10	5.44	1.03
Pressure	3.89	1.30	4.57	1.45
Motivation after the final learning outcomes test^d
Interest	3.67	1.32	3.61	1.20
Perceived competence	3.23	1.15	3.03	1.20
Effort	4.65	0.88	4.39	0.64
Pressure	4.36	1.43	4.48	1.32
Control Variables
Age	21.66	1.96	22.36	3.51
Number of semesters	2.43	0.93	2.43	0,84
Gender	Nfemale = 36	Nmale = 13	Nfemale = 28	Nmale = 4
Mother tongue	NGerman = 45	Nothers = 3	NGerman = 21	Nothers = 4

a Reported by the participants as freely-written text. Students’ answers were categorised in individual statements and thereafter as rehearsal, organisation, or elaboration strategy. b Assessed with a learning outcomes test with four questions. A maximum of 10 points could be achieved. c&d Rated on a 7-point Likert scale ranging from 1 (not at all true) to 7 (very true).

Pre-analyses

Before testing the hypotheses, we analysed several demographic variables in order to ensure the two experimental groups’ comparability. Students in both courses did not differ with respect to the number of female and male students, age, and number of semesters. The majority of students in both courses reported German as their mother tongue, all p > 0.10. We thus regarded them as comparable.

Furthermore, we report correlations between all dependent variables as pre-analyses (see Table 2). With regard to the reported learning strategies used during preparation: the use of elaboration strategies correlated positively with the use of organisational strategies, r = 0.26, p = 0.022. With regard to learners’ motivation, we detected a statistically significant negative correlation between their reported perceived competence and feeling of pressure, when we asked them immediately after writing the announced short test (essay exam or multiple-choice), r = -0.32, p = 0.006.

Table 2. Inter-Correlations between the dependent variables reported learning strategies, sub-scales of the Intrinsic Motivation Inventory (IMI) and results in the final learning outcomes test.

	1	2	3	4	5	6	7	8	9	10	11	12	13
1 Elaboration
2 Organisation	.26*
3 Rehearsal	.10	.04
4 Interest^a	.14	.08	.01
5 Perceived competence^a	.09	−.03	.16	.06
6 Pressure^a	−.00	.09	.06	.09	−.32**
7 Interest^b	.01	.10	−.05	.47**	−.15	−.17
8 Perceived competence^b	−.22	−.03	.04	.14	−.19	−.11	.39**
9 Effort^b	−.05	.01	−.07	−.05	−.06	.15	.02	.10
10 Pressure^b	.07	.10	−.04	−.15	−.02	.43**	−.31*	−.21	.14
11 Declarative knowledge	−.01	.15	−.05	.22	−.08	.09	.17	.34**	−.06	−.07
12 Functioning knowledge	−.09	.03	.13	.15	−.08	−.02	.16	.36**	.23	−.02	.29^c
13 Final learning outcomes test	−.02	.18	−.04	.31**	−.26*	.26*	.20	.40**	.10	−.11	.58^c	.84^c

a Assessed after students had written the essay or taken the multiple-choice test. b Assessed after the final learning outcomes test.; c no test of statistical significance as the measures are inherently interrelated; * p < 0.05; ** p < 0.01; *** p < 0.001.

With respect to the final learning outcomes and motivation, we found that perceived competence, r = 0.40, p = 0.001, assessed after the final learning outcomes test, correlated positively with the learning outcomes. With regard to the perceived competence, we observed the same results when we differentiated between declarative and functioning knowledge questions, declarative knowledge questions r = 0.34, p = 0.006, functioning knowledge questions r = 0.36, p = 0.004. These findings indicate that the more competent a learner felt, the better were the results in the final learning outcomes test.

Students’ reported learning strategies

Descriptive data show large standard deviations for the use all learning strategies. This finding hints at substantial differences between students in how they approach learning. Due to our data’s non-normal distribution, we undertook nonparametric analyses to test whether the announcement of an essay exam increased the use of deep learning strategies (elaboration, organisational), whereas the announcement of a fact-oriented assessment lead to more rehearsal. Results revealed a statistically significant difference in the report of used elaboration strategies, U(76) = 506.00, p = 0.014, with the alignment condition reporting a more frequent use than the downward misalignment condition (see Table 1). This was in line with our expectation that preparing for the essay exam would lead to a more frequent use of learning strategies fostering deeper understanding. We found no differences between the two conditions in the report of used organisational strategies, U(76) = 586.00, p = 0.287, and rehearsal strategies, U(76) = 688.50, p = 0.851. These findings partly confirm our strategy hypothesis.

Final learning outcomes

With respect to final learning outcomes, we expected students in the alignment condition to outperform those in the downward misalignment condition. To test this hypothesis, we conducted a multivariate analysis of variance with the declarative and functioning knowledge scores (final learning outcomes test) as a dependent and the experimental condition as an independent variable. Those results indicated a statistically significant overall effect: Pillai’s trace = 0.09, F(2, 72) = 3.36, p = 0.040, η_p² = 0.09, as well as a statistical significant effect of the univariate analysis of variance for functioning knowledge, F(1, 73) = 6.32, p = 0.014, η_p² = 0.08, but not for declarative knowledge, F(1, 73) = 1.75, p = 0.190, η_p² = 0.02. Learners in the alignment condition who had prepared for tasks on the two upper SOLO levels (i.e., relational and extended abstract) achieved higher test scores in the field of functioning knowledge, while both groups achieved similar test scores in the field of declarative knowledge, reflecting the levels of quantitative knowledge accumulation on the SOLO taxonomy (i.e., uni- and multistructural) (see Table 1). With differences in important prerequisites for learning between the two groups in this quasi-experiment, differences in declarative knowledge would also have been likely. As we found no such differences, this finding can be cautiously interpreted as evidence that the two experimental groups were a priori comparable. Also, the reported findings confirm (in part) our learning outcomes hypothesis.

Motivation

We conducted a multivariate analysis of variance with the sub-scales of the Intrinsic Motivation Inventory (i.e., interest/enjoyment, perceived competence, effort/importance, pressure/tension) as dependent variables and the experimental condition as independent variable. Immediately after writing the announced short tests (essay exam or multiple-choice test), the groups did not differ in their overall motivation scores, Pillai’s trace = 0.10, F(4, 69) = 1.94, p = 0.114, η_p² = -0.10. Despite these non-significant results, we continued to analyse univariate differences for explorative purposes. However, these findings need to be interpreted with caution. Univariate analyses revealed significant differences between the experimental conditions for the sub-scales perceived competence, F(1, 72) = 4.90, p = 0.030., η_p² = 0.06, and pressure/tension, F(1, 72) = 4.21, p = 0.044, η_p² = 0.06. Whereas students in the alignment condition reported higher perceived competence, those in the downward misalignment condition reported feeling more pressure or tension (see Table 1). With respect to the interest/enjoyment, we found no statistically significant differences between the experimental conditions, F(1, 72) = 0.22, p = 0.640, η_p² = 0.00.

When assessing motivation in conjunction with the final learning outcomes, we identified no significant differences between the experimental conditions, Pillai’s trace = 0.04, F(4, 58) = 0.58, p = 0.68, η_p² = 0.04. Exploratory univariate analyses revealed no significant differences (all p < 0.158). These results are in line with our motivation hypothesis. As we had not announced different assessment methods at that point, there was no reason to anticipate differences in motivation. Again, these findings can be cautiously interpreted as indicating the comparability of the two experimental conditions. Without having experimentally varied the testing situation, the groups exhibited similar results with respect to motivation.

The descriptive data showed that motivation decreased from the first assessment, after students had taken the essay exam or the multiple-choice test, to the second assessment, after the final learning outcomes test (see Table 1). This decrease was statistically significant for the sub-scales interest, perceived competence, and pressure/tension (all p < 0.001).

Discussion

We investigated whether learners adjust their learning strategies in accordance with an announced assessment as described in the backwash effect within the model of constructive alignment (Biggs & Tang, 2007). We expected that during self-regulated learning, learners would set learning goals and choose suitable learning strategies that facilitate goal achievement (e.g., Winne & Hadwin, 1998). As passing exams is a key motivator for students, the announced learning outcomes test is an important reference point for goal setting. We therefore expected the alignment to become visible in the learning strategies students reported using when preparing for the assessment, in their motivation, and in their learning outcomes.

Strategy hypothesis

First, we investigated the students’ use of learning strategies by asking them to recall their preparation for the test and report the strategies they remember using. Although rehearsal strategies were predominantly reported in both experimental conditions, the two groups differed significantly with respect to their reported use of elaboration strategies.

Students who had been told they would be taking an essay exam (SOLO level: relational and extended abstract) reported more frequent use of elaboration strategies than those told they would have to write a declarative knowledge test (SOLO level: unistructural and multistructural). Elaboration strategies support the integration of new information within one’s prior knowledge (Weinstein & Mayer, 1986). Elaboration strategies thus foster deep learning and are key to attaining high-level learning goals. Announcing an essay exam motivates students to use more elaboration strategies than a fact-oriented declarative knowledge test does. In consequence, students used learning strategies differently although the teaching goals and teaching sessions in both conditions were identical. We infer that the students thus used the information they got from the announced assessment to adjust their strategy use. Engaging in deeper understanding by using elaboration strategies was associated with students being willing to invest effort in the learning process. Therefore, we conclude that well-aligned learning environments focusing on higher level learning goals may foster learners’ motivation as well. These findings concur with those reported by Wang et al. (2012), who also observed that learners in constructively well-aligned courses reported more frequent use of learning strategies fostering deeper understanding. In courses aligned poorly, however, learners used more superficial learning strategies (Wang et al., 2012).

Learning outcomes hypothesis

Second, we investigated the students’ learning outcomes. In the present study, the teacher introduced the same teaching goals to both groups. She wanted the students to comprehend the learning contents thoroughly. However, the different tests that were announced led to different final learning outcomes. Learners in the alignment condition outperformed learners in the downward misalignment condition with regard to functioning knowledge. The groups did not differ with regard to declarative knowledge.

Motivation hypothesis

Third, we investigated the students’ learning motivation. Immediately after writing the announced short tests, multivariate analysis revealed no significant differences between the two experimental groups. Hence, the reported univariate results need to be interpreted cautiously. Nevertheless, our results hint at motivational differences in the two experimental conditions attributable to the announcement of different assessment methods. Our results suggest that students in the alignment condition felt more competent, which we ascribe to greater awareness of coherence within the learning situation. The announced essay exam corresponded to the teaching goals. The students in this condition probably felt they were heading in the right direction while preparing for the essay, which in turn led to the feeling of competence.

By contrast, learners in the downward misalignment condition experienced a lack of coherence between teaching goals and the announced test. This could have possibly led to confusion and feelings of discontent, which in turn can hamper learning motivation (Brophy, 2013). Our results indeed hint at a stronger perception of pressure in the downward misalignment condition. As the feeling of competence is known to have an impact on students’ motivation (Ryan & Deci, 2000), we infer that a high level of downward misalignment within a learning environment can lead to a decrease in motivation and consequently less effort, and lower learning outcomes.

Theoretical and practical implications

Our findings provide insights into students’ use of learning strategies and underline the important role of assessment in their choice of learning strategies. We observed indications for the positive effect of assessment methods that trigger the use of elaboration strategies on learners’ motivation and their willingness to invest effort in learning.

Furthermore, our findings reveal that the coherence within a well-aligned learning environment can enhance learners’ motivation. In the case of alignment, the assessment method is a useful orientation point for learning in a formal context such as in schools or universities. Therefore, teachers must not only be able to analyse the curriculum, deduce learning goals, and design a learning environment, they must also select an appropriate assessment method. To enable teachers to choose such an assessment, they need a broad repertoire of assessment methods and knowledge about when and how to implement them. To deal with different kinds of assessment, learners need a broad repertoire of learning strategies in order to adapt to different learning goals and assessments.

To ensure that learners are motivated and willing to invest effort in learning, learners must also regard the chosen assessment as being challenging and achievable (Locke & Latham, 2002). Assessments can be recommended that ask learners to display their ability to apply their knowledge instead of just asking them to recall facts (e.g., Bransford, Brown, & Cocking, 1999). On the other hand, a teacher should find it manageable to use them, even with larger groups of learners. Assessment methods are therefore needed that enable high level teaching goals to be assessed economically, for example ordered-outcome tests (Biggs & Tang, 2007). To assess the different levels in the SOLO taxonomy, the ordered-outcome test can comprise different types of questions (e.g., multiple-choice; apply knowledge to a given problem). However, a broad range of different assessment methods should be investigated by asking: What influence does this assessment have on students’ use of learning strategies?

Study limitations and open questions

In this study we found empirical evidence for the backwash effect. Specifically, we observed that essays exert a positive effect on learning strategies, learning outcomes, and motivation for learning. However, we mentioned earlier that results on the motivational effects of varying test formats need to be regarded with caution. They suggest motivational effects, but clear solid conclusions cannot be drawn due to the absence of a statistically significant result on the multivariate level. To ensure that differences in motivation can be definitively ascribed to the test formats, future studies should include a baseline measurement of motivation. Such a baseline measurement will also provide further information on the experimental groups’ comparability.

To demonstrate the comparability of our two experimental groups, we relied primarily on demographic data in this study, that is, information on gender, age, number of semesters, and mother tongue. We complemented this information by measuring dependent variables that further support the comparability of groups. We observed that students in both conditions did not differ in their factual knowledge in the final learning outcomes test. We would argue that pre-existing differences in key learning prerequisites are likely to trigger differences in factual knowledge. Also, when assessing motivation together with the final learning outcomes test, we detected no differences between the two conditions either. As there was no experimental variation at that point, our results on factual knowledge and motivation support the two groups’ comparability. However, the lack of baseline measurements and the students’ freedom to choose which of the two university courses they attended need to be mentioned as study limitations and are points that will be addressed in future studies.

Another issue to consider is the assessment of the learning strategies students used in preparing for the tests. In the present study, students reported their strategy use retrospectively via a free writing task. However, they might have already forgotten some of their learning strategies when preparing for the test. Moreover, they might have reported fewer learning activities than they had actually engaged in because of interpreting what a learning strategy activity is differently. Thus, we cannot guarantee that the strategies the students recalled and reported in the free writing task accurately represent the learning activities they actually engaged in. To address this issue, learners could be asked to report their learning strategies in the concrete learning situation and in greater detail. For example, standardised self-monitoring protocols could be used (Wäschle et al., 2014) in future studies. Applying such protocols would also enable an assessment of students’ strategy use repeatedly over an entire learning period. Examining students’ use of learning strategies over an entire learning period would produce a more detailed picture of their strategy use and the effects of the announced assessment on learning. Furthermore, future research could address the difference between only anticipating a certain assessment format in contrast to actually taking the test. In the present study the participants actually engaged in the announced assessment format, which of course can have a different effect on the results from the mere announcement of a certain type of assessment without taking it.

In this study we aimed to examine the backwash effect by comparing a well-aligned learning setting with a misaligned learning setting in which teaching goals were set on a higher level than the announced assessment. As already mentioned, the factors to which our results can be ascribed should be examined in more detail in future studies. In this investigation we examined “downward misalignment”, that is, having an assessment on a lower knowledge level than that of the teaching goals. However, misalignment is also possible in the opposite direction. Thus, future research should also examine how the direction of misalignment influences learning processes and outcomes. Another potential limitation is also that the alignment group had a bit more practice in working on test questions tapping understanding. Further studies on constructive alignment should control for this aspect, as well.

Jensen and colleagues (Jensen et al., 2014) found that having to write several tests in a certain assessment format during a semester led to a certain “test expectancy” regarding the final exam. In our study we also had students take the well-aligned or misaligned short test before taking the final learning outcomes test.

Conclusion

In the present study, we depict the effect of assessment on students’ learning processes, learning motivation, and learning outcomes. In the process of self-regulated learning, both teaching goals and an announced assessment method provide important orientation points for students’ learning processes. Especially in the institutional learning context, passing exams is crucial for students to earn their final degree – they therefore have a great impact on learning beyond teaching goals or personal interests. Irrelevant information on teaching goals and assessments can mislead students and hamper their learning for understanding, which in turn can lower their confidence in knowing whether their efforts have been mainly invested in handling the assessment demands or whether the knowledge they acquired is useful for later use. Hence, assessment in a formal learning context needs to be carefully designed to ensure that learners derive useful goals from it, thereby leading to meaningful learning processes. To do so, the model of constructive alignment is a useful framework for teachers. Teachers are reminded to align their intended teaching goals with their teaching activities and the assessment format. Such alignment, in turn, encourages learners to use appropriate learning strategies that further the attainment of teaching goals.

Acknowledgment

We would like to acknowledge Tino Endres for double-coding the data.

Disclosure statement

No potential conflict of interest was reported by the authors.

Appendix A

Final learning outcomes test

Test:

Models of Human Memory & Cognitive Load Theory

Name

1.Fill the gaps with the following terms: semantic, episodic

The 24-year-old Alexander is on the way to university. He wants to cross a street just as a car comes around the curve. Alexander steps back, stumbles, and falls with the back of his head on the pavement. His fall is so bad that he loses consciousness. When he wakes up again he no longer knows his name, where he comes from or where he was headed.

Alexander’s______________________memory, in which abilities like reading, walking and riding a bike are stored, has not been impaired by the accident. His general knowledge is still there. He knows that Angela Merkel is the chancellor of Germany and that a dolphin is not a fish. However, his ____________________ memory is gone. Autobiographic, personal experiences are gone, as if someone had pressed the „delete“-button in his brain.

Alexander seems to have lost all his memories of the time before his accident. Which type of amnesia is he suffering from?

□ anterograde amnesia

□ retrograde amnesia

( / 3 P.)

2. By which instructional design principles can the different types of Cognitive Load be reduced?

Connect each type of Cognitive Load to its corresponding principle by drawing a line.

intrinsic cognitive loadprovide tasks that need to be completed (e.g. fill in gaps)

extraneous cognitive loadprovide support to answer a task

germane loaddesign tasks from easy to complex

( / 3 P.)

3. In an experiment by Lewis and Anderson from 1976, participants had to learn true and false facts about a famous person. The results of this study showed that participants needed more time to identify the true facts, the more false facts they had learned.

To which theory about forgetting can these results be attributed? State briefly a reason why you chose this theory.

____________________________________________________________

____________________________________________________________

____________________________________________________________

( / 2 P.)

4.Think back to your time in school:

Imagine you are in the middle of a test. You are reading a question and just cannot think of a way to answer it although you are well prepared and spent a long time studying for the test.

Referring to what you have learned about the models of human memory, formulate hypotheses to explain a blackout during a written exam.

____________________________________________________________

____________________________________________________________

____________________________________________________________

( / 2 P.)

Total points: ( / 10 P.)