Alignment of Course Contents and Student Assessment with Course and Programme Outcomes – A Mathematical Approach

Abstract

Successful course design and delivery relies on aligning course contents and student assessment with course and programme outcomes (CO and PO). In practice however, emphasis of CO and PO in delivered course content units (CCU) and student assessment units (SAU) may significantly deviate from the emphasis intended by course design. In this article, a mathematical approach was used to generate indicators of alignment of CCU and SAU with CO and PO. The indicators include: emphasis of CO and PO in delivered CCU (I_CCU-CO and I_CCU-PO); emphasis of CO and PO in SAU (I_SAU-CO and I_SAU-PO); alignment of SAU with CCU (I_SAU-CO/I_CCU-PO and I_SAU-PO/I_CCU-PO), and achievement of CO and PO based on student performance (I_SP-CO and I_SP-PO). Estimation of the various indicators was based on linking CCU and SAU with CO, linking CO to PO, and assigning weights, on an appropriate scale, to the different links. The mathematical approach requires data that are either readily available to instructors or that can be generated easily. The various indicators provide rational bases for aligning CCU and SAU with CO and PO, as well as for revising CO and re-aligning CO with PO. A spreadsheet can be used to automate estimation of the various indicators, compare the results with limits set by users, and generate a summary report, including tables and figures. The approach was demonstrated using a realistic course example in which the results were used as bases for re-aligning the CCU and SAU with CO and PO the second time the course was offered.

Keywords:

• alignment of course contents and outcomes
• alignment of student assessment and outcomes
• assessment of outcomes
• achievement of outcomes

Introduction

Accreditation of engineering programmes is increasingly becoming a community requirement (Patil & Codner 2007) in many countries and is compulsory in many others. Even where accreditation is not compulsory, voluntary accreditation is expected as a way of assuring stakeholders that a programme meets or exceeds the relevant accreditation criteria. A major criterion for accrediting engineering programmes in particular requires adopting and demonstrating achievement of programme outcomes (PO) common to engineering programmes. PO describe the knowledge and skills that students are expected to attain by the time they complete their programme and graduate. To achieve PO, academic programmes rely on developing students’ knowledge and skills gradually in separate or linked courses and activities (Felder & Brent 2003). Individual courses generally do not lead to achievement of PO, but serve as building blocks that contribute to achieving PO. Achievement of PO requires that the building blocks are appropriate and that their contents and assessment items are aligned constructively (Biggs 1996).

Programme leaders typically assess achievement of PO using a variety of direct and indirect strategies and support their claims using objective evidence (Briedis 2002, Mak & Frezza 2006, Rodriguez-Marek et al. 2008, Craddock & Mathias 2009, Wanousa et al. 2009, Harvey et al. 2010). One strategy can be based on assessing specific student knowledge and skills developed in courses in satisfaction of course outcomes (CO) and consequently PO. This strategy requires aligning course contents and student assessment with CO and PO. Academic programme leaders spend significant efforts on planning, designing and revising academic courses to align CO and PO (Felder & Brent 2003, Fink 2003, Wiggins & McTighe 2005). Reeves (2006) suggested that successful learning requires alignment among eight course factors: goals; content; instructional design; learner tasks; instructor roles; student roles; technological affordances; and assessment. In practice however, course contents delivery and student assessment may not be adequately aligned with CO and PO. Reeves (2006) argued that most instructors tend to use assessment strategies that are misaligned with course goals and contents. Furthermore, the actual delivery of course contents may vary from term to term, or may change over time without revising CO and PO. As a result, assessing achievement of CO and PO based on student performance without paying attention to whether delivered content and student assessment strategies are aligned with CO and PO can lead to bias and misleading results. Rogers (2003) cited the lack of consistency in delivering course contents, assessing student performance, and assigning grades as major obstacles to using student grades to indicate achievement of PO. Clearly, a more constructive assessment approach would be to assess achievement of CO and PO through assessing student performance as well as alignment of delivered course contents and student assessment with CO and PO.

Alignment of course contents and student assessment with CO and PO is typically planned and assessed qualitatively (Biggs 1996). In this article, a simple mathematical approach was used to generate outcome-based indicators of alignment of course contents and student assessment with CO and PO. Instructors typically spend enormous efforts delivering course contents and assessing student performance. In the process, instructors generate valuable data that can easily be used to assess the alignment of delivered course contents, and student assessment with CO and PO. The approach proposed in this article makes it easier for instructors to assess alignment and links student performance to CO and PO. Furthermore, the approach provides rational bases for revising and aligning course contents and assessment strategies with CO and PO, as well as a basis for revising and improving correlation between CO and PO.

Outcome-based alignment indicators

Course contents are delivered in small units referred to herein as course content units (CCU). Examples of CCU include: lectures, tutorials, projects, and presentations. CCU should be designed and delivered in a manner that reflects the intended emphasis of the CO. Furthermore, assessing student learning in courses serves a variety of formative, summative and quality assurance (QA) purposes. Student learning is typically assessed using a variety of tools and strategies, such as exams, homework, projects, and presentations. Performance assessment can be divided into a number of smaller units, referred to herein as student assessment units (SAU), that reflect the intended emphasis of the CO in a balanced manner.

A conceptual model linking CCU and SAU with CO and PO is presented in Figure 1. Successful course design requires that CCU and SAU be aligned with CO and address them in a manner consistent with the intended importance of the outcomes. Similarly, actual delivery of the CCU and SAU should be aligned with CO. In addition, CO should be emphasised in the delivered CCU and SAU in a balanced manner that reflects the importance or weights of the CO. Furthermore, CO should be linked to and aligned with PO, which links the CCU and SAU to PO. Clearly, a CO that is neither covered in delivered CCU nor addressed in SAU over a number of course delivery cycles should not be considered part of CO. As such, assessing balance in emphasising CO in delivered CCU and SAU provides useful feedback for instructors to assess alignment of CCU and SAU with CO and PO. Over time, instructors can use such indicators to realign CCU and SAU with CO and decide which CO are more worthy of emphasis and significantly contribute to achieving PO.

Figure 1 Conceptual model linking CCU and SAU to CO and PO with feedback links indicated.

The conceptual model in Figure 1 was used as a basis for developing a number of simple mathematical indicators useful for assessing the alignment of CCU and SAU with CO and PO, as well as for assessing the achievement of CO and PO based on student performance. The various indicators are listed and are described and discussed in the following sections:

1. Indicators of emphasis of CO in delivered CCU and SAU (I_CCU-CO and I_SAU-CO);

2. Indicators of emphasis of PO in delivered CCU and SAU (I_CCU-PO and I_SAU-PO);

3. Indicators of achievement of CO and PO (I_SP-CO and I_SP-PO).

Alignment of delivered CCU with CO

Although CCU are typically designed to address CO in a balanced manner consistent with the relative importance of the different CO, the actual emphasis of CO in delivered CCU often does not match the intended emphasis. Instructors may vary the teaching pace and over or under emphasise certain topics (i.e. CCU) for a variety of reasons (Felder & Silverman 1988), including the need to respond to the learning needs of students. The actual emphasis of CO in delivered CCU can easily be estimated by instructors, for example based on the number of lectures, tutorials, practical activities and other activities devoted to covering each of the CO. Instructors can also seek input from students on the level of emphasis of CO in delivered CCU and compare students’ opinions with her or his own estimates. Student views can be obtained using a survey distributed to students at the end of the semester. In the survey, students are asked to rate the emphasis of CO in delivered CCU and SAU and indicate levels of achievement of CO.

If the intended emphasis of CO in delivered CCU reflects the planned emphasis by course design, then the ratio of actual to intended emphasis of CO in delivered CCU provides a simple measure of the level of emphasis of CO in delivered CCU. The level of emphasis of CO in delivered CCU (I_CCU-CO) is a fraction that can be equal to, more than, or less than one, with the target value equal to one. If I_CCU-CO is much greater than or much less than one, then the actual level of emphasis of CO in delivered CCU significantly deviates from the intended emphasis designed in the course. Inconsistency in emphasising CO in delivered CCU, as indicated by I_CCU-CO, over a number of course delivery cycles provides a basis for either adjusting CCU, or adjusting CO, or adjusting both, CCU and CO.

The data in Tables 1 and 2 demonstrate the estimation and use of I_CCU-CO for a course taught by the author in 2012. At the time, the course had five outcomes (CO-1 to CO-5) that carried the relative weights indicated in Table 1. It should be noted that courses should not have more than a few main outcomes and the common practice is to limit the number of outcomes to four to six. The CCU of the example course in Table 1 consisted mainly of lectures and tutorials, with each CO addressed by a number of lectures and tutorials as shown in Table 1. Simple calculations, as indicated in Table 1, were used to convert the numbers of lectures and tutorials to relative weights. For example, to add the lectures and tutorials in Table 1, a weight of 0.8 was assigned to lectures and 0.2 for tutorials. The 10 lectures (22.2% of all lectures) and three tutorials (37.5% of all tutorials) in the first row in Table 1 were added as follows: (22.2% × 0.8) + (37.5% × 0.2) = 25.3%, which is equivalent to 25.3% of the total of the weights (14) assigned to the CO (i.e. 25.3% of 14 = 3.5). In Table 2, the I_CCU-CO values were estimated by dividing the weights that represent the number of lectures and tutorials addressing a particular CO on the weight that represented the intended emphasis of the CO by course design. The data in Table 2 also show that two I_CCU-CO values were estimated for each CO, one based on the instructor’s opinion and another based on average students’ opinions (39 students).

Table 1 Estimation of emphasis of CO in delivered CCU.

CO number	Intended emphasis of CO by design		Delivered emphasis of CO in CCU
		Lectures		Tutorials		Lectures + tutorials^Footnoteb
Weight^Footnotea	Percent	Number	Percent	Number	Percent	Percent	Weight^Footnotea
CO-1	4	28.6	10	22.2	3	37.5	25.3	3.5
CO-2	4	28.6	15	33.3	3	37.5	34.2	4.8
CO-3	3	21.4	10	22.2	1	12.5	20.3	2.8
CO-4	2	14.3	10	22.2	1	12.5	20.3	2.8
CO-5	1	7.1	0	0.0	0	0.0	0.0	0.0
Totals	14	100	45	100	8	100	100	14

a Scale: 1 = low emphasis; 2 = moderate emphasis; 3 = high emphasis; 4 = maximum emphasis.

b To add lectures and tutorials, the weight assigned to lectures was 0.8 and the weight assigned to tutorials was 0.2. Tutorials in this case were conducted by a teaching assistant outside the official class time. In-class tutorials however were considered and weighted as normal lectures.

Table 2 Estimation of the level of emphasis of CO in delivered CCU.

CO number	Intended emphasis of CO by design^a	Delivered emphasis of CO in CCU		ICCU-CO based on instructor’s estimate		ICCU-CO based on students’ estimate		Instructor versus students estimate^Footnotec
		Instructor Estimate^Footnotea	Students Estimate	ICCU-CO^b	Comment^Footnoteb	ICCU-CO^b	Comment^Footnoteb
CO-1	4	3.5	3.4	0.9	Aligned	0.9	Aligned	Agreement
CO-2	4	4.8	3.9	1.2	Not aligned	1.0	Aligned	Disagreement
CO-3	3	2.8	3.1	0.9	Aligned	1.0	Aligned	Agreement
CO-4	2	2.8	2.1	1.4	Not aligned	1.1	Aligned	Disagreement
CO-4	1	0.0	1.5	0.0	Not covered	1.5	Not aligned	Disagreement

a See footnote to Table 1.

b Aligned if 0.85 ≤ I_CCU-CO ≤ 1.15.

c Agreement if ΔI_CCU-CO < 0.2.

A scale for I_CCU-CO was set to judge the alignment of delivered CCU with the intended emphasis of CO. The scale was based on considering I_CCU-CO values in the range 0.85 ≤ I_CCU-CO ≤ 1.15 reflective of alignment between the actual and intended coverage of CO in delivered CCU. The alignment limits are meant to alert the instructor to consider the reasons behind misalignment, which can be fully justified. Alternative alignment limits to those proposed in this article can be used. In fact, the alignment limits should be agreed by all instructors in a programme. Given the alignment limits used in this article, the data in Table 2 show that the delivered CCU were not adequately aligned for three of the five CO. Clearly, lack of alignment over a number of course delivery cycles provides a basis for adopting appropriate corrective actions. In addition, CO-5 was not actually covered in the delivered CCU. Furthermore, the I_CCU-CO values estimated based on the instructor’s opinion disagreed with the I_CCU-CO values estimated based on students’ views. Disagreement between students and instructors may be related to wording the survey used to capture students’ views, or may reflect inadequate communication and emphasis of CO in delivering courses. Closing the loop however requires instructors to understand the reasons behind disagreement and introduce modifications to address its causes.

Alignment of delivered CCU with PO

The alignment of delivered CCU with PO can be based on linking CO with PO and assigning weights to the links. An example illustrating the correlation between the CO and PO for the course described in Tables 1 and 2 is shown in Table 3. A scale with three levels from one to three was used to indicate the weights assigned to the CO–PO links (Table 3). The three levels used in this article were based on the recommendations of the Commission for Academic Accreditation (CAA 2014) in the United Arab Emirates (UAE). Other scales with more or less levels can be used to link the CO with PO.

Table 3 Estimation of emphasis of PO in delivered CCU.

CO number	Intended emphasis of CO	Weights of CO–PO links^Footnotea					PO number	Intended emphasis of PO^Footnoteb	Delivered emphasis of PO in CCU^Footnoteb
		PO-1	PO-2	PO-3	PO-4	PO-5			Instructor’s estimate	Students’ estimate
CO-1	4	1	2	—	—	3	PO-1	2.2	2.0	2.1
CO-2	4	—	—	2	2	3	PO-2	2.0	2.1	1.8
CO-3	3	2	—	2	—	—	PO-3	4.0	4.7	4.1
CO-4	2	—	1	2	—	—	PO-4	1.6	1.4	1.9
CO-5	1	—	—	—	1	1	PO-5	4.2	3.8	4.1

Scale: 1 = low level of emphasis; 2 = medium level of emphasis; 3 = maximum level of emphasis.

Emphasis of PO-1 in CCU = (weight of CO-i – PO-1)(emphasis of CO-i in CCU)/ (weight of CO-i – PO-i link).

The totals of the assigned weights to the CO-PO links in Table 3 indicate that CO-2 contributes the most to achieving the PO, while CO-5 contributes the least. Using the weights estimated in Table 3 to the CO-PO links, the intended emphasis of CO in CCU can be translated into measures of intended emphasis of PO in CCU and delivered emphasis of PO in CCU using a simple mathematical calculation illustrated in Table 3.

If the intended emphasis of PO in CCU reflects the planned emphasis by course and programme design, then the ratio of the actual to intended emphasis of PO in CCU represents the level of emphasis of PO in delivered CCU (i.e. I_CCU-PO). Similar to I_CCU-CO, I_CCU-PO is a fraction that can be equal to, more than or less than one, with a target value equal to one.

Based on the data in Table 3, the I_CCU-PO values based on the instructor and students’ estimates for the course under consideration are presented in Table 4. In Table 4, alignment of delivered CCU with the intended emphasis of PO was judged based on the I_CCU-PO values. The I_CCU-PO values in the range 0.85 ≤ I_CCU-PO ≤ 1.15 were considered reflective of alignment between the actual and intended coverage of PO in delivered CCU. As mentioned earlier, alternative alignment limits to those suggested in this article can be used. The I_CCU-PO values in Table 4 indicate lack of alignment and disagreement between the instructor and students on two of the five PO (PO-3 and PO-4). Unjustified misalignment and disagreement over a number of course delivery cycles can be used as bases to introduce modifications to achieve adequate alignment.

Table 4 Estimation of the level of emphasis of PO in delivered CCU.

PO number	Intended emphasis of PO in CCU^Footnotea	Delivered emphasis of PO in CCU^Footnotea		ICCU-PO based on instructor’s estimate		ICCU-PO based on students’ estimate		Instructor versus students estimate^Footnotec
		Instructor estimate	Students estimate	ICCU-PO	Comment^Footnoteb	ICCU-PO	Comment^Footnoteb
PO-1	2.2	2.0	2.1	0.9	Aligned	1.0	Aligned	Agreement
PO-2	2.0	2.1	1.8	1.1	Aligned	0.9	Aligned	Agreement
PO-3	4.0	4.7	4.1	1.2	Not aligned	1.0	Aligned	Disagreement
PO-4	1.6	1.4	1.9	0.8	Not aligned	1.1	Aligned	Disagreement
PO-5	4.2	3.8	4.1	0.9	Aligned	1.0	Aligned	Agreement

a See footnote to Table 3.

b Aligned if 0.85 ≤ I_CCU-PO ≤ 1.15.

c Agreement if ΔI_CCU-PO < 0.2.

Alignment of SAU with CO and PO

Ideally, SAU should cover all CO and should reflect the emphasis of the CO and PO as intended by course and programme design. In reality however, instructors may not adequately cover the CO in delivered SAU. In this section, two indicators suitable to judge the alignment of delivered SAU with the intended emphasis of CO and PO are introduced and used to judge the alignment of the example course described in earlier sections. The two indicators are: level of emphasis of CO in delivered SAU (I_SAU-CO) and level of emphasis of PO in delivered SAU (I_SAU-PO). The two indicators are fractions that can be equal to, more than, or less than one, with a target values equal to one. Indicator values much greater than or much less than one indicate significant deviation from the intended levels of emphasis of CO and PO in delivered SAU. In this article, indicator values within the range 0.85 ≤ I_SAU-CO or I_SAU-PO ≤ 1.15 are considered reflective of adequate alignment, however alternative limits can be used.

Estimating I_SAU-CO requires linking SAU to CO and assigning appropriate weights to the links. Two assessment items (i.e. two SAU) are considered in Table 5 for the example course, which are: exams, which carried a total weight of 70% (i.e. 70% of total course grades were allocated to exams); and other SAU, which carried a total weight of 30% (i.e. 30% of total course grades were allocated to other SAU such as homework and quizzes). In terms of delivered SAU, the instructor estimated that 25% of the grades allocated for exams were devoted to assessing students’ achievement of CO-1, 40% to CO-2, 25% to CO-3, 10% to CO-4, and 0% to CO-5. Similarly, 35% of the grades assigned to the other SAU were devoted to assessing students’ achievement of CO-1, 25% to CO-2, 30% to CO-3, 10% to CO-4, and 0% to CO-5. Based on the allocated weights and grades, the emphasis of CO in delivered SAU was estimated (Table 5). The emphasis of CO in delivered SAU from students’ perspective was also estimated based on student views.

Table 5 Estimation of emphasis of CO in delivered SAU.

CO number	Intended emphasis of CO by design		Delivered emphasis of CO in SAU
			Exams	Other SAU	Exams and other SAU
Weight	Percent	Assigned grades (%)	Assigned grades (%)	Weight^Footnotea	Percent^Footnoteb
CO-1	4	28.6	25.0	35.0	3.9	28.0
CO-2	4	28.6	40.0	25.0	5.0	35.5
CO-3	3	21.4	25.0	30.0	3.7	26.5
CO-4	2	14.3	10.0	10.0	1.4	10.0
CO-5	1	7.1	0.0	0.0	0.0	0.0
Totals	14	100	100	100	14	100

a Scale: 1 = low emphasis; 2 = moderate emphasis; 3 = high emphasis; 4 = maximum emphasis.

b To add grades of exams and other SAU, the weight assigned to exams was 0.7 and the weight assigned to other SAU was 0.3. The weights were based on the grades allocated to exams (70% of the total) and other assessment items (30% of the total).

Following estimation of the emphasis of CO in delivered SAU from instructor and student perspectives, the level of emphasis of CO in delivered SAU (I_SAU-CO) was estimated (Table 6). The results in Table 6 indicate that the SAU were poorly aligned with the CO, with one of the outcomes (CO-5) not addressed at all in the SAU. Such results should alarm instructors to reconsider student assessment and introduce modifications to improve alignment of SAU with CO.

Table 6 Estimation of the level of emphasis of CO in delivered SAU.

CO number	Intended emphasis of CO by design	Delivered emphasis of CO in SAU		ISAU-CO based on instructor’s estimate		ISAU-CO based on students’ estimate		Instructor versus students estimate^Footnotec
		Instructor estimate^Footnotea	Students estimate	ISAU-CO ^b	Comment^Footnoteb	ISAU-CO ^b	Comment^Footnoteb
CO-1	4	3.9	4.1	1.0	Aligned	1.0	Aligned	Agreement
CO-2	4	5.0	3.8	1.2	Not aligned	1.0	Aligned	Disagreement
CO-3	3	3.7	4.2	1.2	Not aligned	1.4	Not aligned	Agreement
CO-4	2	1.4	1.6	0.7	Not aligned	0.8	Not aligned	Agreement
CO-4	1	0.0	0.3	0.0	Not assessed	0.3	Not aligned	Disagreement

a See footnote to Table 5.

b Aligned if 0.85 ≤ I_SAU-CO ≤ 1.15.

c Agreement if ΔI_SAU-CO < 0.2.

Additionally, the levels of emphasis of PO in delivered SAU (I_SAU-PO) were estimated from the perspectives of instructor and students, as shown in Table 7. It should be noted that the estimation of the intended and actual emphasis of PO in SAU was similar to the procedure used earlier for estimating intended and delivered emphasis of CO in SAU (Table 3). The results in Table 7 for the example course indicate that the SAU consistently aligned with the PO except for PO-1, which is mostly linked to CO-3. The results also indicate that the SAU were more aligned with the PO than with the CO, which reflects the fact that each PO is contributed by multiple CO, as shown in Table 3.

Table 7 Estimation of the level of emphasis of PO in delivered SAU.

PO number	Intended emphasis of PO^Footnotea	Delivered emphasis of PO in SAU^Footnoteb		ISAU-PO based on instructor’s estimate		ISAU-PO based on students’ estimate		Instructor versus students estimate^Footnotec
		Instructor estimate	Students estimate	ISAU-PO	Comment^Footnoteb	ISAU-PO	Comment^Footnoteb
PO-1	2.2	2.5	2.8	1.2	Not aligned	1.3	Not aligned	Agreement
PO-2	2.0	1.8	1.9	0.9	Aligned	1.0	Aligned	Agreement
PO-3	4.0	4.2	4.3	1.1	Aligned	1.1	Aligned	Agreement
PO-4	1.6	1.4	1.2	0.9	Aligned	0.8	Not aligned	Agreement
PO-5	4.2	4.1	3.8	1.0	Aligned	0.9	Aligned	Agreement

a See footnote to Table 3.

b Aligned if 0.85 ≤ I_SAU-PO ≤ 1.15.

c Agreement if ΔI_SAU-PO < 0.2.

Alignment of SAU with CCU

Having estimated the levels of emphasis of CO and PO in delivered CCU (I_CCU-CO and I_CCU-PO) and SAU (I_SAU-CO and I_SAU-PO), the alignment of CCU with SAU can be estimated by dividing I_SAU by I_CCU. The I_SAU /I_CCU values that are significantly lower or higher than one indicate that the instructor underemphasised or overemphasised the CO or PO in delivered student assessment relative to delivered course contents. For example, the data in Table 8 suggest that the SAU and the CCU were not aligned for three of the five CO and for two of the five PO. If the ratios I_SAU-CO/I_CCU-CO and I_SAU-PO/I_CCU-PO remain significantly above or below one over a number of course delivery cycles, the instructor should consider realigning the SAU with the CCU through aligning the SAU and CCU with CO.

Table 8 Alignment of delivered SAU with CCU.

Alignment of ISAU-CO and ICCU-CO					Alignment of ISAU-PO and ICCU-PO
CO number	ICCU-CO^Footnotea	ISAU-CO^a	ISAU-CO/ICCU-CO	Comment^Footnoteb	PO No.	ICCU-PO^a	ISAU-PO^a	ISAU-PO/ICCU-PO	Comment^Footnoteb
CO-1	0.9	1.0	1.1	Aligned	PO-1	0.9	1.2	1.2	Not aligned
CO-2	1.2	1.2	1.0	Aligned	PO-2	1.1	0.9	0.8	Not aligned
CO-3	0.9	1.2	1.3	Not aligned	PO-3	1.2	1.1	0.9	Aligned
CO-4	1.4	0.7	0.5	Not aligned	PO-4	0.8	0.9	1.0	Aligned
CO-5	0.0	0.0	0.0	Not addressed	PO-5	0.9	1.0	1.1	Aligned

a See footnotes to Tables 6 and 7.

b Aligned if 0.85 ≤ I_SAU-CO/I_CCU-CO or I_SAU-PO/I_CCU-PO ≤ 1.15.

Achievement of CO and PO based on student performance

Linking student performance on SAU to CO and PO provides a useful measure of the degree of achievement of CO and PO. Two approaches can be used to link student performance to CO and PO, which are: (1) a simple approach; and (2) an elaborate approach. The simple approach is based on the assumption that all CO are achieved to the same level that is indicated by the average performance of all students in a course. Such an assumption appears to be unreasonable, however it is worthy of testing as it is modified by the level of emphasis of CO in delivered SAU. However, the elaborate approach requires recording, or estimating, the performance of students on each CO. The elaborate approach may initially be tedious for instructors; however the process becomes much easier starting the second time it is applied.

The data in Tables 9–11 present a comparison between the simple and elaborate approaches and demonstrate their use in assessing the achievement of CO and PO based on student performance. In Table 9, the average grades earned by students on the different SAU were recorded following the simple and elaborate approaches and the grades were transformed into appropriate weights following the procedure described in Table 9.

Table 9 Average performance of students on each CO as indicated by earned grades.

CO number	Delivered emphasis of CO in SAU^a	Simple approach			Elaborate approach
		Exams (%)	Other SAU (%)	Overall		Exams (%)	Other SAU (%)	Overall
				Percent	Weight^Footnoteb	Percent	Weight^Footnoteb
CO-1	3.9	83.0	91.0	85.4	3.3	72.0	91.4	77.8	3.1
CO-2	5.0	83.0	91.0	85.4	4.2	97.5	96.0	97.1	4.8
CO-3	3.7	83.0	91.0	85.4	3.2	76.0	86.7	79.2	2.9
CO-4	1.4	83.0	91.0	85.4	1.2	70.0	90.0	76.0	1.1
CO-5	0.0		Not assessed		Not assessed

a Delivered emphasis of CO in SAU (from Table 5).

b Weight = overall (%).

Table 10 Relative achievement of CO based on student performance.

CO number	Intended emphasis of CO	Simple approach			Elaborate approach
		Weight^Footnotea	ISP-CO ^Footnoteb	Comment^Footnotec	Weight^Footnotea	ISP-CO ^b	Comment^Footnotec
CO-1	4.0	3.3	0.8	Achieved	3.1	0.8	Adequately achieved
CO-2	4.0	4.2	1.1	Achieved	4.8	1.2	Overemphasised
CO-3	3.0	3.2	1.1	Achieved	2.9	1.0	Achieved
CO-4	2.0	1.2	0.6	Not achieved	1.1	0.5	Not achieved
CO-5	1.0	—	—	Not addressed	—	—	Not addressed

a From Table 9.

b Estimated as ratio of weight/intended emphasis of CO.

c Not achieved if I_SP-CO < 0.6; adequately achieved if I_SP-CO < 0.8; achieved if I_SP-CO < 1.15; overemphasised if I_SP-CO >1.15.

Table 11 Relative achievement of PO based on student performance.

CO number	Weight of CO-PO link^Footnotea	Simple approach			Elaborate approach
		Weight^Footnoteb	ISP-PO ^c	Comment^Footnoted	Weight^Footnoteb	ISP-PO ^c	Comment^Footnoted
CO-1	2.2	2.1	1.0	Achieved	2.0	0.9	Achieved
CO-2	2.0	1.5	0.8	Adequately achieved	1.4	0.7	Adequately achieved
CO-3	4.0	3.6	0.9	Achieved	3.6	0.9	Achieved
CO-4	1.6	1.2	0.7	Adequately achieved	1.4	0.8	Achieved
CO-5	4.2	3.5	0.8	Achieved	3.6	0.9	Achieved

a From Table 3.

b Estimated following the procedure outlined in Table 3 using the weights estimated in Table 9.

c Estimated as ratio of weight/weight of CO-PO link.

d Not achieved if I_SP-PO < 0.6; adequately achieved if I_SP-PO < 0.8; achieved if I_SP-PO < 1.15; overemphasised if I_SP-PO >1.15.

Having related student performance to CO, the relative achievement of CO and PO based on student performance can be estimated mathematically. The estimated level of achievement of CO (I_SP-CO) and level of achievement of PO (I_SP-PO) are presented in Tables 10 and 11. Both, I_SP-CO and I_SP-PO depend on student grades as well as on the relative emphasis of CO and PO in SAU. The indicators, I_SP-CO and I_SP-PO, are fractions that have theoretical target values equal to one that can be achieved if the following two conditions are met: (1) the performance of all students was perfect (i.e. all students achieve perfect grades on all SAU); and (2) the delivered SAU are perfectly aligned with the CO as intended by course design. Consequently, I_SP-CO and I_SP-PO can indicate one of the following outcomes:

1. CO or PO are ‘not achieved’ because the CO are ‘not addressed’ in the SAU;

2. CO or PO are ‘not achieved’ due to either that the students performed badly and/or the CO and PO were ‘inadequately addressed’ in the SAU;

3. CO or PO are ‘adequately achieved’ or ‘achieved’, depending on the limits set to differentiate between the different levels of I_SP-CO and I_SP-PO achievement;

4. some CO or PO are ‘overemphasised’ at the expense of other CO or PO that are underemphasised. Overemphasis results when a particular CO is overemphasised in the SAU delivered by the instructor while simultaneously students perform exceptionally well on all SAU related to that CO.

Analysis of the data in Tables 10 and 11 reveals that the CO were not addressed, or not achieved, or adequately achieved, or achieved, or overemphasised. In addition, the results suggest that both the simple and elaborate approaches resulted in comparable I_SP-CO and I_SP-PO values. Clearly, using the simple approach removes the need to collect student performance data on each CO separately.

Closing the loop: re-alignment of CCU and SAU with CO and PO

Having estimated the levels of emphasis of CO and PO in delivered course contents (I_CCU-CO and I_CCU-PO) and student assessment (I_SAU-CO and I_SAU-PO), and having related the SAU to the CCU (I_SAU-CO/I_CCU-CO and I_SAU-PO/I_CCU-PO) and estimated the relative achievement of CO and PO based on student performance (I_SP-CO and I_SP-PO), the instructor can use the results as bases for re-aligning the CCU and SAU with CO and PO. For example, the original course alignment results described in Tables 1–11 are summarised in Figures 1 to 3 together with results achieved after introducing modifications that aimed at improving the alignment of the CCU and SAU with CO and PO. In this case, the main actions taken to enhance the alignment of delivered CCU and SAU with CO and PO were the following:

1. As indicated in Tables 1 and 5 and Figure 2a (before re-alignment), one of the course outcomes (CO-5) was consistently not covered nor assessed in the delivered CCU and SAU. The relevance of CO-5 was reconsidered, and as a result it was dropped from the list of CO. The CCU and SAU that were originally assigned to CO-5 were redistributed among the remaining four CO.

2. After dropping CO-5 from the list of CO, the links between the remaining CO (i.e. CO-1 to CO-4) with the PO, as well as the weights assigned to the links (Table 3), were considered adequate and therefore were unchanged.

3. The overemphasis of CO-2 and CO-4 in the delivered CCU, as in Table 2 and Figure 2a (before re-alignment) was modified, which resulted in CCU that are adequately aligned with the CO (Figure 2b, after alignment).

4. The overemphasis of CO-2 and CO-3 and underemphasis of CO-4 in the delivered SAU, as shown in Table 6 and Figure 3a (before re-alignment), were modified, which resulted in SAU that are more aligned with the CO (Figure 3b, after alignment).

5. The modifications described in points (1) to (4) above significantly improved the alignment of the CCU and the SAU with the PO (Figure 4a, before re-alignment compared to Figure 4b, after re-alignment).

The modifications described in points (1) to (5) resulted in a highly aligned course experience, as shown in Figures 2–4 (after re-alignment). Furthermore, the relative achievement of CO and PO based on student performance after re-alignment of delivered CCU and SAU with CO and PO became more indicative of student performance and less biased by inconsistency in emphasising the CO in delivered SAU (Figure 4, before and after re-alignment). Clearly, the rating of student performance differs from instructor to instructor and from institution to another. As a result, using student performance as a measure of the level of achievement of PO (I_SP-PO) merely indicates the contribution of a particular course in a programme to achieving the PO based on the instructor’s practice in rating student performance.

Figure 2 (a) Original alignment (before) and (b) re-alignment (after) of delivered CCU and SAU with CO as indicated by I_CCU-CO, I_SAU-CO, and I_SAU-CO/I_CCU-CO.

Figure 3 (a) Original alignment (before) and (b) re-alignment (after) of delivered CCU and SAU with PO as indicated by I_CCU-PO, I_SAU-PO, and I_SAU-PO/I_CCU-PO.

Figure 4 Relative achievement of (a) CO and (b) PO based on student performance before and after re-alignment of delivered CCU and SAU with CO and PO as indicated by I_SP-CO and I_SP-PO.

Supporting evidence and documentation

QA and accreditation practices require that triggers for course and programme improvements be identified and supported by objective evidence (Mak & Frezza 2006) and the processes that lead to adopting such improvements be documented. Furthermore, outcome-based QA and accreditation practices require that CO be directly linked to PO and that the levels of contribution of CO to PO be indicated. Visiting accreditation teams verify, through inspecting course records and interviewing students, that stated CO, levels of emphasis of CO, and contributions of CO to PO are adequate and are reflected in CCU and SAU. Instructors typically use course files as evidence to demonstrate that CO, CCU and SAU are meaningful and appropriate, and that delivered CCU and SAU are aligned with CO and PO. Course files typically contain information such as: course outlines stating CO, assessment items and weights assigned to different assessments, course topics and schedule indicating emphasis of the different topics; textbook information; copies of lecture notes and other teaching and learning materials; copies of assessment items and worked solutions; samples of graded students’ work with evidence of clear feedback to students; student views on CCU, SAU and instructor; and any other relevant course materials.

The Civil Engineering Programme (CEP) at the University of Sharjah (UoS) adopted the mathematical approach outlined in this article as part of the QA system for all engineering courses in the programme. Starting 2008, instructors in the CEP started using a spreadsheet programmed using the mechanism described in the article to assess the alignment delivered CCU and SAU with CO and PO. Furthermore, the indicators outlined in this article were used to successfully demonstrate to local and international programme accreditation teams (CAA, UAE, and Accreditation Board for Engineering and Technology, ABET, USA) between 2008 and 2013 that CCU and SAU in the CEP are relevant and are adequately aligned with CO and PO.

Summary and conclusions

A simple mathematical approach was used to generate a variety of indicators of alignment of delivered CCU and SAU with CO and PO. Additional indicators were also derived from student performance and were used to assess achievement of CO and course contribution to achieving PO. The proposed indicators include: emphasis of outcomes in delivered CCU (I_CCU-CO and I_CCU-PO); emphasis of outcomes in SAU (I_SAU-CO and I_SAU-PO); alignment of SAU with CCU (I_SAU-CO/I_CCU-PO and I_SAU-PO/I_CCU-PO), and relative achievement of CO and PO based on student performance (I_SP-CO and I_SP-PO). Estimation of the numerical values of the various indicators requires minimum effort from instructors, mainly to assign relative weights to the different CO, weights to the links between the CO and PO, and weights to the relative emphasise of CO in delivered CCU and SAU. The proposed indicators can provide, over a number of course delivery cycles, rational bases for aligning CCU and SAU with CO and PO, as well as for revising CO and links between CO and PO. The mathematical approach can easily be programmed in a spreadsheet that can be used to automate the estimation of the various indicators, compare the results with limits set by the user, and generate a summary report with appropriate tables and figures. The application of the mathematical approach was demonstrated using an actual course taught by the author and the results clearly revealed the simplicity and appropriateness of the proposed approach for aligning CCU and SAU with CO and PO. It should be emphasised however that the mathematical approach described in this article merely provides an extra tool in support of achieving constructive alignment in teaching and learning.

Abbreviations

CO	=	Course outcomes
CCU	=	Course content units
ICCU-CO	=	Indicator of level of alignment of CCU with CO
ICCU-PO	=	Indicator of level of alignment of CCU with PO
ISAU-CO	=	Indicator of level of alignment of SAU with CO
ISAU-PO	=	Indicator of level of alignment of SAU with PO
ISP-CO	=	Indicator of level of achievement of CO based on student performance
ISP-PO	=	Indicator of level of achievement of PO based on student performance
PO	=	Programme outcomes
QA	=	Quality assurance
SAU	=	Student assessment units