Forced to improve: open book and open internet assessment in vector calculus

Abstract

Calculus is traditionally tested, at least in part, by a closed book written examination taken under secure conditions. In March 2020, two-thirds of the way through a vector calculus course, our institution went into lockdown and all teaching and assessment had to shift online. While some teachers insisted that remotely written non-proctored examinations remain closed book, we accepted that this would result in inequity, with some students certainly choosing to flout that rule should it be imposed, to their advantage over their classmates. Driven by circumstance, we were forced to design a vector calculus examination that would be written open book and ‘open internet’ with access to online calculators and computer graphing packages. Proficiency at vector calculus is exhibited far more in the interpretation of information and effective decision-making than in the computational solving of integrals. In this article, we shall present and discuss our experiences, address the challenge of non-proctored examinations and argue for the merit of open book examinations in an otherwise standard vector calculus course.

KEYWORDS:

• Vector calculus
• open book testing
• decision-making
• open internet testing

1. Introduction

Assessment drives student learning and informs the student of what the teacher values (Iannone, 2020). If what we value is higher-order cognitive skills and conceptual understanding, then our assessment needs to make these priorities clear. Mathematics service courses taught to students who need to use mathematics as one of many tools in engineering or physics applications need to foster the problem-solving approaches valued by those disciplines. Thoughtfully designed curricula aim to assess conceptual understanding and help develop mathematical thinking, but it is easy in the absence of a call for change to get stuck in a rut where assessment has become routine and procedural. The circumstances of 2020 forced many of us to reconsider the underpinnings of our teaching and assessment and to refresh outdated systems. We chose ‘decision making’ as a core skill in vector calculus problems, such as using informed judgement to choose the right mathematical tool for the task at hand (Pepper et al., 2012) and recognizing available options, and hence chose to design our open book non-proctored tests about that core principle. Decisions related to, for example, conservative vector fields and choice of theorem are the key to proficiency in vector calculus. Adjusting our assessment to this new mode was achieved by presenting information in diagrams requiring interpretation, presenting problems without instructions on how to solve them and hence encouraging decision making, and focusing on setting up but not evaluating integrals. Having been forced to change our assessment, we have come to welcome the shift in approach and to look ahead to future teaching and learning benefit.

An open book test written remotely and non-proctored is necessarily also ‘open internet’ with access to many powerful digital tools. Online calculators are useful for solving integrals whose bounds have already been set, even changing the order of integration if necessary, and computer graphing packages can depict surfaces and curves remarkably well. The opportunities for incorporating digital technologies into mathematics classrooms have become more accessible and more powerful with every year. Dimiceli et al. (2010) discuss some useful aspects of Wolfram Alpha; they point out some weaknesses that no longer exist 11 years later or can be compensated for with other software such as GeoGebra. GeoGebra is versatile free graphing software with a growing library of user-designed dynamic tools. Ponce Campuzano et al. (2019) discuss the use of GeoGebra effectively in the context of vector calculus and Caglayan (2016) recommends GeoGebra for the visualization of double integral and Riemann sum concepts.

Closed book, timed tests, usually without any electronic devices, such as calculators, are the norm in mathematics assessment (Iannone, 2020; Trenholm, 2007). Khan (2015) considers the opportunities afforded by open book testing in statistics and mathematics and observes that while open book tests provide an opportunity for assessing higher level understanding, well-designed closed book mathematics tests often do assess for higher level understanding; he concludes that the argument for open book tests in mathematics is unclear. Faced with an unavoidable shift in assessment due to the coronavirus situation, Bailey et al. (2020) concur that ‘open book exams are better suited to questions requiring higher order skills’ (Bailey et al., 2020, p. 1) and outline ways in which typical closed book questions can be evaluated and modified to suit this mode. They are guided by Smith et al.’s (1996) MATH taxonomy to suggest some practical suggestions not dissimilar to those discussed in this article and elsewhere (Craig, 2021). Iannone (2020) also suggests types of questions suitable for open book testing and goes further to suggest other types of assessment altogether, such as oral tests. In agreement with several of Bailey et al’s and Iannone’s question design suggestions (Bailey et al., 2020; Iannone, 2020), Jungic (2020) changed his calculus exam to involve application and explanatory questions based on real data on current public health events. In short, the coronavirus situation has forced many of us to change our assessment practices in creative ways which, while uninvited, have proved to be worthwhile and worth maintaining.

2. Context

The vector calculus course we teach every year runs from early February to early April. At our institution, almost all courses are undertaken for one academic quarter. The vector calculus course under discussion is covered in the third quarter. The assessment has been a traditional closed book test completed without a calculator. Two-thirds of the way through the course the university ‘locked down’ in response to the coronavirus pandemic, meaning that in-person classes were no longer allowed due to health and safety concerns. All teaching had to take place online and assessment possibilities were limited to online or remote options. Our test date was shifted by one week and we had to completely restructure our plans. With few options available to us at the time we opted for an open book, non-proctored test, handwritten at home and submitted as pdf for grading. We could have insisted on the test remaining closed book (as some colleagues in the same situation in other courses did) and could also have insisted on no use of calculators or computers. We predicted that some students would flout that directive, leading to significant inequity and hence we avoided imposing those constraints.

After a process of reviewing multiple old test papers for the course and breaking down all the sorts of questions we could ask in a limited time test, we settled on a list of criteria for question design. These criteria guided us in designing assessment items, requiring decision-making, offering options, providing diagrams instead of asking students to draw them themselves and setting low to no emphasis on actual computational solving of integrals. In 2021, we continued to apply this model and the examples below are drawn from both academic years.

Two separate sets of challenges confronted us in the two vector calculus courses of 2020 and 2021, the first being setting open book, open internet tests and the second being maintaining academic integrity in the remote non-proctored environment. We regard the steps we took to maintain academic integrity as satisfactory; however, discussion of these steps is beyond the scope of this article. Here, we shall address our response to the first challenge.

3. Examples illustrating principles of question design

Four examples have been chosen to illustrate the principles we used to design our open book, open internet tests. Each is contrasted with an assessment item on the same topic, however, not exemplifying the design criteria (chief amongst which is decision-making) underpinning our more recent approach. These assessment items are drawn from vector calculus tests at our institution, both our own course and others'. To prepare the students in 2020 for this sort of question, a selection of past test papers was made available with accompanying descriptions indicating, which questions were good examples of what they could expect in their upcoming test and which questions were bad examples. In 2021, the students had the previous year’s test to prepare them and weekly quizzes we designed to provoke a thoughtful approach to vector calculus questions. One quiz question, for example, was very similar to the test question in Figure 1.

Figure 1. Decision-making and processing information.

The circulation of a vector field is defined as a line integral along a closed curve, being the boundary of the given surface, described in Figure 1. Stokes’ theorem allows the line integral to be substituted with a certain surface integral either over the given surface or any other surface sharing that boundary curve. To answer the question in the example in Figure 1 the student needs to know that the surface can be replaced and then needs to determine the boundaries of the four surfaces described. The question is answered most rapidly if the students already know this information, but they also had access to their textbook, all their notes and any graphing packages available.

We can contrast the Stokes’ theorem question above to an arguably more standard item such as ‘Use Stokes’ Theorem to evaluate the work done by the velocity field $\mathit{v}=(y+xz)\mathit{i}+(2x+yz)\mathit{j}+(z^2)\mathit{k}$ moving a particle along C where C is the boundary of the hemisphere $z=\sqrt{4-x^2-y^2}$.’ We are choosing this item as contrast because it also includes the option to exchange the given surface (the hemisphere) with another (a planar disc); however, the failure to notice this option is no impediment to following a standard, memorized, computational algorithm.

Figure 2 shows a region in the xy-plane bounded by a parametrically defined curve. The student is required to calculate the area of half of that bounded region. To solve the example shown in Figure 2, the student needs to know (or know how to look up) that area of a bounded region can be expressed as a line integral along the boundary using Green’s theorem. The region described in the example is half of that bounded region. Once having realized that Green’s theorem could be utilized the student would then need to determine the parameter values describing the closed curve. In grading this question, very little weight was given to the computed solution.

Figure 2. Interpreting information, decision-making, using a diagram.

This example, requiring the use of Green's theorem to translate the double integral for area into a line integral along the boundary, can be contrasted with items such as ‘Use Green's Theorem to calculate the area of the disk $D$ of radius $r$ defined by $x^2+y^2\le r^2$.’ This contrasting item stops short of supplying the formula to be used, but otherwise requires very little from the student other than a good memory and computational skill. The bounds of integration are predictable and could be applied by the student with little understanding of what they represent.

Figure 3 shows a region, a lamina, in the xy-plane piecewise bounded by four given curves. The student is required to set up two double integrals, one in each Cartesian order, describing the mass of the lamina. In a standard closed book test, the example in Figure 3 would not include a diagram; drawing a diagram would be part of the expected solution. This question was set in the April 2020 test where the students had no expectation that skill with graphing software might be useful. Requiring that the students include a diagram in their solution would, therefore, result in inequity since two students with equal difficulty in hand drawing such diagrams could fare very differently if one used computer graphing software and the other did not know how to do so. In a significant deviation from how we traditionally asked vector calculus questions we shifted to including diagrams. Representing the low emphasis on computation in our new design scheme this question explicitly did not require any computation. Prior to taking the approach discussed in this article, we have set test questions such as ‘Lamina D is bounded by $x=0$, $y=0$, $y=2$ and $y=3-x^2$. The density $\sigma$ of D at point $(x,y)$ is twice the distance from the point to the y-axis. Evaluate the mass of D.’

Figure 3. Providing a diagram and not requiring solving an integral.

The standard definition of flux in the context provided in Figure 4 is that of an oriented surface integral. The surface is projected onto a coordinate plane and with the aid of a conversion factor it is reframed as a double integral and so solved. In the problem shown in Figure 4, the projected region is given which prompts the student to make decisions related to rewriting the surface equation as a function of x and y and how to proceed to set up the necessary double integral.

Figure 4. Decision-making and getting information from a diagram.

In our experience, students find oriented surface integral problems challenging even if presented in a familiar, textbook-standard format, such as ‘D is the solid described by $\left\{(x,y,z)\in {\mathbb{R}}^3:\sqrt{x^2+y^2}\le z\le 1\right\}$. At the surface S of solid D the outward pointing normal vector is denoted by $\hat{\mathit{N}}$. Given the vector field $\mathbf{F}(x,y,z)=(x-1)\mathbf{i}+2yx\mathbf{j}+xz\mathbf{k}$ calculate $\underset{S}{\iint }\mathit{F}\cdot \hat{\mathit{N}}\text{d}S$ directly, without using Gauss’ Theorem.’ Our example in Figure 4 does not represent a more challenging question than this standard item but rather one that is less susceptible to using an online calculator to solve (before having to make certain decisions) and is less likely to immediately match a standard solution template.

4. Takeaway

Our key takeaway from teaching and assessing vector calculus in 2020 and 2021 is that vector calculus is surprisingly accessible to open book (and open internet) assessment as long as the questions are designed with decision-making as the core guiding principle, backed up by using diagrams in useful ways and placing low emphasis on solving integrals. In 2020 and 2021, we allowed students to use online calculators and graphing software; however, the use of these tools has not yet been built into the curriculum more generally. Moving forward, we shall maintain open book testing and encourage the use of online tools.

Topics other than vector calculus could benefit from allowing options for decision-making in the assessment. An example from linear algebra is determining whether the representative matrix of a linear transformation is invertible or not. Options include finding the determinant by direct computation or by inference from other information, determining the kernel and/or image of the transformation, or geometrically describing how vectors are transformed. The assessment item could provide enough information for more than one of these options to be possible and the student is then placed in a position of having to both recognize the availability of options and decide how to choose among them.

5. Conclusion

All going well we shall not have to run a remote non-proctored test again with its associated weakness with regard to cheating. Similarly, we could go back to the traditional closed book test, but we shall not do so. Having been forced into the position of redesigning our assessment for the context of access to all notes and the tools available on the internet, we have been pleasantly surprised by the experience.

We designed assessment questions which placed decision-making, providing diagrams, providing information through diagrams and placing a low emphasis on computation at the core of the design process. This form of question in the open book, open internet context was well received by the students and the grade profile was not significantly different from previous years. The grading process was carried out in a standard manner although equity considerations remain, such as how to grade when one student solves an integral by hand and another uses Wolfram Alpha.

Certain types of questions we used to ask are no longer suitable for the open book, open internet environment, such as ones heavily reliant on computation or on drawing a diagram, in essence any question easily outsourced to an advanced calculator (see also Bailey et al., 2020). Proficiency at vector calculus is not demonstrated by computation, however (Bollen et al., 2015). We argue that it is demonstrated by information interpretation, recognizing available options and effective decision-making. There is not a great deal of research taking an overview on the teaching, learning and assessing of vector calculus (Martinez, 2019), although valuable work has been done on specific topics, such as line integrals (Mallet, 2013) or double integrals (Trigueros & Martínez-Planell, 2018). We hope that our experiences discussed here contribute to this valuable field.

In 2022 and beyond when we hope to be back in a physical classroom we shall retain the best features of our new assessment design. Our institution has a bank of Chromebooks available for assessment purposes. With forward planning, these can be set up to allow access to a virtual desktop and certain white-listed online resources so that students can use computer graphing packages, powerful online calculators and have access to certain course documents such as lecture slides. Planning has already started for the 2022 vector calculus test to be written in this form.

In March 2020, we greeted the new situation with dismay but swiftly realized that the difficult process of redesign was forcing us to improve our assessment offering. While we shall be relieved to put much of the 2020 and 2021 global experiences behind us we are pleased with the changes we have wrought in this aspect of our work and are excited to move forward.

Disclosure statement

No potential conflict of interest was reported by the author.