Assessment strategies: what is being measured in student course evaluations?

Abstract

In developing classroom teaching strategies, it is important for the accounting instructors to obtain feedback that allows them to adjust their teaching methods to fully meet the needs of their students. One important form of feedback comes from students' course evaluations. These course assessment instruments are used in promotion and tenure decisions as well as to provide faculty with feedback about the success of their teaching methods. Currently, students' evaluations represent the primary means used by most accounting programmes to assess their accounting courses. For this reason, it is important to determine the feedback provided from these evaluations to improve teaching methods. Two paradigms for learning are considered in this paper along with their underlying learning activities. Then, a qualitative review of course evaluations collected from a sample of 267 US accounting programmes is used in developing a theory of assessment consequences.

Keywords:

• student course assessment
• teaching evaluations
• grounded theory
• learning theories
• NUD∗IST

Notes

¹By 1995, there was a total of 1500 references dealing with research on student evaluations of teaching (Cashin, 1995). In accounting journals, the number of references is less.

²Although there is a great deal of commonality in academic terminology and practice from one country to another there are also distinct differences. The data for the study was collected from US accounting programmes and the paper adopts US terminology, but the several distinctions between US and other countries' academic systems should be mentioned. In the USA, it is common practice to administer students' course evaluations at or near the last day on which the course meets.

In the USA, a 15 week unit of study is called a semester. During each semester a student will normally attend five different courses (classes) with different topics of study completing 150 minutes in a classroom each week for each course. Most classes meet for 50 minutes three times during the week. A program or program of study is the student's major area of study such as accounting or biology.

Under the tutorial systems, found in New Zealand, and Australia, for example, an 11 week unit of study is called a term rather than a semester. A class is conducted with a tutorial session and possibly workshop sessions. Here, the term course may more commonly refer to an entire programme of study, such as accounting or biology, rather than a single class as in the USA. The purpose of the tutorial is to allow students to meet in small sessions and discuss topics of interest in more depth than would be available in the large lecture hall meetings. The workshop session is designed to help students with specific homework problems. Thus, the weekly class meeting times under this system would be longer than the 150 minutes in the US system. The student would spend between 240 and possibly 300 minutes per week in class. In addition, the tutorials provide the potential for addressing multi-faceted problems with complex solutions.

Although team teaching is used in the USA, in most US universities there is a culture of having one instructor conduct one course. In British-based tutorial systems, usually one instructor conducts the lecture and another conducts the tutorials and workshops. Thus, there is less of the ‘one instructor, one course’ rule in effect in these systems.

These differences should be considered as the paper is read.

³A web search of Texas A&M, Kuwait University, University of Illinois at Urbana-Champaign, Kansas State University, University of Illinois at Chicago, Louisiana State University, University of Notre Dame, University of Nebraska, and the University of Northern Colorado shows that each has higher-level learning objectives outlined in their accounting programme mission statements.

⁴Another topic separate from the one discussed in this paper deals with the correct review of the summary course evaluation data and its use by reviewers. Ill-conceived review practices can misconstrue the entire course evaluation.

⁵The Appendix provides a chronological listing of the articles published within the main accounting education journals over the last 28 years dealing with students' course evaluations. Several of these studies empirically tested the validity of course evaluation scores with ‘learning’ or ‘teaching’ measures.

⁶In addition to these two opposing theories, there are generative learning theory (Wittrock, 1992) and cooperative learning (Davidson and Worsham, 1992). Both of these latter models are supportive of constructivism. Connectionism (Plunkett and Sinha, 1992), with its emphasis on developing memory associations and use of repetitious feedback for learning, is similar to the stimulus-response methods of behaviourism.

⁷Jonassen et al. (1999) describe a series of statements that can be used to measure improvement in the learning environment. The measures are designed to separate constructivist and behaviourist approaches. For example, a more constructivist approach would make students responsible for developing their own learning goals. The areas of assessment are related to activity (four statements), construction (two statements), cooperation (four statements), authenticity (four statements), and intentionality (six statements). Each statement is designed for a Likert survey. Table 1 is based on Jonassen's et al. descriptive statements.

⁸In Table 1, the terms in Column 5 under ‘NUD∗IST nodes’ are used in evaluating the sample collected in the study and will be referred to later in the paper.

⁹Likert statements are usually rated on a one to five scale of ‘strongly agree’ or ‘strongly disagree’, for example. These statements are written as a positive opinion statement (high score) or negative opinion statement (lower score) about a topic under consideration such as value of the course to the student.

¹⁰The acronym NUD∗IST stands for Non-numerical Unstructured Data Indexing Searching and Theorizing software. It is an example of qualitative data analysis software such as the Atlas, Qualrus, and Ethnograph packages. For information about the use of NUD∗IST software see Richards (1998) and Weitzman (2000). Additional information is also available at the NUD∗IST web site at: http://www.qsr.com.au. For explanations of using qualitative data analysis computer packages in educational research see Holbrook and Butcher (1996).

¹¹For examples of qualitative studies using NUD∗IST software see: Gray and Smith (2000), Lewis (2001) and Agosto (2002).

¹²Prior to beginning the analysis with the NUD∗IST software package, the collected surveys had to be recoded for entry into the data analysis software. Initially all the collected course evaluation surveys were scanned and the resulting images cleaned, checked, and saved into a word processing software package in text-only or ASCII mode. At this point, all collected survey questions dealing with age, gender, class status, point average, etc. were dropped out of the database. The remaining raw data was imported into NUD∗IST with the smallest data element for investigation identified. In this case, the smallest data element is the survey statement. The statement data was then explored and placed into nodes for further linking of concepts underlying the data. Nodes represent the qualitative indexing system that is being developed from the raw database.

¹³All generative learning theory and cooperative learning methods were identified with constructivism. Connectionism was identified with behaviourism. This initial analysis on teaching feedback is performed to help identify the primary assessment areas that are developed in the course surveys.

¹⁴Students' course surveys ask students to evaluate ‘real-world’ or accounting practice-related problems. The question may not be appropriate for every course. The purpose of the question is to determine if students are spending time on activities that occur outside the classroom such as: (1) a student in an accounting internship for a portion of a semester, or (2) a student team involved in meeting with a company's executives as part of a course assignment to solve a real-world problem faced by the company. It may appear that students with no practical work experience are being asked to provide answers to questions for which they have no background. But most students are aware as to whether accounting course activities involve them with real-life problems outside the classroom.

¹⁵Although there were other statements about course participation, these two statements were directly related to ‘student interactions’.

¹⁶Thus, 3.9% of the collected statements used behaviourist or constructivist approaches. If only one statement about learning theories was on each collected questionnaire at least 53 statements about behaviourist or constructivist approaches would be present in the data. Of the 53 questionnaires collected, the 38 statements in Table 2 were found on 36 returned questionnaires. The average number of statements on the 53 returned questionnaires was 19. At this point with the limited number of statements in the NUD∗IST nodes, no attempt was made to cross-categorize the data.

¹⁷The criteria for including a factor as part of a grounded theory include the following: (1) the category's centrality in relationship to the other categories; (2) the frequency of a category's occurrence in the data; (3) its inclusiveness and the ease with which it is related to other categories; (4) the clarity of its implications for a more general theory; and (5) its allowance for variation in terms of dimensions, properties, conditions, and consequences (Strauss, 1987).

¹⁸To develop the Tables, the NUD∗IST database was explored for teacher actions such as explanations, grades, teacher responses, illustrations, and encouragements. Teacher/student interactions in the areas of assignments, homework, cases, presentations, tests/exams, discussions, feedback, availability, and office visits were explored. The searches were cross-classified and overlapped into teacher/student interactions with feedback, student understanding, and teacher presentations. NUD∗IST nodes and tree displays were developed for the categories.

The categories were based on student-teacher interactions. There were other statements in the NUD∗IST database related to ‘course’ evaluation. The course-related statements were not included in the analysis as the development of thinking skills is based on the teaching interactions between the instructor and the student. Consequently, statements related to course assessment, per se, are not included in the Tables.

¹⁹It is not considered necessary to place all 978 Likert statements into interaction categories. NUD∗IST categories coded under ‘student/teacher interactions’ represent 12% of the total statement sample and exhausted the classifications in that area.

²⁰Feldman's (1976) dimensions are value of the course, teachers' interest in the course, enthusiasm, subject matter knowledge, breadth of subject coverage, preparation and organization, presentation skills, speaking skills, sensitivity to student achievement, clarity of objectives, value of supplementary materials, classroom management, frequency and value of feedback, course difficulty, fairness, openness, encouragement and challenge, availability, and respect and friendliness.

²¹The American Accounting Association published the Accounting Teachers' Guide in 1953. Chapter 8 is titled The Student Evaluates the Teacher. One guideline for assessment was the major contact points between the instructor and the students. The categories of assessment were: homework, tests, teacher's attitude, and teachers' habits. The short answer questions regarding homework policies are reproduced here because they illustrate the more specific nature of students' course evaluations that allow for clear revisions in an instructor's teaching methods.

The following questions are asked of students regarding their homework.

1. Is the amount reasonable for the course credit?

2. Is the work reasonably well distributed over the course?

3. Is the purpose of the assignment clear and its relationship to the subject matter indicated?

4. Does the homework cover the minimum required for understanding of the course and yet open avenues to challenge better students?

5. Do the problems cover the subject without being unnecessarily long or repetitious?

6. Are the problems representative of business situations?

7. Are standards explained and upheld fairly?

8. Are ambiguous statements eliminated from the problem or explained in advance?

9. Are arithmetic or typographical errors avoided or explained?

10. Are papers graded fairly and in sufficient detail?

11. Are papers returned promptly?

12. Is the homework considered a proper share of the final grade? (pp. 237–38)

These questions bring out more details of teaching methods in a way the statements about ‘reasonable’ or ‘fair,’ homework in today's course evaluations cannot. It should be noted that these questions were written more than 50 years ago.

It is also worth noting the format in which a question on ‘student thinking’ is presented:

Among other things, it is felt that a college education should train a student to think and reason (not just memorize) his way through problems clearly and objectively. It is a task of the instructor to train the student to think in this manner. Form this standpoint, I consider my instructor to be:

• ( ) Decidedly inadequate

• ( ) Somewhat inadequate

• ( ) Adequate

• ( ) Good

• ( ) Among the best (p. 245)

²²If a Likert statement was stated as follows: My instructor says ‘Hello’ to me in the hallway. A strongly agree (higher response) from the students would be reflective of a better course, but it would not be reflective of any thinking skill development among the students.

²³‘Student representatives in campus governance pressed hard for a system of student evaluation of faculty performance in the classroom. Faculties tended generally to go along with this demand, and formal or informal arrangements were made on most campuses in this study to enable students, at the end of a course, to evaluate the competence of the instructor. The real issue, however, was not student evaluation of faculty performance, but faculty use of it. On some campuses the evaluation information was made available only to the instructor. On other campuses, it was also made available to department chairman’. (Millet, 1978, p. 218).

²⁴Dressel and Faricy (1972) writing on university responsibilities shortly after the campus unrest of the 1960s state that:

The autonomy of universities will be constrained by an increased sense of social responsibility, but certainly by public demand and intervention by those who provide the resources (p. 188).

Also see Bonk and Smith (1995/1996).

²⁵Critical reports on the technical teaching used in the US accounting curriculum and the need for change can be traced back to the beginning of the last century (Nelson, 1995).

²⁶A statement written by Bedford et al. (1961) more than 40 years ago still has pertinence today as its references to a decline in the quality of accounting students is echoed by Albrecht and Sack (2000). …No longer is it sufficient that an accountancy education confine itself to training a student to become a mechanical expert in accounting procedures. Failure to adjust an accounting curriculum to an educational level somewhat above this, wherein concepts, reasoning, and understanding are of the essence of the program of study, will lead ultimately to a decline in the quality of students interested in the field. More directly, however, the profession of accountancy is changing and expanding, and with change has come the realization on the part of practitioners of the profession that the educational process should provide a broader approach to the practice of accountancy. (p. v–vi)

²⁷‘The centerpiece of grounded theory research is the development of a theory closely related to the context of the phenomenon being studied. This theory…can assume the form of a narrative statement’ (Creswell, 1998, p. 56).

²⁸…we specify the conditions that give rise to certain phenomena…and explain what consequences occur as a result of those actions/interactions (Strauss and Corbin, 1990, p. 267).

²⁹The IDEA Center is located at 211 South Seth Child Road, Manhattan, KS. See: http://www.idea.ksu.edu. The Centre will process and summarize the result of the students' course survey for any accounting programme.