Designing and implementing a comprehensive assessment plan for a graduate accounting programme

Abstract

Accounting programmes are being challenged (e.g. Albrecht and Sack, 2000) to make accounting curricula more relevant and to present assessment data regarding the efficacy of course and curricular offerings. In addition, assessment is embedded in current accreditation standards in the USA by the Association to Advance Collegiate Schools of Business (AACSB) International as fundamentally important to motivating continuous improvement (AACSB 2005). However, survey results in this area (viz., Kimmel et al., 1998, Issues in Accounting Education, 13(4), pp. 851–868 and Calderon et al., 2003, Best Practices in Accounting Program Assessment, Sarasota, FL: Teaching and Curriculum Section, American Accounting Association) indicate that assessment is not as widespread as might be expected. Further, Akers et al. (1997 Issues in Accounting Education, 12(2), pp. 259–280) note that the accounting education literature is limited with respect to descriptive accounts of assessment programme design and implementation results. This paper offers a framework, consisting of three key attributes (broad-based involvement, multi-trait/multi-method approach, and assessment follow-up), that can be used to guide the development and implementation of a programme-level assessment plan in accounting. A case study applies this framework to the development and implementation of an assessment plan for a graduate accounting programme in the USA. This plan includes multiple assessment measures and provides feedback regarding both academic (curriculum-related) and administrative (operational) dimensions of performance. To demonstrate the value of such a plan, a discussion of programme changes made in response to assessment data collected to date is included. While the application of this framework is to a USA context, the framework is such that it can be used more generally to guide the design and implementation of an effective programme-level assessment plan.

Keywords:

• AACSB accreditation
• program-level assessment
• outcomes assessment
• direct assessment measures
• indirect assessment measures
• graduate accounting programs
• multi-trait/multi-method assessment plan

Notes

1. Elsewhere in the document, the AACSB International states that ‘accredited accounting programs must provide evidence that learning goals have been achieved or that graduates have met the set of learning goals specified by the program.’

2. We differentiate between course-level (i.e. course-embedded) and programme-level assessments. Examples of course-level assessments (not the subject of the present paper) are found in, Bence and Lucas (1996), Sangster (1996), and Ammons and Mills (2005). Bence and Lucas (1996) report on the use of Objective Testing (OT) in first-year undergraduate accounting courses in the UK. Sangster (1996) looks at the use of formative OT in a ‘second-level’ accounting course in a UK university; and Ammons and Mills (2005) focus on the development of an assessment mechanism regarding the development of a ‘cross-functional view of business’ within the context of a six-credit, undergraduate team-taught course (‘The Design and Delivery of Goods and Services’) at a US institution. Palomba and Banta (1999, p. 5) suggest the following potential benefits of course-embedded assessments relative to programme-level assessments: reduction in data collection time (i.e. instruments can be designed from assignments already planned for the course in question); enhanced student motivation (i.e. the grading of course-level assignments may cause students to respond more seriously to the instruments); timeliness of information supplied to faculty; and, use of assessment results to improve instructional strategies and course design.

3. The authors report using at ‘three different universities with different missions’ the assessment instrument they developed.

4. As indicated later, the comprehensive assessment plan discussed in this paper utilizes several student self-report inventories, each of which provides indirect evidence of learning outcomes.

5. Hand et al. (1996) provide an alternative framework for the development of what they refer to as the ‘teaching and learning paradigm.’ Essentially, the model depicted by Hand et al. consists of a cyclical process with three primary components: programme design, programme delivery, and programme assessment.

6. Some detail regarding the case study in this paper is warranted. The title of the programme is ‘Master of Accounting and Professional Consultancy’ (MAC for short). The programme consists of ten three-credit courses, spread out over the course of two summers and intervening autumn and spring semesters. During each summer session, students take four (three-credit) courses. During each intervening semester, students work full-time in accounting and take one three-credit course on a 100% distance-learning (DL) basis. Almost all students in the programme were undergraduate accounting majors who had graduated the prior year. Thus these students are young (average age 23), with limited full-time work experience and limited exposure to DL courses. The programme commenced in the summer of 2001 (thus, students who entered the programme at this point graduated in August of 2002); the fourth class graduated in August 2005.

7. Calderon et al. (2003, p. 12) offer a set of ‘eight characteristics of a best practice assessment programme for accounting departments.’ A reading of this listing indicates significant overlap with the three attributes associated with the framework offered in this paper.

8. An example of using the Balanced Scorecard (BSC) as a framework for assessment of accounting programmes at a US institution (Creighton University) is given in Chapter 6 of Calderon et al. (2003).

9. In a later section of this paper changes made to the case-study programme based on assessment data collected to date are discussed.

10. It is worth reiterating here that the overall goal of this paper is to provide a framework or process for the development and implementation of a comprehensive programme-level assessment plan. Thus, the specifics of an assessment plan developed by other institutions will, and probably should, differ to some extent from the specifics of the plan presented in this paper.

11. The specific survey instruments used in the present case, as well as statistical results based on data collected to date, are available from the authors on request.

12. These instruments are recommended in addition to the normal end-of-term evaluation instruments (‘student evaluations of teaching,’ or SET) that are used by many organizations. Smith (2004), for one, raises issues regarding the value of student evaluations. His point, based on a survey of evaluation instruments from 53 US accounting programmes, was that such instruments, by and large, measure student satisfaction, not other important objectives such as student skill development. A consequence of this, according to Smith (2004), is that such instruments do not yield useful information that can be used to guide instructor-teaching activities. Owing to this deficiency, the author recommends (p. 24) that at least two course-evaluation methods be used, a position that is conceptually aligned with the ‘multi-trait, multi-method’ approach to assessment plan design embraced by the present paper.

13. A separate course dealing with Risk Management and Assurance Services (taught by an accounting faculty member) has been part of the MAC curriculum from the programme's inception.

14. Thus, students in the programme are still required to take two finance courses: Global Financial Markets, and Advanced Corporate Finance.

15. The literature of accounting education, unfortunately, is practically devoid of examples of direct assessment measures. (As noted earlier in the paper, Ashbaugh et al., 2002 and Ammons and Mills, 2005, are notable exceptions.) Thus, and as indicated in Table 2, the MAC committee felt it prudent to structure the initial assessment programme around the use of indirect measures and then to experiment with the development of a set of direct assessment measures.

16. Results in the USA may be obtained through the National Association of State Boards of Accountancy (NASBA), www.nasba.org.

17. Davidson et al. (1996) present evidence regarding the validity of using the Canadian Uniform Final Examination (UFE) as a measure of ‘professional readiness’ in terms of technical knowledge in accounting, auditing, and tax. The UFE is required for admission as a chartered accountant (CA) in Canada.