A Project-Driven Second Course

Abstract

I trace the development of a new course in modern data analysis involving a wide spectrum of statistical techniques. Because the course is based entirely on case studies, real-data settings, and student projects and is computer-intensive, a series of challenges facing many instructors are addressed. In a single semester, students explore data using tools from EDA, multiple regression, analysis of variance, time series analysis, and categorical data analysis. The focus is on understanding and forecasting in a variety of data settings, learning how to summarize relationships and measure how well these relationships fit data, and how to make meaningful statistical inferences when the usual assumptions do not hold. The course emphasizes what the statistical process is all about: how to conduct studies, what the results mean, and what can be inferred about the whole from pieces of evidence.

• Active learning
• Data analysis
• Data collection
• Problem-based learning

1 Background and Development

1 This paper discusses the development of a new elective course in Data Analysis given by the Department of Operations Research and Operations Management of the Weatherhead School of Management at Case Western Reserve University. The original idea was to create a single course that would simultaneously serve as a bridge course for students in advanced graduate programs in the Weatherhead School to prepare them for advanced work in statistics, and as a terminal seminar in applications of statistics for MBA students with quantitative interests.

2 Previously, the Department offered two two-semester sequences in statistics. The MBA at Weatherhead required a fairly traditional core in quantitative methods, taught by various professors. In this stream, the first course pulled topics from exploratory data analysis, rudimentary probability, sampling, interval estimation, hypothesis testing, and perhaps some goodness-of-fit tests, experimental design, or simple regression. The second course discussed multiple regression, linear programming, simulation, and project management. A more mathematical sequence attracted students in the Masters' and Doctoral programs in Operations Research and Operations Management. Here, the first course covered probability, and the second was a basic mathematical statistics course, with very limited applied work, sometimes none at all, depending on the instructor.

3 In designing the new course (designated OPRE 404), it turned out to be especially, perhaps even unusually, useful to start from a product-consumer model. The design process focused on appealing to common student interests. Evaluations from graduates and current students indicated substantial interest in a course providing hands-on experience with statistical tools. Many MS and even some Ph.D. graduates left the Department without ever having used statistical software for classwork. Several of our terminal Masters' students, eager to obtain internships with Cleveland-area companies, found themselves in the awkward position of trying to develop a “working knowledge” of statistical software rapidly and with minimal guidance in order to meet the requirements of a specific position. Also, a small group of MBA students was interested in more quantitative careers. These students wanted more training in applied statistics, beyond that offered in the core, but were not prepared for or interested in the more mathematical sequence.

4 There can be considerable resistance in the academic community to the notion of incorporating student feedback into the design of a new course. While it is easy to conjure up horror stories and insinuate that “the lunatics will run the asylum,” the design of any new course can be substantially aided by prospective students. The primary concern, as it developed, was that it would be impossible to create a course that could be appropriate for such a diverse group.

5 To address these concerns, several informal interviews with students and faculty representing the various potential constituents were held in the early stages of the course design process, to try to identify the most crucial issues. In general, students with deeper mathematical backgrounds expressed interest in working on real-world problems, while MBA students (perhaps more accustomed to the use of real data from the MBA core) expressed greater concern regarding required mathematics and were particularly concerned with diversity of applications.

6 OPRE 404 provides motivation through the heavy use of computer packages for calculations and display, real-world “messy” applications, and exposure to a wide array of data settings. Based on the original surveys and interviews, the students are assumed only to have some background in multiple regression. Each semester, the instructor meets with all potential students (ideally in advance and in any case before the second class meeting) to try to tailor the material to the diverse group, and to, eventually, get everyone working from the same “spine” of material.

7 In four offerings, class size has ranged from 17 to 28 students, who are typically taking three or four other semester courses simultaneously. Case studies are used to motivate every new statistical tool. The course meets weekly for two hours. Typically, there is no formal lecturing at all. The instructor has made use of a wide array of statistical software packages in class, to demonstrate and try out potential analyses for each new case study, project, or investigation. More recently, the course has been given in a special classroom with computers in front of each pair of students, allowing for even more active participation and learning.

2 The Course

8 OPRE 404 surveys the fundamentals of time series analysis, regression modeling, categorical data analysis, and experimental design. Five modules have been developed for the course, although there is still considerable flexibility and variation between semesters. The first module investigates the problem of predicting continuous data from continuous data (regression analysis). The second module is concerned with predicting continuous data from categorical data (ANOVA and ANCOVA). The third module addresses some models for the analysis of time series data (smoothing techniques, Box-Jenkins ARMA). The fourth module involves the prediction of categorical data from categorical data (measures of association, log-linear models). Finally, the fifth module (much shorter than the previous four) is concerned with predicting categorical data from continuous data (logistic regression). Throughout the course, especially in introducing each new area, issues of presentation, primarily the effective use of graphics, are highlighted.

9 In lieu of a textbook, students purchase a bulkpack containing background information about all of the main case studies and some of the primary analyses, along with some “textbook-style” discussion of new topics. Readings from the bulkpack are assigned regularly, and supplemented occasionally by additional handouts and postings to a local World Wide Web page. Additionally, students have access (both on-line and in office hours) to direct computer assistance and other related reference materials.

10 In class, the instructor constantly solicits suggestions from the students about how to proceed, and attempts to encourage active, cooperative learning as often as possible by posing open-ended problems and allowing students to work cooperatively to come up with suggestions. In a majority of sessions, students are asked to walk through an analysis while the instructor serves as moderator and coach. This approach is motivated by the suggestion of Garfield (1993) that the teacher's role should be as “facilitator of learning” as often as possible, rather than merely “source of information.” At other times, the instructor takes on a more traditional role, suggesting new tools and providing focus for the interpretation of results.

11 A fair amount (perhaps 30%) of the early class time is spent discussing “sample projects,” following the lead of Fillebrown (1994) and others. The main goal of this approach in the early stages of the course is to encourage students to treat the effective presentation of information, particularly through graphics, as an area of data analysis worthy of their careful attention. Later, sample projects are used to motivate the use of a variety of statistical software tools. Fortunately, the instructor enjoys a good deal of autonomy in terms of tailoring the course to each new group of students. The four sessions conducted so far have all considered material from the four main sections of the course, and the fifth module, on logistic regression, has appeared twice. The biggest change from session to session has been in the amount of time series material discussed, as student interests have varied most in this area, both in terms of the early-term interviews, and in terms of in-class participation and demonstrated interest in forecasting.

12 The instructor has not found a single “best” software choice for all students and all analyses. For in-class analyses, students are limited to the software available on the school's network, which includes Microsoft Office, Statistix for Windows, Storm, and SPSS for Windows. MBA students in the course express serious interest in using tools that will be available to them in their future careers (especially Microsoft Excel), and are also interested in continuing to use Statistix for Windows, which is introduced in the MBA core. In pre-term interviews, more advanced students express special interest in developing their skills with SPSS and SAS. This is motivated more by their perception of what will be useful to them in their job searches, than by any prior experiences with the software.

3 Pre-Planned Deliverables

13 Assignments for OPRE 404 use both “canned” and student-generated datasets. The assignment schedule for the course emphasizes two of the main recommendations from the various Making Statistics More Effective in Schools of Business (MSMESB) conferences. First, the deliverables, like the class meetings, focus on problems in search of statistical techniques, as opposed to statistical techniques in search of problems. See Chatterjee and Tenenbein (1988), Easton, Roberts and Tiao (1988), and Roberts (1992, 1994) for more details. Second, given the importance of effective oral and written communication skills, the deliverables require students to communicate their results to several different audiences.

14 Each of the first four sections of the course (Regression, ANOVA/DOE, Time Series, and Categorical Data) is followed by a series of seven to nine investigations, which are open-ended problems with data appropriate for the analyses described in the section. The students formally analyze two or three of the problems as assignments, and present some piece of their results as fodder for class discussion. The instructor tries to match people with areas of interest or expertise when asked, but students seem to prefer making their own selections. Students submit brief write-ups of their results, along with edited output, and then in class they discuss results in great detail, commenting on different approaches and interpretations. After the first two runs of the course, a teaching assistant has eased the instructor's load considerably. In particular, this student grades the write-ups and sketches out several of the approaches suggested for each problem, which are then provided to students as a follow-up.

15 Data for the investigations is provided electronically. Some of the investigations are fairly straightforward applications of techniques studied in class, such as the following question, which leads naturally to ANOVA or a median-based alternative.

Suppose that a custom golf equipment manufacturer is making a new driver for a statistician. The manufacturer wants to determine which of three different possible lofts of club (9, 10.5, or 12 degrees) and which of four different sets of shaft specifications (1 = steel, 2 = graphite, 3 = tour-weight graphite, or 4 = titanium) is best suited for the player. Each of the 12 combinations was available in a demonstration club, and, after a suitable warmup, the golfer hit five shots with each club in a randomized design. In each case, the response of interest was a score from 0 to 100 (higher is better) which combined information equally based on the length and accuracy of the shot. Is there clear evidence as to which specification is most appropriate?

16 Note that this particular investigation was motivated by the statistician's purchasing decision, and that a very substantial interaction was observed, complicating the nature of the analysis, though not the eventual choice of best-suited club. Several of the ANOVA and experimental design investigations were motivated by suggestions from Hunter (1977), and at times the class has performed experiments related to the case studies and investigations. This idea was also motivated by Snee's (1993) call for experiential learning, as well as Sowey's (1995) goal of creating “learning that lasts.” Specifically, some of Bisgaard's (1991) suggestions for teaching experimental design using a series of paper helicopters were very productive with a group of students who had a substantial engineering background.

17 Other investigations are much more open-ended, with little more than an interesting dataset and some background information. Singer and Willett (1990), Roberts (1991), and Andrews and Herzberg (1985) were useful starting places for these examples.

18 Over the term, several classic datasets are introduced. Some serve merely as motivation for case studies, designed to introduce a new method or idea, while others provide interesting and well-traveled examples of messy problems, with no one clear “best” solution.

19 One of the more complex investigations involves the classic Elton-Nicholson (1942) dataset, used in many time series analyses. The data are numbers of Canadian lynx trapped in the MacKenzie River District of Northwest Canada for the years 1821 to 1934. The students are provided with some background information, drawn from several sources, and then asked whether they can model the data appropriately and what conclusions they can draw about the data. After the students have worked with the data for a while, the section on time series analysis is summarized using the lynx data and several approaches to their analysis, taken from the literature.

4 Projects

20 In addition to the investigations of instructor-supplied data, and the case studies interactively analyzed in class, the students are directly involved in the planning, gathering, and data analysis phases of four projects over the course of the term. Each student, either individually or in a group of two or three, completes projects in regression analysis, experimental design, and time series analysis, and a final project studying a larger dataset gathered over the course of the entire semester. Alwan and Alwan (1995) emphasized the importance of this direct student involvement in study planning and data collection, while McKenzie (1992), Sevin (1995), and Roberts (1991, 1992) have written especially persuasively on projects with more elementary classes, emphasizing the importance of clear and detailed instructions and plenty of feedback. Ledolter (1995) provides several sample projects from a second-semester MBA course that can be used to lead students in the direction of interesting analyses.

21 Initially, the instructor's time required to supervise and manage the projects was a substantial obstacle. Teaching this course for one semester makes up 20–25% of the instructor's burden for an academic year. The example of Roberts (1991) suggested that better management could lead to substantial reduction in time. In that spirit, students are now asked to e-mail the instructor with a brief proposal for each project. In this proposal, they describe their interest in the datasets, list one or two research questions they hope to answer, and provide some initial data and suggested analyses. The fast turnaround time for e-mail proposals allows for a feedback flow, where the instructor can catch unsuitable datasets or inappropriate statistical techniques quickly, and encourage students to try out ideas. With a class size of 20–25 students, the instructor typically spends 15–20 minutes a day reading and responding to e-mail about the projects for the course. Often, responses are posted to all students on the course World Wide Web page, and the instructor is starting to incorporate some past comments into the syllabus material on the projects. This setup has essentially eliminated project disasters, at the (possible) expense of the instructor being a little more forceful in specifying appropriate limits on data collection and analyses.

22 For each project, students present a final report in two parts: first, a non-technical summary (not to exceed two pages) of assumptions, analysis, and conclusions; and second, a statistical paper (generally fairly modest in length ‐‐ in the neighborhood of seven pages) justifying the analysis and conclusions presented in the summary. Students are asked to include support for any conclusions made in the first part of the paper, as well as a thorough description of each of the variables in the dataset, along with a copy of the data so that the instructor can duplicate and perhaps augment their analyses. The second part of the report explains the analysis to the instructor, and can be filled with as many of the technical details as are required to make points clear. Thus, students address two different audiences.

23 The projects are returned at the next class meeting, peppered with commentary and suggestions from the instructor. Additionally, each student receives a brief summary, prepared by the instructor, of each of the other student projects. Each of the projects is then discussed in some detail, led by the project investigator(s). After the first project, the students receive extra feedback on their writing style and a copy of Ehrenberg (1982). After some discussion in class of this material and of some of the ideas presented in Radke-Sharpe (1991), the quality of the written analyses has improved substantially. This has been well received by several students in end-of-semester course evaluations.

24 In completing their project assignments, working students have usually analyzed data from their firm, often using the firm's statistical software. There is no requirement to use a specific software package, and most students used several over the course of the term. In particular, students have used Statistix, Excel, SPSS, Systat, Statgraphics, Minitab, STORM, PowerPoint, Quattro Pro, Lotus, BMDP, and SAS to obtain and present results. Several students (especially Ph.D. candidates) have stated that they signed up for the course primarily to become familiar with the capabilities of a new statistical package (usually SPSS or SAS), and were generally pleased with their success.

25 A wide variety of topics have been explored by students in the class, and a brief summary of a few of the more interesting projects may prove helpful to teachers interested in developing course materials along these lines. Lists of good past projects seem to be helpful to students looking for topics that might be amenable and appropriate. All of the instructor's courses require a project. In addition to these projects, the lists in Fillebrown (1994), Hunter (1987), Ledolter (1995), and Roberts (1991) have been successful starting points.

26 Students in OPRE 404 seem to have less trouble finding interesting regression project problems on their own than they do finding project topics for the other sections of the course. Many MBA students have already designed a multiple regression model in the context of an introductory course in management of information and decision systems, in addition to completing a regression project in the second semester of the quantitative methods course.

27 A particularly interesting regression project from the Fall 1995 version of the OPRE 404 course examined a proprietary chemical application, with nine independent variables, several dependent variables of interest (in particular, current) and a subset of 937 observations from a 12,000 observation database. The conclusions were somewhat surprising to the investigator, and led to a renewed focus on experimental design within the firm, and a series of follow-up reports by the student over the course of the term.

28 Another regression project, done by a student in Accounting, studied the nature of the relationships among a firm's property, plant, and equipment expenditures over a two-year cycle, expecting that a firm's profitability, age of assets, and ability to make capital expenditures would all be tied in to the fixed asset expenditures made in the following year.

29 In an interesting experimental design project, a sample of 150 toys from the J. C. Penney 1995 Christmas Catalog was studied to see if the price of a toy depended on the age group or gender (0 = either, 1 = female, 2 = male) for which the toy was intended. Data were also collected on the section of the catalog where the toy appeared. A two-way ANOVA studying price as a function of age and gender revealed no clear interaction, and indicated that age alone was a significant factor.

30 A group of students studied a very interesting dataset that was published in the August 20, 1995, issue of the Cleveland Plain-Dealer on Northeast Ohio's public school districts. They looked primarily at the effect of the wealth of the community (taken at two levels ‐‐ median family income below or above 25,000 dollars) and the percentage of parents who attended college (at three levels ‐‐ <40%, 40–60%,> 60%) on the quality points given the school system by the newspaper.

31 Another group studied a wide array of packaged foods (in total, 59) in three categories: cookies, crackers, and salad dressings. In particular, they investigated the idea that the fat and sodium levels of these items would be related to their prices and dietary sugar and carbohydrate levels. The foods were divided into non-fat (0 g of fat), low-fat (according to package labels), or full-fat, and also into lower and higher (more than 10% of U.S. RDA) sodium levels.

32 Some of the students adapted statistical tools to problems in operations management. One student studied total throughput in a manufacturing system in light of the shift time (three 8-hour shifts), as well as the number of packing machines, the number of people working the shift, and the number of supervisors available. The shift factor did seem to have an effect on total throughput, which was primarily explained by the larger numbers of people working on the first shift as opposed to the other two.

33 A more theoretically minded student developed a series of simulations to test the impact of changing arrival time distributions in a first come, first serve queue on the resulting steady state service times in the queue (on average). Exponential arrivals appeared to result in longer service lengths than similarly centered uniform and gamma arrivals.

34 For time series analysis projects, several students looked at data from local firms. One analyzed data from a local high-end sunglasses manufacturer on monthly sales figures (in thousands of dollars) for a three-year period, both to measure the success of the corporation and to assess staffing needs for production of the sunglasses. Another student analyzed the monthly sales from January 1993 through October 1995 of an automotive after-market rear-view mirror adhesive, which turned out to be a highly seasonal product.

35 Two students with interests in finance studied the daily prices of Reebok stock, in an attempt to assess the effect of a series of independent variables. These included economic measures, stock market activity figures, and company earnings. The main results were blasted into space by a series of enormous and especially relentless collinearity problems, and the class suggested a series of exotic transformations of the data which, unfortunately, failed to solve the problem.

36 A student working at the Federal Reserve Bank (FRB) made several attempts to predict the demand M2, also known as velocity (broadly, the money supply), to determine the effects of money demand on the interest rate, controlled by the FRB. The best choice, according to several schemes of model selection, used board opportunity cost, nominal Gross Domestic Product, personal consumption expenditure, effective return on M2, and level of thrift deposits.

37 Three students looked at data from student course evaluation forms for courses in five departments at the Weatherhead School during the Fall 1994 semester. They studied which questions on the form best predicted the overall course and instructor ratings, differences between departments and courses, and the effect of type of instructor (lecturer, Ph.D. candidate, assistant professor, associate professor, or professor) on the ratings.

38 Thanks to the success of the Indians and the disappearance of the artists formerly known as the Cleveland Browns, baseball dominates the conversation of Cleveland sports fans these days, and two students spent a good part of the semester with three different datasets involving baseball players, teams, and their salaries for the 1987 season. This project sparked a good deal of conversation in class, and led the instructor to bring in papers by Lackritz (1990) and, eventually, Hoaglin and Velleman (1995) on the analysis of baseball salaries. Eventually, the class collected more current data on several teams in an attempt to provide a financial justification for the Cleveland Indians' recent successes. The students found the Hoaglin and Velleman paper extremely interesting because it discussed several different analyses of a similar dataset from the 1988 American Statistical Association Data Analysis Exposition, some of which were closely related to the approaches we had developed in class. The students had reached many of the same conclusions on their own, and were very pleased to see their ideas and approaches for tackling the data justified in this way. This project led the students to the World Wide Web and other data sources, and encouraged them to get more involved in studying the underlying issues, particularly in model selection and diagnostics.

5 Compromises in a Project-Driven Course

39 On the whole, the course seems to be a substantial success. The course is being given with increasing frequency, and enrollments remain around 25. There are, naturally, several problems with such a course. First, not surprisingly, is the time commitment for the instructor. Initially, finding interesting and appropriate real datasets was very time-consuming, as was grading projects and investigations. Unfortunately, the instructor did not read Singer and Willett's (1990) suggestions until the middle of the first pass through the course, but their comments and annotated bibliography have proven very helpful in the subsequent development of material.

40 A second problem is the lack of a text at the appropriate level covering all of this material. Recent moves toward designer editions of large texts will help, but there is no clear choice of text for a course in regression diagnostics and model selection, analysis of variance, and analysis of covariance, along with simple factorial and fractional factorial designs, time series analysis using ARMA and exponential smoothing, measures of association and log-linear models, as well as logistic regression. The Statistix computer package has been very effective for many students, providing neat and clear output, and a reasonably well-written text explaining procedures with examples. This choice was motivated in part by the use of Statistix in the casebook written by Chatterjee, Handcock and Simonoff (1995), which has been effectively used as a review text in regression by a few of our better students preparing to take the OPRE 404 course after a bit of a layoff from statistics.

41 From the World Wide Web, the instructor has made substantial use of the Data and Story Library [lib.stat.cmu.edu/DASL/], the Chance database [www.geom.umn.edu/docs/snell/chance/welcome.html] and the Journal of Statistics Education Teaching Bits pages. The UCLA Department of Statistics page [www.stat.ucla.edu] and the general StatLib material at Carnegie-Mellon University [lib.stat.cmu.edu] have also been extremely helpful. The Internet has been a substantial source for data collection and reference material for students in the course. Of course, this has introduced several additional problems. First, it is often difficult for students to find the raw data so crucial for projects. Too often, a site provides sufficient statistics, or the results of an analysis, rather than the original collected information. Also, many datasets are available with little or no background, making it difficult for students to judge the quality of the underlying study or experiment. Sites disappear in mid-semester. Others fail to give sufficient instruction regarding credit for the posted work, leaving students in the unenviable position of referencing anonymous sources in their project reports.

42 In any course driven by projects, feedback and guidance from the instructor is critical. Roberts (1991) was especially helpful here, as was Sevin's (1995) discussion of assessment. This was a very time-consuming part of the endeavor for the instructor and still requires 40–45 minutes per group project, but many students commented on the value of the feedback through the term and on the course evaluations. As the course is given for the fourth time, the instructor has been able to cut the time requirements a bit. A teaching assistant now grades the investigations of “canned” data sets (but not the projects). Also, a fairly large group of old projects and datasets is now available, cutting the time required to prepare new material tailored to class interests. After the first three runs of the course, it seems clear that at least 60% of the case studies in the bulkpack are of sufficiently general interest that there is no reason to modify them each semester. For what remains, the development of new course material tailored to student interest requires careful attention to opportunities currently in the news. The instructor has also made heavy use of suggestions from several course veterans now out in the field, who return occasionally with a new situation or batch of data that can be tailored to the new group of students.

43 The different levels of students' oral and written English skills have always been a problem. Some students dominate the conversation in class, although there have been some signs of improvement. With a group of 20–25 students, it is possible to include all of the students at least occasionally, but oral feedback has not been equally distributed. All of the students improved their written communication skills over the course of the term. Some students would like a textbook that they could read outside of class in order to be better prepared for class discussions.

6 Summary

44 Seventeen students enrolled in the first class. This group included an MBA student, three undergraduates, and three additional students from outside the Department. All of the students did well in the course, and most showed very substantial improvement in their work over the course of the semester. The instructor found the course to be extremely enjoyable, though the preparation for each week's session was on the order of 15 hours per week. In subsequent runs of the course (with 20–28 students), the overall instructor time spent on the course has averaged 8–10 hours per week, including two hours spent answering e-mail and monitoring the course Web site, two hours spent in class, and four office hours. Grading and preparing feedback for the projects requires about 10 additional hours four times each semester.

45 Student ratings from the end-of-semester course evaluations have been extremely high (the worst mean course rating was 4.53 on a 5 point scale for the three completed sessions), and comments have also been very positive, primarily focusing on the benefits of the projects and experiential exercises, and also the value of the investigation discussions in class. As noted above, several students remarked on their improved technical writing skills. The remaining student complaints involve hardware and software problems in our computer lab, which has undergone substantial turnover.

46 Enrollment each time has been higher and more diverse than in the first attempt. The instructor expects to be able to serve at most 30 students successfully in this format, so some changes may be required as the course becomes a more popular MBA elective.

47 Finally, with the OPRE 404 course as prerequisite, the Department of Operations Research and Operations Management has resurrected two additional advanced statistics courses, one in experimental design and one in time series analysis, which have now been taught at a substantially more interesting and effective level in light of the existence of a more dependable level of applied understanding.

Acknowledgments

An earlier draft of some of the material in this paper was presented to the Section on Statistical Education of the American Statistical Association at the 1996 Joint Statistical Meetings in Chicago. The author wishes to thank the editor and three anonymous referees for their comments which improved the presentation of this paper.