Using Intended Learning Objectives to Assess Curriculum Materials: the UCGIS Body of Knowledge

Abstract

The first edition of the UCGIS Body of Knowledge (BOK) is a recommended baseline for the geographic information science and technology (GIS&T) undergraduate curriculum. Part of a larger seven-year effort to rebuild the top of the world's GIS&T intellectual pyramid, it attempts to define the knowledge base that one might expect students receiving an undergraduate specialty in the field should be exposed to. Although it is not a set of lecture notes, exercises or laboratories for educators, it does provide over 1600 learning objectives spread over 10 general knowledge areas. Because the document is designed to serve the needs of the educational community it is important to know whether these learning objectives are set at appropriate cognitive learning levels. This paper examines the range of cognitive levels suggested by the learning objectives enumerated for each of the 10 Knowledge Areas. By using text-based content analysis of the BOK learning objectives, the author compares the action verbs associated with Bloom's cognitive levels to those in the learning objectives. Results show a surprisingly consistent distribution of requisite cognitive levels for the learning objectives, with distributional modes for most Knowledge Area averages falling between Bloom's levels 2 and 3 (knowledge and comprehension, respectively). The single exception to this pattern is the Design Aspects (DA) knowledge area where the frequency of action verbs is normally distributed, with the mode falling just between Bloom's levels 3 and 4. The underlying premise of the curriculum project is that students graduating from a GIS&T programme should be able to identify and solve spatial temporal problems; in short that they should demonstrate critical thinking skills commonly associated with Bloom's levels 4, 5 and 6. This suggests that there is a need to increase the number of learning objectives requiring analysis, synthesis and evaluation. These results further indicate that the existing learning objectives, while amenable to use in developing course materials and evaluative rubrics for GIS&T coursework, materials and rubrics focusing on critical thinking will need carefully to be selected to include higher cognitive level objectives. Given that the BOK is meant to evolve as the discipline's foci and directions change, the learning objectives and content of the document will necessarily have to change. This will require continued dialogue between the educators and those responsible for revising the BOK document.

Keywords:

• GIS
• Body of Knowledge
• educational goals
• intended learning objectives

Introduction

In 2006 the first edition of what is hoped will be a continuing series of geographic information science and technology body of knowledge (GIS&T BOK) documents was published ((DiBiase et al., 2006). It was a direct product of a larger effort by professionals from industry, education and government to examine the state-of-the-art in geographic information science, to rebuild the intellectual underpinnings of the discipline (Marble, 1998, 1999), and to create a model undergraduate curriculum not unlike that already in place in computer science (Gorgone et al., 2002). Early on this effort was funded to various degrees by the Environmental Systems Research Institute, GE Small World, and the Intergraph Corporation to allow the process to move forward. Eventually the working group's activities were acknowledged by the University Consortium for Geographic Information Science (UCGIS) and are now endorsed by the educational component of that organization.

The BOK is a reflection of the task force's underlying objectives to define the current state of the body of knowledge, reduce the recognized shortage of well-educated GIS&T personnel, and correct the observed mismatch between the educational process and industry needs. It is meant to be a living document, requiring substantial comment from interested parties at frequent intervals (Johnson et al., 2006), and is based on six basic guiding principles (UCGIS, 2003). These are (1) to define a curriculum core that (2) incorporates disciplinary diversity, (3) is adaptable to different institutional missions and (4) to content change, (5) includes the institutional context of GIS&T, and (6) focuses on spatio-temporal problem identification and analysis.

The publication of the BOK shows that the first of these principles has been met: the curriculum core has been defined. The core has been presented to a variety of organizations, with mixed reviews, suggesting that there will be a need for revision in future editions. This reflects principle four above—that of adaptability to content change. The purpose of this article, however, is not to examine the BOK for completeness, but rather to examine the relationship between the core in its first edition and its ability to be translated into the classroom. In particular, I examine principle 6, the focus on spatio-temporal problem identification and analysis. The implications of this principle for education are that the student who successfully completes a GIS&T programme should not only be able to demonstrate factual knowledge (Bloom's level 1), and illustrate an understanding of its basic principles (Bloom's level 2), but also be able effectively to employ them (Bloom's level 3). This employment is characterized by a demonstrable ability of the student to identify and solve realistic, moderately complex spatio-temporal problems using the available geospatial technology (Bloom's level 4). One might hope that higher levels of thought, specifically those related to innovation and creation (Bloom's level 5) and evaluation (Bloom's level 6), would also be an outcome for at least some components of the existing knowledge areas. It is, however, not likely that many undergraduates will achieve this level without experience beyond the university setting.

Problem Statement

Among the things that have yet to be completed for the model curriculum is a determination of the relative importance that each knowledge area should have for appropriate job classifications and for different educational settings. While the guiding principles of the model curriculum development process suggest that a general minimum standard might be expected to be at or near a use or application level, the task force has not as yet determined whether this should be the same for all of the BOK's 10 knowledge areas:

1.	AM: Analytical Methods;
2.	CF: Conceptual Foundations;
3.	CV: Cartography and Visualization;
4.	DA: Design Aspects;
5.	DM: Data Modelling;
6.	DN: Data Manipulation;
7.	GC: Geocomputation;
8.	GD: Geospatial Data;
9.	GS: GIS&T and Society;
10.	OI: Organizational and Institutional Considerations.

Each of these 10 Knowledge Areas is further subdivided into more explicit units. For example, the BOK document breaks Knowledge Area DN (Data Manipulation) into three units as follows:

•	DN1: Representation and Transformation;
•	DN2: Generalization and Aggregation;
•	DN3: Transaction Management of Geospatial Data.

Each of these units is further decomposed into topics and it is at this level that the BOK provides associated learning objectives. These give some indication of the performance standards for each topic. While the task force had originally considered the possibility of enumerating the performance level that might be expected of an undergraduate student for each of the 12 topic areas and their unit, the process was abandoned in favour of first providing a baseline body of knowledge linked to learning objectives.

Unfortunately, until the existing levels as suggested within the current document are quantified it is difficult to make the necessary adjustments to decide the appropriate level for each topic later on. This is particularly important given that the overarching guiding principle of this work has been to adjust to a diversity of disciplinary, mission and institutional contexts. My research attempts to fill this gap by using a relatively rigorous, yet well-established educational standard upon which to make this evaluation.

The standard to which I refer is called Bloom's Taxonomy of Cognitive levels (Bloom, 1956). The taxonomy was designed to provide a common language for assessing learning goals, with specific meaning for broader curricular goals that allows objectives at any level to be articulated (Almer, 2006). In their seminal work on teaching geography in higher education, Gold et al. (1991) adopted Bloom's Taxonomy as a standard for university geography education, but correctly indicated that curricula should not be so tightly linked to this philosophical model as to overemphasize conformity and correctness over expressive objectives (Eisner, 1975), particularly those that allow a degree of creativity and individual expression. While such expressive objectives have merit for the encouragement of creativity, there is yet to emerge a formal method of assessing such objectives, thus limiting one's ability to determine the degree to which any curriculum addresses these important aspects of education.

Beyond its focus on correctness and conformity, Bloom's Taxonomy also has been criticized recently because of its focus on critical thinking over acquisition of factual knowledge (Booker, 2007). While this criticism has some merit, it seems mostly a function of the misapplication of the taxonomy rather than the taxonomy itself. The purpose of this research, then, is not to suggest that one might necessarily emphasize one level of learning over another but rather to suggest that there be a necessary mix of the cognitive domains of the taxonomy and merely to enumerate what those levels are within the current GIS&T Body of Knowledge learning objectives. More specifically, the goal is to determine the degree to which the BOK satisfies its sixth objective—that of emphasizing spatio-temporal problem identification and analysis. Thus it is necessary to compare the BOK learning objectives quantitatively against an established set of cognitive levels such as those enumerated by Bloom (1956).

There are generally acknowledged to be six cognitive domain levels based on Bloom's original work although some educators have employed seven. The levels are as follows:

1.	Knowledge—memorization;
2.	Comprehension—grasping meaning;
3.	Application—adapting knowledge;
4.	Analysis—decomposing into component parts;
5.	Synthesis—recomposing to new forms;
6.	Evaluation—judging value for a purpose.

Each cognitive level has associated with it a list of (mostly) unique ‘action’ verbs that allow educators to relate specific behavioural objectives to these cognitive levels (McBeath, 1992). While not exhaustive, the vast majority of alternative action verbs are often synonymous with the existing McBeath action verbs. With these action verbs educators can formulate specific evaluative environments—whether testing or performance related—that are appropriate to both the cognitive level and to its concomitant demonstrable action. Thus, the appropriateness of learning levels as determined by the action verbs within the learning objectives can be quantified.

Methods

I examined all the suggested learning objectives for each of the 10 Knowledge Areas in the BOK. Using the electronic text version of the learning objectives I employed Textstat© (Hüning, 2000–2006) to count all the action verbs found at the beginning of each learning objective. Because variations in sentence structure result in the occasional action verb occurring in the body of a learning objective, these were captured by subsequent examination of the text. This process also accounted for redundancies often associated with sentence constructs such as ‘compare and contrast’ that might result in inflated action verb numbers to be identified. After identifying the action verbs and the context within which they are embedded I compared them with those suggested within Bloom's taxonomy and enumerated by McBeath (1992). For example, under the first unit breakdown area for Knowledge Area one (Analytical methods: AM-1 Academic Foundations) the first learning objective states that a student should be able to: “Differentiate between confirmatory and explanatory geospatial data analysis”. The operational verb for this learning objective is ‘differentiate’ which occurs at Bloom's level 4, indicating analysis which is a ‘critical thinking’ level. This level shows that the student should not only be capable of defining the terms confirmatory and explanatory geospatial analysis, but also capable of knowing the appropriate conditions relevant to performing each type of analysis. It might also assume that the student is capable of performing these types of analysis and explaining the results. For this learning objective I recorded a Bloom's level value of 4. I performed this same procedure for each of the five learning objectives for AM-1, and continued with the remaining 11 units of the Knowledge Area. I then tabulated the number of action verbs for each of the six Bloom's levels and calculated the group mean Bloom's level for the entire Knowledge Area. I continued this process for each of the remaining nine Knowledge Areas.

Because the Knowledge Areas have variable numbers of units, topics and individual learning objectives their tabulations were normalized by converting the verb frequencies by Bloom's level to percentages. I compiled a single weighted mean frequency Bloom's level for each Knowledge Area and tabulated the results. Additionally, I plotted the results of the percentage means for each of the six Bloom's levels for all 10 of the Knowledge Areas to allow examination of internal variation among Knowledge Areas.

Results

Figure 1 summarizes the results. While there is a wide range in the numbers of units and topics for each Knowledge Area and the numbers of learning objectives, all Knowledge Areas demonstrated a range of action verbs at Bloom's levels between 1 and 6. The range of overall average Bloom's levels for all Knowledge Areas was between a low of 2.807 (Knowledge Area AM: Analytical Methods) and a high of 3.641 (Knowledge Area DA: Design Aspects). The range gets considerably tighter (2.807 to 3.015) if the Design Aspects value is eliminated. This suggests a surprising level of consistency for all but the Design Aspects Knowledge Area.

Figure 1 Frequency distribution of average bloom's level by knowledge area.

This similarity is even more obvious if one compares the frequency distribution of action verb levels for all 10 Knowledge Areas as in Figure 2. All Knowledge Areas, except Design Aspects (DA), show significantly skewed distributions where the modal value falls clearly in Bloom's level 2 and the skew is to the right toward higher cognitive levels. This shows that the learning objectives for these nine Knowledge Areas are at the comprehension level. The Design Aspects (DA) distribution is clearly normally distributed and has a modal value of 3.5, or somewhere between application and analysis. This indicates that the learning objectives enumerated for this Knowledge Area are the only ones requiring significant critical thinking skills.

Figure 2 Comparison of frequency curves of knowledge areas showing skew toward lower learning levels.

Conclusions

GIS education has seen a significant growth in both numbers of programmes offering GIS coursework and in the variety of courses offered. This expansion begs a revisiting of the review of GIS education performed by a special issue of the Journal of Geography (Baker & Bednarz, 2003) which echo Downs's (1994) view of the desperate need for research in geography education. Most importantly it indicates the requirement for empirical data regarding cognitive levels, learning objectives and course content. The results of this study provide a first approximation of the recommendations of literally dozens of GIS educators as compiled in the GIS&T Body of Knowledge (DiBiase et al., 2006).

Results indicate that, in most cases, the BOK is defined mainly at the comprehension or applications level of knowledge (Bloom's Levels 2 or 3) for nine of the 10 Knowledge Areas. Design Aspects (DA) is the only Knowledge Area for which the average is at a level commensurate with significant critical thinking. Considering the stated objective of the curriculum development effort to produce students capable of identifying and solving reasonably complex problems, it might be expected that it would ask for cognitive levels somewhere between 3 (applications) and 4 (analysis). It would seem that only the Design Aspects (DA) learning objectives with their average of 3.5 consistently approach this level. The remainder of the Knowledge Area cognitive levels have averages at or just below Bloom's level 3, suggesting that, although they satisfy the stated goal of applicability, there is an overabundance of cognitive levels at level 2 (comprehension). This suggests that the existing BOK's learning objectives are marginally below those suggested in the overall goals of the original strawman document which were to develop and solve reasonably difficult problems using GIS technology.

These results suggest that the translation of the existing learning objectives to course materials and grading rubrics will require careful selection to identify a higher percentage of learning objectives requiring cognitive levels at level 4 or above than those published in the document. The distributional consistency of nine of the 10 Knowledge Areas suggests that, regardless of the recommended cognitive levels in the original strawman document, this might be the appropriate level. This is a topic for future examination.

It is highly unlikely that any undergraduate curriculum will result in students achieving Bloom's level 6 for more than a minimum of topics within the Knowledge Areas. Results from this tabulation clearly demonstrate that this is only suggested for a very few factors. Based on this initial analysis of action verbs associated with Bloom's taxonomic levels it appears that the model curriculum as exemplified by the existing BOK are slightly below the guiding principle of focusing on spatial temporal problem identification and solution. Given industry's demand for this ability, it is likely that educators will need to consider selecting higher cognitive level objectives and/or supplementing the existing ones within their own courses. While the cognitive rigor of the BOK learning objectives is slightly less than recommended, the sheer volume of learning objectives provides an ample set from which coursework and evaluation mechanisms can be developed.

Discussion

While this initial quantification of learning objectives suggests a lack of success of the BOK in achieving the guiding principle of spatial temporal problem identification and solution, there is room for discrepancy. The action verbs used in this analysis require interpretation by a human and thus do not allow for exact re-creation of the study. Although every effort was made to be absolutely objective, contextual evaluation of non-designated action verbs is less than precise. While some investigator variation is expected, it is likely that the general results would remain the same with only slight variation. Some of the learning objectives are also less explicit than others and either incorrectly apply action verbs or employ more than one action verb in each learning objective. I have attempted to account for both conditions through contextual analysis.

Some learning objectives are quite numerous while others are few in number. This suggests a common use of terminology, such as ‘discuss’ and ‘explain’, where more precise terminology might actually suggest a higher or lower cognitive level. One reason for this is that no standardized set of action terms was provided to the writers during the preparation of the BOK. Finally, the learning objectives often reflect the personal interests of the team members who suggested them, their individual backgrounds, institutional contexts, levels of experience, and the nature of their audience.

In short, the results of this initial review of learning objectives should be viewed with a healthy level of cynicism. More importantly, the document should be examined by as many individuals as possible. Both the learning objectives and the Bloom's levels need to be re-examined as well. Even the suggested minimum cognitive level needs to be re-examined to determine whether or not it is justified.

As suggested in an early review of the learning objectives (DeMers, 2004), it might prove useful in future iterations of learning objective development that each should have a single, explicit action verb linked to an appropriate demonstrable product. It might further be recommended that a set of McBeath (1992) action verbs be employed during their development so subsequent evaluation can be effective. Regardless of its current condition, the BOK will require constant revision by both professionals and educators. It would also prove useful to systematically query potential employers to ascertain what skills and cognitive levels they are anticipating from GIS&T graduates. This will determine the validity of the minimum standard against which the learning objectives will be compared. Finally, a necessary next step in GIS education evaluation would be an examination of the degree to which the BOK, as it evolves, contributes to students' psychomotor, affective and even expressive learning.