Assessing twenty‐first century skills through a teacher created video game for high school biology students

Abstract

As twenty‐first century skills become a greater focus in K‐12 education, an infusion of technology that meets the needs of today’s students is paramount. This study looks at the design and creation of a Multiplayer Educational Gaming Application (MEGA) for high school biology students. The quasi‐experimental, qualitative design assessed the twenty‐first century skills of digital age literacy, inventive thinking, high productivity, and effective communication techniques of the students exposed to a MEGA. Three factors, as they pertained to these skills, emerged from classroom observations. Interaction with the teacher, discussion with peers, and engagement/time‐on‐task while playing the MEGA suggested that students playing an educational video game exhibited all of the projected twenty‐first century skills while being engrossed in the embedded science content.

Keywords:

• biology
• genetics
• video games
• twenty‐first century skills

Introduction

In the economic environment of the new millennium, education plays a critical role in maintaining prosperity and stimulating economic growth (Stevens and Weale 2003). How we teach and train our high school students will directly correlate with the future success of the country’s economic growth and power. The notion of teaching high school students twenty‐first century skills is quickly becoming a critical component of school curricula. The question still remains as to what twenty‐first century skills are and how we might teach them.

One thing is clear: today’s teenagers play video games in large numbers. In 2005, the Federation of American Scientists held a meeting billed as the Summit on Educational Games. The crux of this meeting was to find a solution to the aforementioned question. They concluded that teenagers are spending more time playing video games than on their academic subjects (Trotter 2005). Oblinger and Oblinger (2005) agreed with Trotter by characterizing today’s teenagers as the ‘Net Generation’. They suggest the following needs that the Net Generation innately desires: they like to be connected; they desire immediate responses; they learn through experiential practice; and they want social interaction.

Purpose of the study

Games themselves have been an integral part of teacher pedagogy for decades. Social or solitary, simple or complex, collaborative or competitive, games give us an opportunity to exercise the sense of playing to learn. Children are able to learn games before they can talk (for example, ‘peek‐a‐boo!’) and continue to expand their gaming repertoire as they mature. Games are engaging and adaptable to almost any subject. They are particularly useful for teaching cause‐and‐effect relationships, and the lessons learned from games stay with students because of the interactive, immersive nature of the learning experience (The New Media Consortium [NMC] 2005). While technology is by no means necessary for educational games, games and technology are increasingly being combined in interesting ways (NMC 2005).

This study sought to find out how video games might be used in educational settings and if students learn twenty‐first century skills from the use of game playing as a teaching and learning tool. As part of the HI FIVES (Highly Interactive Fun Internet Virtual Environments in Science) project, students were observed during an exposure to a teacher‐created MEGA (Multiplayer Educational Gaming Application) (Annetta et al. 2006).

The question of this study was: do students exposed to a MEGA as an instructional method learn the critical twenty‐first century skills of interaction with the teacher, discussion with peers, and engagement/time‐on‐task while playing the MEGA?

Literature review

Twenty‐first century skills

For a region, state, or nation to gain a competitive advantage, the workforce needs to possess high technical skills. Critical to that competitive advantage are the education and skills adults acquire in primary and secondary schools. The workers of the twenty‐first century must have science and mathematics skills, creativity, information and communication technologies (ICT) skills, and the ability to solve complex problems (Business Higher Education Forum 2005). In the US context, one part of a comprehensive solution to this challenge is to increase the number of scientists, engineers and technicians produced and retained (Committee on Science, Education, and Public Policy 2006). To accomplish this goal, education must adhere to the notion that if we fail to plan, we plan to fail (Spires 2008). A primary challenge for US education is to transform children’s learning processes in and out of school and to engage student interest in gaining twenty‐first century skills and knowledge. Lemke (2003) reported a link between twenty‐first century skills and academic achievement making the case for incorporating teaching activities that adhere to twenty‐first century skills. Education must also align curriculum and learning to a whole new economic model. Linking economic development, educational evolution, workforce development, and strengthened social services is essential to meeting this challenge (Dede et al. 2005).

Rapid advances in information technologies are reshaping the learning styles of many students of the Net Generation. These students have grown up in a world where technology is second nature to them and to those around them. Dede (2005) reported some emerging learning styles by these students as: fluency in multiple media and simulation based virtual settings; communal learning; balancing experiential learning; guided mentoring and collective reflection; non‐linear representations and co‐designing learning experiences that are personalized to individual needs and preferences. Because of these new learning styles, it is crucial that teachers adapt their teaching styles accordingly.

However, as with any other education movement, there are impediments to the completion of this task. Wheeler (2004) detailed that there are not enough tools for teachers and students, not enough insight for integration, and not enough national commitment for such an extreme shift in teaching and learning practices. Students continue to learn in a twentieth century environment but they are very fascinated by twenty‐first century technology. Teacher quality and preparation are critical for the integration of science and technology into the classroom.

Wheeler (2004) continued by suggesting that the way to reduce the gap between developing software products for business and education is to have teachers who understand how to utilize advanced technologies to enhance teaching and learning, and thus develop new tools and applications to achieve this goal. Teachers are highly motivated to use new teaching tools if they think they will engage students (Annetta et al. 2006). A national commitment is needed for this to occur.

Learning through video games

It has long been accepted in the field of education that knowledge must be constructed if it is to be properly assimilated. The assessments of learning should address the skills of manipulating, applying, analyzing, synthesizing, and evaluating knowledge (Wang 2004). The design of a video game can dictate the assessment design. A well‐designed technology program must be based on sound pedagogy. Gaming environments encourage students to explore beyond the boundaries of given material, thus allowing for a proactive and exploratory nature, training the student to become a self‐reliant learner (Taradi et al. 2005). Gaming environments facilitate experiential learning by allowing the student, regardless of gender, to explore the confines and components of the environment at their own pace in real time (Annetta et al. in press). Due to their 3D nature, video games also have great potential for demonstrating complex abstract concepts, for example, representing a magnetic field (Dede 1995) and adhere to the laws of physics (Dede, Salzman, and Loftin 1999).

Not all games are mindless entertainment. In education especially, there is room for games where the goal is to solve a problem cooperatively, and everyone can win. If the outcome of a game is not to have a single winner, but to have a group derive a perfect solution to a problem, more than one group may achieve the game’s goals. Thus the thrust becomes problem‐solving and working together rather than winning or defeating opponents (NMC 2005).

Educational games need to be designed on the tenets of problem‐based learning (Annetta, Cook, and Schultz 2007). One way to produce more meaningful educational games would be to design games in which players are engaged in richer, more meaningful practices (Squire 2002). Referencing research in anchored instruction and problem‐based learning environments, Bransford and Schwartz (2001) found that students perform best when given access to lectures in the context of completing open‐ended complex problem‐solving tasks. More studies like these are critical if we are to use video games in education and provide the Net Generation with the skills they need to be successful in the twenty‐first century.

Methods

Setting of the study

The MEGA was created in Activeworlds, a 3D virtual environment originally created for social networking. The HI FIVES project staff manipulated the Software Development Kit (SDK) to make it amenable to game play. The SDK allowed project staff to embed assessments and build a back end data collection mechanism. The Activeworlds interface became the pilot for a structured Graphical User Interface (GUI) using a commercial game engine. The goal of the HI FIVES project is to instruct science and mathematics teachers in Grades 5–9 on game theory, game design, and how to effectively integrate video games into the curriculum. This study was positioned as a feasibility study to see if teachers were willing and able to learn how to create video games and to investigate the feasibility of integrating video games into a science class.

Multiplayer Educational Gaming Application (MEGA) design

The teacher who created this MEGA was a part of a game design project where she learned to identify game goals, create a back story, story board and integrate formative assessments within the game design through a problem‐based approach. Further the Activeworlds interface allowed for the teacher design to incorporate web pages into the MEGA which drove the storyline and allowed the students to understand the goals of the game while providing scaffolded instruction as to when to use Punnett squares and pedigrees (a major thrust of the teaching unit) to attain the game’s goals. The MEGA, entitled The stolen fortune of I.M. Megabucks, was set in a multiplayer environment where students had to solve a murder case using skills of science and scientific inquiry. Specifically, students were asked to solve a mystery using their understanding of pedigrees, Mendelian inheritance, blood types, and DNA fingerprinting – these being goals in the high school genetics curriculum. The background story of the game became a Crime Scene Investigation (CSI) initiative that was chosen to engage students based on student reports about the popularity of the CSI television series. An executive summary of the background story that introduced the students to the MEGA goals follows:

Mr and Mrs I.M. Megabucks have recently died, leaving a large inheritance to a number of relatives. They passed away tragically when they were struck by lightning while climbing up a playground’s metal slide during a storm. When everyone showed up for the reading of the will, the inheritance was stolen out of the safe. Only family members had access to the safe’s combination. There was a small amount of blood left at the crime scene. The only other clue was that the butler witnessed the crime. He observed a figure wearing a ski mask running out of the room carrying a large bag (presumably full of the inheritance). As the masked burglar rushed by the butler, the burglar gave the ‘thumb’s up’ sign. The butler noticed that the burglar had a bent thumb.

Students had to go to the lab and analyze the clues to solve the mystery (see Figure 1).

Figure 1 Screen shot from the CSI lab inside The stolen fortune of I.M. Megabucks.

Sample

This quasi‐experimental design incorporated a qualitative data collection mechanism. The MEGA served as a review of the genetics unit following the teacher’s standard instructional methodologies of lecture (didactic/teacher‐centred) and laboratory (group/collaborative/student‐centred). The students in this study were academic level (i.e., college prep but not on the advanced placement track). The students that participated in this study were working towards a subsequent university‐based education but were not amongst the highest ability students in their cohort. The participants comprised biology students from four different classes taught by the same teacher. The students were not formally trained on using the interface but quickly adapted to using it after basic instructions from the teacher. The sample for this study consisted of 131 (70 males and 61 females) high school biology students of similar ability (based on the track in which the US school system placed these students). In the US, high school classes are often stratified by ability as was demonstrated in previous classes. For example, the honours track consists of students in the upper tier of their age group and are more likely to go to college. The academic track is the second tier of students who may be college‐bound but are often more likely not to pursue university coursework upon graduating from high school. In this study all students were on the academic track. There was also a variation in the amount of time these students reported playing video games for entertainment. The majority of students said they played video games at least 10 hours per week while some students (three) said they don’t play video games at all.

Data collection and analysis

The game was used for reinforcement and review at the end of an instructional unit on genetics. The research question on twenty‐first century skills was assessed using qualitative measures of classroom observation and was coded by three researchers. All classes were video recorded for further analysis. Recorded events from the three observations were compared to ensure inter‐rater reliability. Reliability was calculated at an alpha of 0.78, which was deemed an acceptable level of agreement. The targeted twenty‐first century skills of digital age literacy, inventive thinking, high productivity, and effective communication were derived from the Lemke (2003) report from NCREL/Metiri Group (see Figure 2) and operationally defined as such.

Figure 2 Link between twenty‐first century skills and academic achievement (the videos were coded with tallies when the targeted twenty‐first century skills were observed). Reproduced with kind permission of the Metiri Group, © 2009.

The observations were recorded every three minutes of the 90‐minute classes from the time the teacher started the class until the class ended. The engagement of twenty‐first century skills was operationally defined by the Lemke (2003); please see Appendix A.

One researcher took field notes while viewing the classes. From these data, three themes (student interactions with the teacher, discussion with peers, and engagement/time‐on‐task while playing the MEGA) arose.

Findings

This section will first summarize the field notes recorded by an observer and will be followed by a summary of the twenty‐first century skills observed. It is important to note that the observer recording field notes was also one of the three researchers who recorded twenty‐first century skills.

Students understood the background story and began the game by exploring the Megabucks estate. Without instruction to do so, a leader in each class arose and the students worked together by dividing the estate and collecting clues in their respective areas. By clicking on characters and/or objects in the MEGA, web pages would pop‐up giving the student more insight into the surrounding crime scene. Blood smears on a police car, talking to the deceased Megabucks (who took the form of ghosts in the family graveyard), getting fingerprints off the family safe, and attaining physical descriptions of the possible suspects led the students to create Punnett Squares and delve into the Megabuck pedigree to conclude the who dun it? The clues led them to the mansion where all of the students reconvened and discussed the possible suspects. A line‐up of all possible suspects stood on the second floor of the mansion waiting for the deliberation. If the students chose to arrest a suspect who didn’t match the clues set forth in the game, they were teleported back to the beginning of the game where they were forced to start the process again. This allowed the students to re‐explore the virtual world to see where they might have gone wrong in their decision‐making process and served as negative reinforcement for guessing without obtaining all possible clues and using the knowledge they gained throughout the genetics unit. If they chose the correct suspect, the students were teleported to a police station where the suspect was behind bars and the theme music from the television show Cops played in the background. This concluded the game and the teacher debriefed each student to see how they concluded the suspect was indeed guilty and the techniques used to come to that decision.

Through an embedded text chat device in the Activeworlds™ interface, students communicated in real time rather than talking to one another across the room. For example, when students explored the mansion, the classroom was almost silent with the exception of the tapping of keys on the keyboards. In one class the Internet seemed to slow down and the text chatting was not as rapid as the students would have liked. The students managed this by talking with each other and sending notes around the room. Although the in‐game chat was captured, the information was only analyzed to the degree to which the researchers could ascertain content of communication.

Results of the field notes suggested findings in three themes: interactions with the teacher, interaction with peers, and engagement with the MEGA.

Interaction with the teacher

Students were very engaged through interactions with their teachers while playing the MEGA designed for learning about genetics. The students were eager to raise their hands to ask questions regardless of whether the questions related to technical issues or the scientific problems embedded within the game. Throughout both classes, the teacher continuously answered questions while being certain not to give specific clues to the game’s conclusion. The MEGA provided students with an opportunity to interact with their teacher directly, as well as allowing the students to not only learn science but also to hone their computer skills.

Discussion with peers

Based on the information the video game provided them, students had to finish three tasks: making a pedigree, eliminating suspects, and identifying the culprit of the stolen fortune. In order to achieve these three tasks, students actively discussed the game‐play and science knowledge acquisition with their peers both in‐game (through text chat) and around the classroom (through verbal communication and note passing). The students provided opinions to each other and attempted to piece the crime together as a team.

Engaged in playing MEGA

Most students were fully engaged in the activities. Except for interacting with the teacher or discussing ideas with their peers, students spent the entire time sitting in front of the computer playing the MEGA. After finishing the tasks, the teacher asked, ‘So what do you guys think? Was it fun?’ The students emphatically agreed that it was very much fun. Obviously, for students, MEGAs are very appealing to them, hence they prefer to be patient and spend a whole lesson investigating scientific phenomena within the gaming environment.

As a supplementary teaching and learning tool, MEGAs provide students with an opportunity to review and practice what they have learned in the lecture. Because the video game had a problem‐based storyline, students could practically connect the science knowledge they had learned in the lecture within a practical setting. This is a critical step in the learning process. As the dialogue between teacher (T) and students (S) excerpt from the video suggests, although the student had already learned the concept of pedigrees in the lecture, they were still unclear on how to draw it until they played MEGA.

[Student is drawing a pedigree…]

S: Ms Mary, how would you add a relationship?

T: That’s a good question. So how do you indicate a marriage?

S: Line.

T: What kind of line? What kind of line do you do for this case?

S: A straight line.

T: Is that horizontal or vertical?

S: Horizontal.

T: So a marriage is a horizontal line. So draw a horizontal line. That’s a vertical line.

S: I am sorry.

T: That would mean his daughter was OK. And now who is who married to? George marries…

S: Stellar.

T: OK, Stellar. She is female, so what symbol do you always draw for females?

S: Circles.

T: Right. Do they have any children?

S: Three children.

T: So how do you indicate children?

S: Vertical line.

T: Yep, and they have three. OK. So you did it right! Great job!

S: Thanks. It makes it easier now that I know what each person looks like now and I understand how to connect all of the family members. This is awesome!

Twenty‐first century skills acquisition

The four main areas of twenty‐first century skills set forth by Lemke (2003) were used as a framework for noting student skills illustrated during MEGA play. Digital age literacy was consistently shown throughout both of the classes. Students exemplified more than basic technological skills, as they needed very little, if any, instruction as to how to navigate the MEGA. Their basic scientific skills were used by 94% of the students as they constructed Punnett Squares and pedigrees to solve the problem of the MEGA. Visual and information literacy was also shown consistently throughout the duration of both classes. Being immersed in a 3D virtual environment was clearly not a novelty to these students. They quickly learned how to ascertain information in the MEGA and used that information appropriately when creating the Punnett Squares and pedigrees. The final piece of the digital age literacy of multicultural and global awareness was only observed once in both classes. This was based on a response from a student stating they were glad the teacher did not make a black character the criminal.

Inventive thinking was the skill set seen most during the observations of these classes. All of the students observed showed ability in managing a complex system, adapting to the MEGAs scaffolds, and illustrating self‐direction through trial‐and‐error. Because of the high engagement of all students in these classes, it was easy to note that students were curious, creative and were willing to take risks. What was of particular interest was that many students’ initially failed in arresting the correct suspect but that failure served as motivation rather than a detraction from further discovery as we often see in traditional teaching. Finally, student success was grounded in high‐order thinking skills and sound reasoning to win the MEGA. All but three students positively finished the game in both 90‐minute classes.

High productivity was also seen throughout MEGA play. Although one of the skill sets (prioritize, plan and manage for results) is part of this twenty‐first century skills category, most game players do not usually show these. As many gamers learn the virtual surroundings through trial‐and‐error, it is not until they feel comfortable with the environment that they begin prioritizing, planning and managing information. That being said, observations on this skill set did not appear until about 10 minutes into each of the four classes observed. The students observed showed effective use of real‐world tools; most of which were learned in the unit. It is also important to reiterate that high quality products in the form of Punnett Squares and pedigrees were necessary to complete the game correctly and on time.

Finally, effective communication techniques were observed during student MEGA play. Because of the multiplayer aspect of the interface, students effectively used in‐game chat and when that was not possible they adapted to talking and passing notes. Many of the comments observed and coded in the text chat were of a collaborative nature. The collaborative groups were neither pre‐designed nor structured by the teacher or researchers. Students gravitated to groups that they felt comfortable in; which is generally the way social networks emerge online. Generally students were providing their peers with clues they had found in the MEGA and it was not observed until near the end of each class that students were telling their peers whom to arrest. It can be speculated that because of the CSI theme, students were highly productive through using real‐world tools in a virtual world. A possible reason for the results in the effective communication group might be the fact that the online connection was not robust enough to support multiplayer interactions. Although the high school in which this game was delivered is set in an affluent area, the reality of out‐dated technology in schools became a factor. Thus, students began talking to each other and passing notes showing the need for immediate information (Oblinger and Oblinger 2005). The objects were slowly cached and it was obvious, as observed, that the students playing the game were frustrated by the slow frame rates that affected play.

The MEGA was designed in a 3D virtual environment that lacked much of the photorealistic modeling and game play events, behaviours and artificial intelligence that many commercial game engines provide. Students who were immersed in this world simply walked through the environment and were able to perform laboratory techniques that they would not normally be able to perform in a high school laboratory. Although the story of the game was creative and engaging, there was a concern that students would simply discover a way to play the MEGA rather than playing the MEGA to learn content. It is a special craft to embed content in a stealthy manner in a video game. Serious game designers have been struggling with this for years. In the commercial gaming world, many gamers find ways to cheat (bypassing levels to achieve a goal) and thus this style of game play was infused into the educational game. Belanich, Sibley and Orvis (2004) suggest that game players often recall procedures and information relevant to the game rather than recalling facts embedded in the game.

Implications for practice

Educators could learn from the experience of the business field. It commonly occurs that educators seek answers without first consulting with the consumer (in this case, the students). To design a good educational video game, it is good practice to include the students in the creation process (Cobb et al. 2002). Ask their opinion and take their input. The teacher, to enhance the MEGA with academic achievement in mind, used results from this study. Wheeler (2004) recommends that technology companies use classrooms as alpha and beta test centres. They are the gamers and the people we are trying to reach. The skills gained by the students can be an avenue by which the twenty‐first century skills can be acquired and used as a driver of economic growth and power.

As with any other teaching strategy, especially with technology, teachers need to be trained on how to correctly use the material in their teaching practices. This will not only enhance their pedagogy but also make them critical evaluators of using the games and assessing content within the games. Teachers can then be called upon as collaborators in game design. Simulation design requires the ability to step outside a traditional, linear approach to content creation – a process that is challenging and counter‐intuitive to many untrained teachers (Morrison and Aldrich 2003).

What can be gleaned from this study is that students were exposed to, and began to, ascertain twenty‐first century skills. Leu et al. (2004) described twenty‐first century skills in terms of literacy. It can be argued that students playing educational games that have rich problem‐solving and complex problems embedded in the back‐story are being exposed to a new scientific literacy. This is literacy in the form of understanding real world problems and solving those problems in a simulated virtual environment. Gee (2003) has extensively written about the potential of video games in education but admits there is not enough hard evidence to support or refute their effect on student learning. This study is a beginning to the much‐needed future research in the area of video games in education.

Video games in education are not a panacea. However, the value of games in education has been speculative to this point. Studies such as this are critical if we are committed to finding the right mix of games and pedagogy for student learning to occur. The research community, education programs and software developers should plan cohesively to implement video games as a teaching and learning tool. Using gaming in education brings about new issues not previously encountered in professional development or education programs. There must be consideration for students in the game design, as well as professional development for teachers on how to use games in class, use of a real game engine, and creating the game around the assessment or vice‐versa. It is important to reiterate that the students who played the game became the game designers for the future of I.M. Megabucks. They helped the teachers to redesign the game to make it more fun, while the teacher remained the content and pedagogical expert. The real power of educational gamers might lie in the students as game developers as a new form of constructing knowledge. Educators often subscribe to the notion that you learn material best when you have to teach it. If students strive to create a great game, then they will need to learn content to be sure the game is created correctly. This is where the teacher becomes the facilitator of knowledge being certain that misconceptions are not perpetuated in an educational game.

The biggest factor limiting games in education is arguably the lack of good artificial intelligence to generate believable conversations and interactions (Gee 2003). Although Microsoft has recently marketed a game development SDK for the Xbox, there is barely any development software available to non‐programmers. Moreover, there is little evidence that schools have game consoles such as Xbox in the classrooms. This is where the HI FIVES project has evolved. The project has moved from Activeworlds to building a user interface around Valve’s Half‐Life 2 game engine. The source code allows for multiplayer, recorded events, behaviours and artificial intelligence commonly found in commercial games. One major limitation to the evolution is that most school hardware does not have the RAM, processor and/or graphics cards to render the photorealism used in many popular game engines. Future teacher‐created MEGAs will include these elements and further research will shed light on how this variable affects student learning and the acquisition of twenty‐first century skills.

Acknowledgements

The authors would like to acknowledge the work of Ms Maya Schultz for her commitment to this project and her exemplary science teaching.

Appendix. Twenty‐first century skills (Lemke 2003)

Table

Basic literacy	Can students demonstrate language proficiency (in English) and numeracy at levels necessary for success on the job and in a digital‐age society?
Scientific literacy	Do students have the knowledge and understanding of scientific concepts and processes required for personal decision‐making, participation in civic and cultural affairs, and economic productivity?
Economic literacy	Can students identify economic issues; analyze incentives; examine the consequences of changes in economic conditions and public policies; collect and organize economic evidence; and weigh costs against benefits?
Technological literacy	Do students know what technology is, how it works, what purposes it can serve, and how it can be used efficiently and effectively to achieve specific goals?
Visual literacy	Can students interpret, use, appreciate, and create images and video using both conventional and twenty‐first century media in ways that advance thinking, decision‐making, communication, and learning?
Information literacy	Are students able to evaluate information across a range of media; recognize when information is needed; locate, synthesize, and use it effectively; and accomplish this using technology, communication networks, and electronic resources?
Multicultural literacy	Can students understand and appreciate similarities and differences between the customs, values, and beliefs of their own culture and the cultures of others?
Global awareness	Do students recognize and understand relationships among international organizations, nation‐states, public and private economic entities, sociocultural groups, and individuals across the globe?
Adaptability/managing complexity	Can students modify their thinking, attitudes, or behaviors to be better suited to current or future environments? Can they handle multiple goals, tasks, and inputs while understanding and adhering to organizational or technological constraints of time, resources, and systems?
Self‐direction	Are students able to set goals related to learning, plan for the achievement of those goals, independently manage time and effort, and independently assess the quality of learning and any products that result from the learning experience?
Curiosity	Do students have a desire to know or a spark of interest that leads to inquiry?
Creativity	Are students able to bring something into existence that is genuinely new and original, whether personally (original only to the individual) or culturally (where the work adds significantly to a domain of culture as recognized by experts)?
Risk‐taking	Are students willing to make mistakes, advocate unconventional or unpopular positions, or tackle challenging problems without obvious solutions, such that their personal growth, integrity, or accomplishments are enhanced?
Higher‐order thinking and sound reasoning	Are students adept at cognitive processes of analysis, comparison, inference/interpretation, evaluation, and synthesis, as applied to a range of academic domains and problem‐solving contexts?
Teaming and collaboration	Can students cooperatively interact with one or more individuals, working with others to solve problems, create novel products, or learn and master content?
Interpersonal skills	Are students able to read and manage their own and others’ emotions, motivations, and behaviors during social interactions or in social–interactive contexts?
Personal responsibility	Do students demonstrate a depth and currency of knowledge about legal and ethical issues related to technology, combined with an ability to apply this knowledge to achieve balance, integrity, and quality of life as citizens, family and community members, learners, and workers?
Social and civic responsibility	Can students manage technology and govern its use in ways that promote the public good and protect society, the environment, and democratic ideals?
Interactive communication	Do students generate meaning through exchanges using a range of contemporary tools, transmissions, and processes?
Prioritizing, planning, and managing for results	Do students organize to efficiently achieve the goals of specific projects or problems?
Effective use of real‐world tools	Can students use real‐world tools (i.e., the hardware, software, networking, and peripheral devices used by information technology (IT) workers to accomplish twenty‐first century work) to communicate, collaborate, solve problems, and accomplish tasks?
Ability to produce relevant, high‐quality products	Are students adept at developing intellectual, informational, or material products that serve authentic purposes and occur as a result of their using real‐world tools to solve or communicate about real‐world problems? These products include persuasive communications in any media (print, video, the Web, verbal presentation), synthesis of resources into more useable forms (databases, graphics, simulations), or refinement of questions that build upon what is known to advance one’s own and others’ understanding.