Comparing Inspections and User Testing for the Evaluation of Virtual Environments

REFERENCES

• Adamo-Villani , N. , Wilbur , R. and Wasburn , M. Gender differences in usability and enjoyment of VR educational games: A study of SMILE™ . Proceedings of the 2008 International Conference . Washington, DC. pp. 114 – 119 . IEEE Computer Society .
   Google Scholar
• Agarawala , A. and Balakrishnan , R. 2006 . “ Keepin’ it real: Pushing the desktop metaphor with physics, pile and the pen ” . In Proceedings of the ACM CHI 2006 , 1283 – 1292 . New York : ACM Press .
   Google Scholar
• Andre , T. S. , Hartson , H. R. , Belz , S. M and McCreavy , F. A. 2001 . The user action framework: a reliable foundation for usability engineering support tools . International Journal of Human-Computer Studies , 54 : 107 – 136 .
   Web of Science ®Google Scholar
• Bach , C. 2004 . “ Élaboration et validation de Critères Ergonomiques pour les Interactions Homme-Environnements Virtuels [Development and validation of ergonomic criteria for human virtual-environments interactions] ” . In Unpublished doctoral thesis, Université Paul Verlaine , France : Metz .
   Google Scholar
• Bach , C. and Scapin , D. L. 2003 . “ Adaptation of ergonomic criteria to human-virtual environments interactions ” . In Proceedings of Interact’03 , 880 – 883 . Amsterdam : IOS Press .
   Google Scholar
• Bach , C. and Scapin , D. L. Critères ergonomiques pour les interactions homme-environnements virtuels: Définitions, justifications et exemples [Ergonomic criteria for human-virtual environments interactions: definitions, justifications, and examples] . Tech. Rep. No. 5531 . 2005 . Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA)
   Google Scholar
• Bastien , J. M. C. and Scapin , D. L. 1992 . A validation of ergonomic criteria for the evaluation of human–computer interfaces . International Journal of Human-Computer Interaction , 4 : 183 – 196 .
   Google Scholar
• Bastien , J. M. C. and Scapin , D. L. Ergonomic criteria for the evaluation of human–computer interfaces . Tech. Rep. No. 156 . 1993 . Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA)
   Google Scholar
• Bastien , J. M. C. and Scapin , D. L. 1995 . Evaluating a user interface with ergonomic criteria . International Journal of Human-Computer Interaction , 7 : 105 – 121 .
   Web of Science ®Google Scholar
• Bastien , J. M. C. , Scapin , D. L. and Leulier , C. 1999 . The ergonomic criteria and the ISO/DIS 9241 - 10 dialogue principles: A pilot comparison in an evaluation task . Interacting with Computers , 11 : 299 – 322 .
   Google Scholar
• Bowman , D. , Gabbard , J. L. and Hix , D. 2002 . A survey of usability evaluation in virtual environments : classification of methods . Presence: Teleoperators and Virtual Environments , 11 : 435 – 455 .
   Google Scholar
• Bowman , D. A. and Hodges , L. F. 1997 . “ An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments ” . In Proceedings of the 1997 symposium on Interactive 3D graphics , 35 – 38 . New York : ACM Press .
   Google Scholar
• Bowman , D. , Kruijff , E. , LaViola , J. and Poupyrev , I. 2005 . 3D user interfaces: Theory and practice , Boston : Addison-Wesley .
   Google Scholar
• Chapanis , A. 1982 . “ Man/computer research at Johns Hopkins ” . In Information Technology and Psychology: Prospects for the future , Edited by: Kasschau , R. A. , Lachman , R. and Laughery , K. R. 238 – 249 . New York : Praeger .
   Google Scholar
• Chattratichart , J. and Lindgaard , G. 2008 . “ A comparative evaluation of heuristic-based usability inspection methods ” . In Proceedings of the ACM CHI '08 extended abstracts on Human factors in computing systems , 2213 – 2220 . New York : ACM Press .
   Google Scholar
• Cockton , G. and Lavery , D. 1999 . “ A framework for usability problem extraction ” . In Proceedings of Interact’99 , 344 – 352 . Amsterdam : IOS Press .
   Google Scholar
• Cockton , G. , Woolrych , A. , Hall , L. and Hidemarch , M. 2003 . “ Changing analysts’ tunes: The surprising impact of a new instrument for usability inspection method assessment ” . In Proceedings of People and Computers XVII: Designing for Society , 145 – 162 . New York : Springer Verlag .
   Google Scholar
• Conkar , T. , Noyes , J. M. and Kimble , C. 1999 . CLIMATE: A framework for developing holistic requirement analysis in Virtual Environments . Interacting with Computers , 11 : 387 – 402 .
   Google Scholar
• Connell , I. W. and Hammond , N. V. 1999 . “ Comparing usability evaluation principles with heuristics: problem instances vs. problem types ” . In Proceedings of Interact’99 , 621 – 629 . Amsterdam : IOS Press .
   Google Scholar
• Dow , S. , MacIntyre , B. , Lee , J. , Oezbek , C. , Bolter , J. D. and Gandy , M. 2005 . Wizard of Oz support throughout an iterative design process . IEEE Pervasive Computing , 4 ( 4 ) : 18 – 26 .
   Web of Science ®Google Scholar
• Dubois , E , Abou Moussa , W. , Bach , C. and Bonnefoy , N. 2008 . “ Modelling and simulation of mobile mixed systems ” . In Handbook of research on user interface design and evaluation for mobile technology , Edited by: Lumsden , J. 346 – 363 . Hershey : IGI Global .
   Google Scholar
• Dubois , E. , Truillet , P. and Bach , C. 2007 . “ Evaluating advanced interaction techniques for navigating Google Earth ” . In Proceedings of the 21st BCS HCI Group Conference HCI 2007 , Vol. 2 , 4 – 7 . London : BSC .
   Google Scholar
• Dubois , E. , Nedel , L. P. , Dal Sasso Freitas , C. M. and Jacon , L. 2005 . Beyond user experimentation: Notational-based systematic evaluation of interaction techniques in virtual reality environments . Virtual Reality , 8 : 118 – 128 .
   Google Scholar
• Durlach , B. N. I. and Mavor , A. S. 1995 . Virtual reality: Scientific and technological challenges , Washington, DC : National Academy Press .
   Google Scholar
• Faulkner , L. Reducing variability—Research into structured approaches to usability testing and evaluation . Proceedings of Annual Conference of the Usability Professionals Association . July .
   Google Scholar
• Fischoff , B. 1982 . “ Debiasing ” . In Judgment under uncertainty: Heuristics and biases , Edited by: Kahneman , D. , Slovic , P. and Tversky , A. 422 – 444 . Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Gabbard , J. L. and Hix , D. 1997 . A taxonomy of usability characteristics in virtual environments , Blacksburg, VA : Unpublished master's thesis, Virginia Tech .
   Google Scholar
• Gray , W. D. and Salzman , M. C. 1998 . Damaged merchandise? A review of experiments that compare usability evaluation methods . Human-Computer Interaction , 13 : 203 – 261 .
   Web of Science ®Google Scholar
• Green , C. S. and Bavelier , D. 2006 . Effect of action video games on the spatial distribution of visuospatial attention . Journal of Experimental Psychology: Human Perception and Performance , 32 : 1465 – 1478 .
   PubMed Web of Science ®Google Scholar
• Griffiths , G. , Sharples , S. and Wilson , J. R. 2006 . Performance of new participants in virtual environments: The Nottingham tool for assessment of interaction in virtual environments (NAIVE) . International Journal of Human-Computer Studies , 64 ( 3 ) : 240 – 250 .
   Google Scholar
• Hand , C. 1997 . A survey of 3D interactions techniques . Computer Graphics Forum , 16 : 269 – 281 .
   Web of Science ®Google Scholar
• Hartson , H. R. , Andre , T. S. and Williges , R. C. 2001 . Criteria for evaluating usability evaluation methods . International Journal of Human-Computer Interaction , 13 : 373 – 410 .
   Web of Science ®Google Scholar
• Hornbaek , K. 2009 . Dogmas in the assessment of usability evaluation methods . Behaviour & Information Technology , : 1 – 15 .
   Google Scholar
• Hornbæk , K. and Frøkjær , E. Comparing usability problems and redesign proposals as input to practical systems development . Proceedings of ACM Conference on Human Factors in Computing Systems . New York. pp. 391 – 400 . ACM Press .
   Google Scholar
• Hughes , C. E. , Smith , E. , Stapleton , C. B. and Hughes , D. E. 2004 . “ Augmenting Museum Experiences with Mixed Reality ” . In Proceedings of Knowledge Sharing and Collaborative Engineering 2004 , Calgary, , Canada : Acta Press .
   Google Scholar
• International Standards Organisation . 2000 . ISO / TS 16982 Ergonomics of Human-System Interaction–Usability methods supporting Human Centred Design , Geneva : Author .
   Google Scholar
• Kalawski , R. S. 1999 . VRUSE: A computerised diagnostic tool for usability evaluation of virtual/synthetic environment systems . Applied Ergonomics , 30 ( 1 ) : 11 – 25 .
   PubMedGoogle Scholar
• Karampelas , P. , Grammenos , D. , Mourouzis , A. and Stephanidis , C. 2003 . “ Towards I-Dove, an interactive support tool for building and using virtual environments with guidelines ” . In Proceedings of the 10th HCI International , Vol. 3 , 1411 – 1415 . Mahwah, NJ : Erlbaum .
   Google Scholar
• Kaur , K. 1998 . Designing virtual environments for usability , London : Unpublished doctoral thesis, City University .
   Google Scholar
• Kaur , K. , Maiden , N. and Sutcliffe , A. Design practice and usability problems with virtual environments . Proceedings of Virtual reality World’96 conference . Stuttgart, Germany. IDG conferences .
   Google Scholar
• Kaur , K. , Maiden , N. and Sutcliffe , A. 1999 . Interacting with virtual environments: an evaluation of a model of interaction . Interacting with Computers , 11 : 403 – 426 .
   Web of Science ®Google Scholar
• Kennedy , R. S. , Lane , N. E. , Berbaum , K. S. and Lilienthal , M. G. 1993 . Simulator sickness questionnaire an enhanced method for quantifying simulator sickness . International Journal of Aviation Psychology , 3 : 203 – 220 .
   Google Scholar
• Law , E. , Scapin , D. L. , Cockton , G. , Springett , M. , Stary , C. and Winckler , M. Maturation of Usability Evaluation Methods: Retrospect and prospect . Proceedings of COST294-MAUSE Closing Conference . Toulouse, France. IRIT Press .
   Google Scholar
• Leulier , C. , Bastien , J. M. C. and Scapin , D. L. 1998 . Compilation of ergonomic guidelines for the design and evaluation of Web sites (Contract report) , Rocquencourt, , France : French National Institute for Research in Computer Science and Control (INRIA) .
   Google Scholar
• Lindgaard , G. 2006 . Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? . International Journal of Industrial Ergonomics , 36 : 1069 – 1074 .
   Web of Science ®Google Scholar
• Mankoff , J. , Fait , H. and Tran , T. 2005 . “ Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind ” . In Proceedings of the ACM CHI '05 , 41 – 50 . New York : ACM Press .
   Google Scholar
• Molich , R. and Dumas , J. S. 2008 . Comparative usability evaluation (CUE-4) . Behaviour & Information Technology , 27 : 263 – 281 .
   Web of Science ®Google Scholar
• Microsoft Game Studios. (2000). StarLancer. Redmond, WA: Author http://www.microsoft.com/games/da/starlancer/ (http://www.microsoft.com/games/da/starlancer/) (Accessed: 12 June 2008 ).
   Google Scholar
• Nielsen, J. (1993). Usability engineering. San Diego, CA: Academic.
   Google Scholar
• Pollier , A. Evaluation d'une interface par des ergonomes: Diagnostics et stratégies. [User interface evaluation by ergonomists: diagnoses and strategies] . Research Rep. No. 1391 . 1991 . Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA)
   Google Scholar
• Poupyrev , I. and Ichikawa , T. 1999 . Manipulating objects in virtual worlds: Categorization and empirical evaluation of interaction techniques . Journal of Visual Languages and Computing , 10 ( 1 ) : 19 – 35 .
   Google Scholar
• Scapin , D. L. 1990 . Organizing human factors knowledge for the evaluation and design of interfaces . International Journal of Human-Computer Interaction , 2 : 203 – 229 .
   Google Scholar
• Scapin , D. L. and Bastien , J. M. C. 1997 . Ergonomic criteria for evaluating the ergonomic quality of interactive systems . Behaviour & Information Technology , 16 : 220 – 231 .
   Web of Science ®Google Scholar
• Scapin , D. L. and Law , E. 2007 . “ Review, report and refine Usability Evaluation Methods (R3 UEMs) ” . In Proceedings of the COST294-MAUSE 3rd International Workshop , Greece : Athens .
   Google Scholar
• Schmettow , M. and Vietze , W. 2008 . “ Introducing item response thery for measuring usability inspection processes ” . In Proceedings of the ACM CHI '08 , 893 – 902 . New York : ACM Press .
   Google Scholar
• Shneiderman , B. 1998 . Designing the User Interface: Strategies for Human-Computer Interaction , MA : Addison-Wesley .
   Google Scholar
• Stanney , K. M. 2002 . “ Handbook of Virtual Environments: Design ” . In implementation, and applications , Mahwah, NJ : Erlbaum .
   Google Scholar
• Stanney , K. M. and Davies , R. C. 2005 . “ Augmented reality in Internet applications ” . In Handbook of human factors in Web design , Edited by: Proctor , R. W. and Vu , K-P L. 647 – 657 . Mahwah, NJ : Erlbaum .
   Google Scholar
• Stanney , K. M. , Mollaghasemi , M. , Reeves , L. , Breaux , R. and Graeber , D. A. 2003 . Usability engineering of virtual environment (VEs): Identifying multiple criteria that drive effective VE system design . International Journal of Human-Computer Studies , 58 : 447 – 481 .
   Web of Science ®Google Scholar
• Stanney , K. M. , Mourant , R. R. and Kennedy , R.S. 1998 . Human Factors issues in Virtual Environments: A review of the literature . Presence: Teleoperators and Virtual Environments , 7 : 327 – 351 .
   Web of Science ®Google Scholar
• Stapleton , C. B. and Hughes , C. E. Mixed reality and experiential movie trailers: Combining emotions and immersion to innovate entertainment marketing . Proceedings of 2005 International Conference on Human-Computer Interface Advances in Modeling and Simulation . New Orleans. pp. 40 – 48 .
   Google Scholar
• Sutcliffe , A. 2003 . Multimedia and virtual reality : designing multisensory user interfaces , Mahwah, NJ : Erlbaum .
   Google Scholar
• Sutcliffe , A. and Gault , B. 2004 . Heuristic evaluation of Virtual Reality applications . Interacting With Computers , 16 : 831 – 849 .
   Web of Science ®Google Scholar
• Sutcliffe , A. and Kaur , K. 2000 . Evaluating the usability of virtual reality user interfaces . Behaviour & Information Technology , 19 ( 6 ) : 415 – 426 .
   Web of Science ®Google Scholar
• Tromp , J. G. 2001 . Systematic usability design and evaluation for collaborative virtual environments , Nottingham, , United Kingdom : Unpublished doctoral thesis, University of Nottingham .
   Google Scholar
• Tromp , J. G. , Steed , A. and Wilson , J. R. 2003 . Systematic Usability evaluation and design issues for collaborative Virtual Environments . Presence: Teleoperators and Virtual Environments , 12 : 241 – 267 .
   Google Scholar
• Wickens , C. D. and Baker , P. 1995 . “ Cognitives issues in virtual reality ” . In Virtual environments and advanced interface design , Edited by: Barfield , W. and Furness , T. A. III . 514 – 541 . New York : Oxford University Press .
   Google Scholar
• Williams , J. S. and Harrison , M. D. 2001 . A toolset supported approach for designing and testing virtual environment interaction techniques . International Journal of Human-Computer Studies , 55 : 145 – 165 .
   Google Scholar
• Wilson , J. R. , Eastgate , R. M. and D'Cruz , M. 2002 . “ Structured development of Virtual Environments ” . In Handbook of Virtual Environments. Design, implementation, and applications , Edited by: Stanney , K. M. 353 – 378 . Mahwah, NJ : Erlbaum .
   Google Scholar
• Witmer , B. G. and Singer , M. J. 1998 . Measuring presence in virtual environments: A presence questionnaire . Presence: Teleoperators and Virtual Environments , 7 : 225 – 240 .
   Web of Science ®Google Scholar
• Zachary , W. 1986 . Cognitively based functional taxonomy of decision support techniques . Human–Computer Interaction , 2 : 25 – 63 .
   Google Scholar