Design evaluation of digital consumer products using virtual reality-based functional behaviour simulation

Abstract

To realize the virtual design and prototyping of every digital consumer product faithfully, it is important to allow all the people involved in the design process to experience its realistic appearance and functional behaviour. Proposed in this paper is a novel approach to design evaluation of a digital consumer product, which can satisfy such requirements using virtual reality-based functional behaviour simulation. In the approach, a product model, multimedia contents data, a functional behaviour model and a finite state machine are combined to constitute the virtual product model of a product of interest. A state transition methodology is adopted to capture the functional behaviour of the product, which is simulated by the finite state machine in a virtual reality environment. Based on the approach, a product design evaluation system has been developed and applied for the design evaluation of various products. A preliminary user study has been conducted to show the usefulness of the proposed approach by comparing it with other approaches using real products and two-dimensional screen prototypes.

Keywords:

• Product design evaluation
• Human–machine interaction
• Functional behaviour simulation
• State transition methodology
• Virtual product model

1. Introduction

Every digital consumer product has an embedded system, which contains computer logic on a chip inside it. Its functional behaviour implanted in the system is very complicated and nearly all is expressed as human–machine interaction (HMI) tasks, each of which may change the state of the product. In the market of digital consumer products including MP3 players, mobile phones, and portable multimedia players, consumers buy products according to ease of use, feature set and user interface. For successful entry of a new product into the competitive world market, it is imperative to dramatically reduce time to market as much as possible while precisely translating customer needs into the actual forms and functions of the product (Lee 1999, Ottosson 2002, Ulrich and Eppinger 2004).

An essential activity to meet these requirements is the efficient and extensive use of prototypes during the product development process. Prototypes represent important features of a product. They are used to prove design alternatives, to perform engineering analysis and manufacturing planning, to support management decisions, and often just to show a product to the customers. Physical prototyping usually involves creating physical models and the external structural design of a product. Computer-aided design (CAD) software is also used to design the external look of the product. Physical prototyping is very time consuming and expensive, although rapid prototyping may alleviate the shortcomings (Lee 1999, Ulrich and Eppinger 2004). Moreover, physical prototyping as well as CAD software generally cannot reflect the behaviour of the product.

With the recent advances in computer technology, virtual prototyping (VP) has been considered new and a much more powerful prototyping solution to overcome the shortcomings of conventional prototyping methods (Clayton et al. 1996, Bullinger et al. 1999, Lee 1999, Ottosson 2002, Zorriassantine et al. 2003). Powerful tools including stereoscopic display systems, head-mounted displays (HMD), data gloves and haptic devices have been introduced and combined to construct VP systems that provide realistic display of products in a simulated environment and offer various interaction and evaluation means (Burdea and Coiffet 2003). Bochenek et al. (2001) compared the performance of four different virtual reality (VR) displays in a design review setting and mentioned that the best approach for design review activities could be a combined technology approach. There have been some recent researches on virtual prototyping based on the combined technology. Verlinden et al. (2003) suggested the concept of augmented prototyping that projects the perspective images of the product on the physical object made by rapid prototyping techniques. Several vision-based mixed reality methods (Nam 2005, Lee and Park 2006, Park and Moon 2007) have been proposed to improve tangibility and interaction in product development processes. In those methods, the rendered images of the product are overlaid on the physical product in the real-world environment. Greenberg and Fitchett (2001) presented toolkits called Phigets allowing designers to explore tangible user interface for interactive product design.

Although the concept of VP has been widely employed and implemented in many industrial fields including automotive and airplane industries (Dai et al. 1996, Söderman 2005), most works have been focused on visualization (Lee 1999, Burdea and Coiffet 2003), assembly and disassembly testing (Jayaram et al. 1997, Siddique and Rosen 1997, Shyamsundar and Gadh 2002), manufacturing process simulation (Ong et al. 2002, Choi and Chen 2004), structural analysis (Lee 1999, Zorriassantine et al. 2003), and ergonomic analysis (Lee 1999, Burdea and Coiffet 2003). Significant works have not been conducted on capturing and utilizing the functional behaviour of digital consumer products in VP applications. In order to realize the virtual design and prototyping of a digital consumer product faithfully, it is very important for the people involved in product development to understand the functional behaviour of the product and to visualize its appearance as easily and rapidly as possible.

In this paper, we propose a novel approach to design evaluation of a digital consumer product, which can satisfy such requirements by combining VR-based interaction with functional behaviour simulation. In the approach, a product model, multimedia contents data, an HMI functional model, and a finite state machine (FSM) are combined to constitute the virtual product model of a product of interest. The product model has geometric part models of the product and their assembly and kinematics information. The multimedia contents data are used to provide visual and auditory display information required for realistic experience of the product and vivid simulation of its functional behaviour. A state transition methodology is adopted to capture the functional behaviour of the product into a mark-up language-based information model. The FSM is used to control the functional behaviour based on the information model and to perform the functional simulation of the product. The operations of the virtual product model in a VR environment facilitate the realistic visualization of the product and the vivid simulation of its functional behaviour.

Based on the approach, a product design evaluation system has been developed and applied for the design evaluation of various products. The system is designed to provide an open architecture that allows VR-oriented devices such as HMD, dual monitors, and stereo speakers to be integrated with ease into a simple low-cost desktop VR environment. A preliminary user study has been conducted to evaluate the proposed approach with other approaches using real products and two-dimensional (2D) screen prototypes. Its results have been so encouraging that the system can be expectedly used as an important tool for the review and evaluation of digital consumer products: it can allow users to conduct functional evaluation and usability test before engaging in the costly and time-consuming process of building physical prototypes; and it can greatly help to reduce communication errors occurring in a typical product development process.

The rest of the paper is organized as follows. In section 2, the proposed approach is described in detail. Section 3 explains the operations of the virtual product model of a digital consumer product in a VR environment. Section 4 addresses the implementation and the application of the product design evaluation system based on the proposed approach. Section 5 describes a preliminary user study to show usefulness of the approach. Section 6 closes the paper with some concluding remarks.

2. Proposed approach

Shown in figure 1 is the architecture of the product design evaluation approach proposed in this paper. There are five main tasks required for relevant product design evaluation and downstream applications: creation of a product model, acquisition of multimedia contents data, generation of an HMI functional model, construction of a FSM, and their use in a VR environment. Figure 2 shows a graphical diagram depicting key components used for the proposed product design evaluation and their relations. In the diagram, an MP3 player is used as an example of a digital consumer product. A product model, multimedia contents data, an HMI functional model, and a FSM constitute a virtual product model whose operations in a VR environment facilitate the realistic visualization of the product and the vivid simulation of its functional behaviour.

Figure 1. Architecture of the VR-based product design evaluation.

Figure 2. Key components of the proposed approach and their relations.

2.1 Creation of a product model

Creating a product model is the most basic step for building the virtual product model. The product model includes the geometry, the attributes of material and colour, the assembly structure, and the kinematics of the part components of the product (Lee 1999). In general, geometric models of the part components can be created with CAD software. In case only physical prototypes or soft mock-ups are available, the geometric models are created with reverse engineering (RE) tools (Varady et al. 1997).

In this work, we approximate the geometric models by triangular meshes that are fine enough to provide realistic visualization, and we store them in OBJ formats. Figure 3 shows RE-based product model generation for an MP3 player. After scanning point cloud data of part components of the product with a three-dimensional (3D) scanner, we generate 3D geometric models of the part components from the scanned data. To obtain the complete geometric model from the point clouds, we need to perform various tasks of point data processing, such as noise removal, registration of the point clouds, triangulation, merging, smoothing, hole filling, decimation, subdivision (Varady et al. 1997). After splitting it into pieces by part components, we specify the colour and material of each piece.

Figure 3. RE-based geometric model creation: (a) scanned data at multiple views; (b) single triangular mesh; (c) final geometric model.

2.2 Acquisition of multimedia contents data

Multimedia contents data are used to create visual and auditory display information required for realistic visualization of a product and vivid simulation of its functional behaviour. In this work, we use three kinds of the multimedia contents data: graphical images, audio sounds, and video animations. Every digital consumer product has at least one LCD display to show visual information related to its specific states. Such visual information includes graphical images and video animations. Also, the product has an audio output devices or speakers to output auditory information; that is, audio sounds specific to its states. We use the JPEG file format for graphical images, MP3 or WAV file formats for audio sounds, and the AVI file format for video animations. The multimedia contents data, which can be purchased or acquired by audio/video recording, are managed with a database management tool for their adequate use in the VR environment. Shown in figure 4 are some graphical images used for the LCD display of the MP3 player. Small images are automatically pasted onto the big blue background image to create state-specific images during HMI functional simulation (see later figure 9).

Figure 4. Examples of graphical images used for LCD display of the MP3 player.

Figure 9. VP model for the MP3 player at four different states: (a) MP3 play; (b) mode select; (c) FM radio; (d) hold; and (e) VR-based product design evaluation.

2.3 Generation of an HMI functional model

The HMI functional model of a product is an information model that represents the HMI-related functional behaviour of the product. In this work, we adopt a state transition methodology (Rumbaugh 1988, Harel 1987, Feng 2004) to capture the functional behaviour of a digital consumer product by breaking it down into the following entities:

•  All objects related to HMI tasks.

•  All HMI events to be occurred in the product.

•  All states in which the product can be.

•  Activity information related with each state.

•  State transitions occurring in each state.

•  Event–condition–action information related with each state transition.

Every digital consumer product has some part components (lamps, switches, buttons, sliders, indicators, displays, timers, and speakers) involved in the interaction between the user and the product. They are called objects, making the basic building blocks of functional simulation. Every object has a pre-defined set of properties and functions that describe everything it can do in a real-time situation. The overall behaviour of a product can be broken down into separate units of behaviour, which are called states. For example, the product can be ‘on’ or ‘off’. These are two distinct states. Each state represents a unit of behaviour expressed by a set of tasks. A product equipped with a complex embedded system would have many other states. A state can be as brief as a fraction of a second, or extend to several seconds, minutes, or hours. Each state, no matter how brief, is a distinct unit within the system's lifecycle.

The state of the product can be changed to another state. This is called state transition. For example, the state of an MP3 player can be changed from ‘on’ to one of the other states, such as ‘play’, ‘off’, ‘volume up’, ‘volume down’, ‘next music’, and ‘previous music’. Every state transition is triggered by one or more events associated with it. For example, clicking the play button triggers the transition from ‘on’ to ‘play’. As shown in figure 5, the transition occurs only when one of these events is activated and some specified conditions are met. Events are statements, such as ‘when the play button is pressed’ and ‘when the hold slider is moved to the left’. Conditions are statements, such as ‘if the power is on’ and ‘if the current MP3 music is paused’. Some tasks called actions can be performed before transition to a new state.

Figure 5. State transition.

In order to define the actual behaviour of the product, all the tasks performed in each state are specified. These tasks are called activities. They only occur when their state becomes active. Actions and activities are constructed using the objects’ properties and functions. Each action or activity consists of a set of statements. Each statement can be either the assignment of some value to a variable or the calling of a function of an object. For example, ‘switch the lamp on’, ‘turn up the volume’, or ‘play MP3 music’. Each statement can be a composite statement with a conditional statement as follows: ‘If the hold button index equals to 1, print the hold-on message in the display. Otherwise, print the current volume size in the display.’

In this work, we have devised a simple mark-up language to represent the functional behaviour of the product. After analysing and gathering the above-mentioned entities of the functional behaviour, we write them into a set of text files based on the mark-up language to build the HMI functional model. In the HMI functional model, each action or activity consists of a set of logical statements expressed with the objects’ properties and functions. The concrete execution according to each action or activity is conducted during functional simulation.

The HMI functional model can be used to create a state transition chart, which represents all the states and the possible state transitions between them. Figure 6 shows a state transition chart for an MP3 player where we consider only two main functional features: playing MP3 music and listening to FM radio.

Figure 6. State transition chart of an MP3 player.

2.4 Construction of a finite state machine

The FSM is used as an engine to control the transition between states of the product based on the HMI functional model. In this work, we developed a module to compile the text files for the HMI functional model and to operate the FSM. Figure 7 shows the control flow of the FSM module. Note that actions or activities, logically defined in the HMI functional model, are invoked during functional simulation. As they may contain statements invoking functions related to image synthesis and display, playing audio sounds, and handling video animation data, some function libraries for processing audio/image/video data are often required.

Figure 7. Control flow of the finite state machine.

3. Operations of a virtual product model

Once we have obtained the product model, multimedia contents data, the HMI functional model, and the FSM to constitute a virtual product model, we can start the design evaluation of the product in an environment equipped with VR tools. Figure 8 shows a schematic diagram to operate the virtual product model. Input devices include the keyboard, mouse, and data gloves. Output devices are classified into devices for visual output, auditory output, and force feedback. Visual output devices include mono monitors, HMD, and stereoscopic monitors. Auditory output devices include stereo speakers and headphone sets. Haptic devices can be used to provide users with force feedback. These devices are properly combined to provide HMI that fits application-specific situations (Lee 1999, Burdea and Coiffet 2003). The VR engine consists of several modules for dynamic viewing, visualization, audio/image/video data operations, and environment parameter settings. If the haptic devices are used, a module for feedback control must be included in the VR engine. For any combination of devices, when the user creates events with the input devices, the VR engine should recognize input events and distribute them to its modules, each of which generates the proper results sent to the output devices.

Figure 8. VR-based operations for product design evaluation.

According to HMI events recognized, the FSM change states of the product and activate associated actions or activities. Through the output devices, the user can experience the appearance, kinematics animation and functional behaviour of the product. With this scheme to operate the virtual product model, we can perform functional evaluation and usability testing of the product, which are inevitable tasks required in a typical product development process.

3.1 User interaction

Every digital consumer product has buttons to push or sliders to move. In order to create HMI events in this work, the user places the mouse pointer on specified regions (e.g. buttons or sliders) of the virtual product model and presses the left mouse button (LMB). A button or slider are considered to be pushed or moved (i.e. an HMI event occurs) if the following conditions are satisfied: the distance from the mouse pointer to the button or slider is the shortest among the distances from the mouse pointer to the other buttons and sliders; and the distance is kept to be smaller than a tolerance for a specified time. According to the size of the specified time, the HMI events related to buttons can be recognized differently. Although the mouse is replaced with a data glove, the user interaction is performed in a similar manner.

3.2 Realistic visualization

The visualization and kinematics animation of the virtual product model is accomplished as follows. The input events trigger various tasks, such as performing the state transition of the product, changing the viewpoint, setting visualization parameters (colour, light, material, etc), or modifying the position and orientation of part components. The state-specific visual information, created by the FSM during the state transition, is combined into the visualization module, which creates the visual image of the product and sends its result to visual output devices. Visualization quality depends on the accuracy of the geometric model of the product, the quality of the visualization module, and the performance of the visual output devices. Basically, the visualization module supports dynamic viewing (rotating, panning, and zooming), real-time shading, and 3D texture mapping.

The dynamic viewing allows the user to change the viewpoint with ease. The user can rotate the viewpoint by moving the mouse pointer on the reference monitor screen while pressing the control key and the left mouse button (CTRL+LMB). Similarly, the user can pan the viewpoint by moving the mouse pointer while pressing the shift key and the left mouse button (SHIFT+LMB). He or she can also zoom in and out by moving the mouse pointer while pressing the middle mouse button (MMB).

To create stereoscopic images of the virtual product model, we adopt dual stereo rendering (Bourke 2002, Brettle and Baranoski 2005). In dual stereo rendering, we use two view ports of the same size, create a stereo pair of images of the virtual product model at a given frame, and display them with the visual output devices (e.g. dual monitors or HMD). The left and the right images are rendered when the virtual product model and its environment are seen from the left and the right viewpoints, respectively. They are separately sent to the left and the right monitors of the visual output devices at the same time. When independently viewed, the images will present an acceptable image to the visual cortex, which will fuse the images and extract the depth information. The distance between two viewpoints is usually given as the average distance between a human's eyes.

3.3 Functional behaviour simulation

The functional behaviour simulation of the product is completed as follows. When the user creates an input event, the VR engine checks whether the event is related to the functional behaviour of the product or not. If so, the FSM module refers to the HMI functional model of the product and determines if the event triggers state transition. If the state transition is confirmed, the FSM module does the specified actions and changes the state to a new one, and performs the activities of the new state (see figure 7). The execution of the actions and activities yields state-specific visual and auditory data. The state-specific visual data are sent to the VR engine to update the visual image of the product, and the state-specific auditory data are sent directly to auditory output devices.

Described below is an example of HMI functional simulation of the virtual product model of an MP3 player. When the VP engine detects an event indicating that the power button of the MP3 player is pressed, it passes the control to the FSM module, which refers to the HMI functional model of the MP3 player, changes the state of the MP3 player from ‘off’ to ‘on’, turns on the LCD display with a starting display image, and returns the control to the VR engine (see figure 6). Now consider that the VR engine detects an input event of pressing the start button. This causes the FSM module to conduct the following tasks: change the state from ‘on’ to ‘play’, play MP3 music by sending the music sound to the speaker, and print the music name on the LCD display. Similarly, consider that the VR engine detects an input event of pressing the volume-up button—this causes the FSM module to perform the following tasks: change the state from ‘play’ to ‘volume up’, turn up the volume by one level, update the volume level on the LCD display, and return the state to ‘play’.

4. Implementation and application

Based on the proposed approach, a product design evaluation system has been implemented in C and C++ languages on a windows-based IBM-compatible personal computer. The developed system was designed to provide an open architecture that allows VR-oriented devices to be easily integrated into a simple low-cost desktop VR environment. As input devices, we used a keyboard and a mouse. As visual output devices, we used an HMD and two LCD monitors in a dual set-up to support 3D stereoscopic viewing. For the HMD, its type of optics is natural see-through, its field of view is 45°, and its resolution is per each eye. The resolution of each LCD monitor was set to . As auditory output devices, we used two speakers to support stereo sound.

For product model generation, we used CAD software called ^TMRhino 3D version 3.0 and RE software called ^TMRapidForm 2004. Graphical images, video animations, and audio sounds required as multimedia contents data were purchased or acquired by recording. We stored the HMI functional models of digital consumer products into mark-up language-based text files and used them in the FSM module. The VR engine includes modules for visualization, dual stereoscopic viewing, OBJ file handling, I/O interface handing, audio sound play, LCD image display, and environment parameter setting. With the LCD image display module, small images are automatically pasted onto the background image to create state-specific images of the products. The state-specific image can be updated in real time or between state transitions. For the visualization module, we used OpenGL and GLUT as graphics libraries. For the audio sound play module, we used Direct Show for MP3 decoding. We developed the other modules by writing our own source code.

We have applied the developed system for the design evaluation of various products including an MP3 player made by iRiver and a game phone made by LG electronics. Figures 9 and 10 show the virtual product models (i.e. virtual prototypes) of the MP3 player and the game phone at four different states and their operations in the VR environment, respectively. At some states, video animations are displayed and sounds (i.e. button click or music sounds) are played. Using the system, the users performed the design evaluation (e.g. functional evaluation and usability testing) of the products by operating their virtual product models. Usability testing is used for ensuring that the intended users of a product can carry out the intended tasks efficiently, effectively and satisfactorily. In usability testing, users are asked to perform certain tasks in an effort to measure the product's ease-of-use, task time, and the user's perception of the experience (Lee 1999, Kuutti et al. 2001, Ulrich and Eppinger 2004, Verlinden et al. 2004, Söderman 2005).

Figure 10. VP model for the game phone at four different states: (a) on; (b) calling; (c) multimedia menu; (d) movie; and (e) VR-based product design evaluation.

5. Preliminary user study

We are in the process of conducting a formal user study to investigate the usefulness and quality of the proposed approach. We carried out a preliminary user study of the MP3 player and the game phone with a subject group consisting of 10 university students. Of the 10, eight students learned the basics of 3D geometric modelling from CAD/CAM courses. Of these eight students, five had additional experience of using HMD. Simple task performance measures and questionnaires were used to evaluate the approaches using three different virtual prototypes: traditional 2D screen prototypes (2DSCR), 3D screen prototypes (3DSCR), and 3D stereoscopic prototypes (3DSTR). Note that 3DSTR corresponds to the proposed approach. As we have built the virtual prototypes from commercial products based on RE, we could include the use of real products (REAL) in the tests as the target reference of the prototyping approaches. Obviously, using a real product is the best, but the most costly approach to its design evaluation. As shown in Figures 11, 2D screen prototypes present front and/or side views of products. The image of the views usually has a limited resolution. In this work, they were built with ease by combining the images of views with multimedia contents data, HMI functional models, and the FSM. Similar to those created with 2D-based functional simulation tools, such as ^TMRapidPlus, these 2D screen prototypes allow users to perform the image-based functional simulation by clicking buttons on the views. In the developed system, we can test two 3D-based prototypes (3DSCR, 3DSTR) interchangeably. When using the screen prototypes (2DSCR and 3DSCR), the users just look at the 2D images of the products in a single LCD monitor screen (they do not wear the HMD).

Figure 11. 2D screen prototypes: (a) MP3 player and (b) game phone.

In order to evaluate the task performance of the prototyping approaches, we asked the subjects to complete two tasks for the MP3 player and four tasks for the game phone. Details of the tasks are described in table 1. We first introduced the subjects to three kinds of prototypes (2DSCR, 3DSCR, 3DSTR) of the two products for 10 min. Each subject was given some time (about 15 min) to learn how to manipulate them (i.e. how to click buttons and how to move, scale, and rotate 3D prototypes). The subject was then asked to conduct the six tasks using the four prototypes in the following order: . For each prototype, the tasks were assigned to the subject in the following order: . The prototypes and the tasks in braces {…} were ordered randomly. This random ordering was used to minimize the learning effect occurred during the repetitive tasks. Before performing each task, the subject could have access to a simple graphical manual describing the steps required to complete the task. The results of the performance measures are summarized in table 1 and plotted in figure 12.

Figure 12. Graphical plots of task performance measures for (a) the MP3 player and (b) the game phone.

Table 1. Task descriptions and performance measures.

			Average time required (s)
Product type	Task	Task steps	2DSCR	3DSCR	3DSTR	REAL
MP3 player	(T1) Play the third MP3 music with volume level 30	Turn the MP3 player on.	6.5	5.2	5.9	5.3
		Play the music.
		Move to the third music.
		Increase the volume up to 30 levels.
		Hold on the buttons (move ‘hold’ slider to the right).
	(T2) Play the second FM radio station with volume level 20	Hold off the buttons (move ‘hold’ slider to the left).	8.8	7.3	8.1	7.5
		Enter the menu.
		Select the FM radio submenu to play FM radio.
		Move to the second FM radio station.
		Decrease the volume down to 20 levels.
Game phone	(T3) Make a call to the first tester	Turn on the game phone.	10.0	6.7	7.7	6.4
		Enter the phone number using number buttons.
		Press ‘call’ button.
	(T4) Play an animation	Enter the main menu.	9.5	6.4	7.3	6.7
		Move to the multimedia submenu (press ‘6’ button).
		Select the second bin for animation files.
		Play the animation by pressing ‘OK’ button.
	(T5) Change the caller ring into the third one	Enter the main menu.	9.0	5.8	6.6	5.6
		Move to the caller ring submenu (press ‘2’ button).
		Select the third one.
		Set it to the current caller ring by pressing ‘OK’ button.
	(T6) Send a message ‘hi’ to the second tester	Press ‘message’ button to enter the message menu.	20.3	14.0	15.0	13.0
		Enter the phone number.
		Press ‘OK’ button.
		Enter the message ‘hi’ using 12 buttons *, #, 0∼ 9).
		Press ‘OK’ button to send the message.

From the results, we found that the overall ranking of task performance was as follows: . The 2DSCR approach required more time to complete the tasks than the others. The 3DSCR approach and the REAL approach showed similar results in task performance. Between two 3D-based approaches, the 3DSTR approach showed slightly worse results. We found that the task performance largely depends on the complexity of button layout, the ease of button clicks, and the visibility of prototypes. For the MP3 player, it has a few buttons, but their layout in the 2D screen prototype is rather confusing as the buttons were distributed into three views (one front view and two side views). This often caused the subjects to make mistakes when picking buttons. For the game phone, it has over 30 buttons whose layout is rather complex, and some buttons are small and compact. Moreover, the visibility of the 2D screen prototype in either case is not good. Some subjects commented that they felt inconvenience and made mistakes when clicking buttons with the mouse pointer in the 2D screen prototype. In the 3D-based approaches, subjects can go closely to the prototypes to see them with finer details. This helps to make the subjects click buttons more easily and accurately. As Kuutti et al. (2001) pointed out, it might be unnatural to go more closely to an object than where the eye can be focused. Nonetheless, it must be one of the advantages of using 3D virtual prototypes to freely manipulate (move, scale, and rotate) them in a VR environment. Some subjects without experiences of using HMD commented that in the 3DSTR approach they had some inconvenience in visibility of 3D prototypes while wearing the HMD. This can explain the slight decrease of task performance in the 3DSTR approach compared with the 3DSCR approach.

After completing all the tasks, the subject was asked to complete questionnaires, in order to capture qualitative aspects (i.e. understandability of functions, ease of use, sense of realism, legibility) of his or her experience of three prototyping approaches (2DSCR, 3DSCR, 3DSTR). Verlinden et al. (2004) performed similar qualitative comparison of their virtual prototyping approach with the 2DSCR approach. The questions asked are summarized in table 2. All responses were scored on a five-point scale and each question included a field to add some comments. Figure 13 shows the analysis results of the questionnaires collected from all the subjects and from the subjects with experiences of using HMD. In most of the questions, the 3D-based approaches (3DSCR, 3DSTR) received higher points than the 2DSCR approach. We also found some trade-offs between the 3D-based approaches.

Figure 13. Graphical plots of questionnaire results for (a) all subjects and (b) five subjects with experiences of using HMD.

Table 2. Questionnaire contents (translated from Korean).

Q1	Can you understand the product functions by using the virtual prototype?
Q2	Is it easy to click buttons when using the virtual prototype?
Q3	Does the virtual prototype look like the real product?
Q4	Can you figure out the size of the product with the virtual prototype?
Q5	Can you feel a three-dimensional effect when using the virtual prototype?
Q6	Is the display of the product legible?
Q7	Do you think the use of virtual prototype interesting?
Q8	Does the virtual prototype offer enough information for a decision to buy the product?

As the same HMI functional models were integrated into all the three kinds of prototypes, there were no significant differences between the three approaches in understandability of the product functions. The 3D-based approaches gave most of the subjects with more ease of button clicks and better sense of realism than the 2DSCR approach. Regarding the 3D effect, the subjects perceived depth the most vividly in the 3DSTR approach. In the 3D-based approaches, the subjects could go closely to the prototypes to see the display of the products more clearly, so they felt that the display was more legible in the 3D-based approaches than in the 2DSCR approach.

On the other hand, the subjects had difficulty in figuring out the size of the product with all the three approaches. Intuitively, we expect to mitigate this problem by replacing the 2D-based picking by the mouse pointer with the 3D-based picking by a digital hand model. They all felt a strong interest in using the 3D-based prototypes (especially the 3DSTR approach). The subjects felt as if they were playing 3D games. Most of the subjects answered that 3D-based prototypes were more appealing to customers than the 2D screen prototype.

Additionally, some subjects pointed out that wearing the HMD in the 3DSTR approach might be less convenient as it often prevents their realistic view and makes them feel some unnatural weight gain in their heads. They also commented that it would be much better, if they could touch and grasp the prototypes in the 3DSTR approach, and that the 3D-based prototypes, if they can be accessed via the Internet, could draw great interests from a number of remote customers. Based on the preliminary user study, we found that the subjects’ feedback about the 3DSTR approach was encouraging as they could figure out the functional behaviour and the visual appearance of the product more easily and vividly.

6. Concluding remarks

The functional behaviour implanted in every digital consumer electronic product is very complicated and nearly everything is expressed as HMI tasks. To enhance the review and evaluation of product features and functions, and thereby help to realize significant reduction of cost and time in the product development process, it is important to provide not only the realistic visualization of the product, but also the vivid simulation of its functional behaviour.

We have proposed a novel approach of VR-based product design evaluation, which can satisfy such requirements by embedding HMI functional simulation into a simple low-cost VR environment. We adopted the state transition methodology to capture the functional behaviour of the product, from which we constructed a FSM for its functional simulation. Based on the approach, a product design evaluation system has been developed and applied for the design evaluation of products, such as MP3 players and mobile phones with highly encouraging feedbacks from users.

During VR-based design evaluation of a digital consumer product, the users may detect any problems in the overall appearance, the assembly structure, or the functional behaviour of the product. In such cases, product designers correct the problems and update the product model or the HMI functional model. As shown in figure 1, the users and the product designers can promote the product design and development by repeating the process of product design evaluation with the product model and the HMI functional model updated. Expectedly, the system can enhance the review and evaluation of digital consumer products, and thereby it can be used as an important tool for taking a step towards eliminating the need for physical prototypes and realizing faster product design and development.

The product design evaluation system developed is similar to the commercial software ^TMPlayMo 3D in that it can support functional behaviour simulation using 3D geometric models. However, with the software, it is not easy to represent the functional behaviour of complex products in detail. Moreover, through the software, one cannot combine VR-oriented devices, such as data gloves and HMD with the software since it does not provide an open architecture.

The more finely a virtual prototype represents the behaviour of a digital consumer product, the more closely it can reach the goal of virtual prototyping. In most cases, it is very time-consuming to describe all the states of the real product and represent its functional behaviour to the finest level of detail. It is, however, necessary to make such an effort at least once during the product development. Mostly, similar efforts are made in the process of creating the embedded software of the product. For future work, it is very meaningful and challenging to devise an efficient way to reduce the time and effort required to generate the HMI functional model of the product by reusing the information built in the embedded software.

The results of the preliminary user study guided the future research directions for improving the proposed approach. We will expand the developed system to be run on web-based environments that allow easy access to product design evaluation via Internet (Shyamsundar and Gadh 2002). For better reality and immersion, we are currently improving the system to have the following features:

• Integration of a virtual hand model into the system to improve the immersion of picking operations and the sense of scale of virtual product models.

• Manipulation of virtual product models using tangible objects based on augmented reality (Azuma 1997, Ishii and Ullmer 1997, Lee and Park 2006, Park and Moon 2007).

• Enhanced picking of buttons or switches of virtual product models using tangible objects integrated with touch sensing devices (Greenberg and Fitchett 2001, Nam 2005).

Moreover, we will perform formal and structured user studies about the digital prototyping approaches that were proposed herein and will be improved afterwards.