Defining requirements for an Augmented Reality system to overcome the challenges of creating and using design representations in co-design sessions

ABSTRACT

Within co-design sessions involving designers and non-designers, the type and characteristics of the design representations employed is known to impact the performance of such sessions in terms of idea generation, idea evaluation and communication. This study captures the challenges practitioners face in creating and using design representations for co-design sessions and goes on to investigate the potential of Spatial Augmented Reality (SAR) to overcome those challenges. The advantages of SAR in this application are that, multiple concepts can be represented using one physical model, concepts can be modified live during the session, and additional equipment (such as head mounted displays or handheld devices) is not required, thus eliminating any possible interference with the natural interactions between participants. Interviews with design practitioners and trials with a prototype SAR system are used to identify the key challenges faced by practitioners in their current use of design representations, and to capture the technology requirements for a SAR system for use in co-design sessions. These findings can inform the work of technology developers and researchers working on systems to support co-design sessions.

KEYWORDS:

• Co-design
• co-creation
• Spatial Augmented Reality
• prototype
• design representation
• requirements

1. Introduction

Design representations are of critical importance throughout the product development process in that they help to express design ideas and capture the proposed design in its current state of development and are updated as the design evolves over successive iterations. Previous research has shown that the type of design representation that is used can have a significant impact on the design process in terms of supporting or hindering communication (Billinghurst et al. 2003) and creativity (Atilola, Tomko, and Linsey 2016; Häggman et al. 2015). Practitioners therefore need to give careful thought to the type of design representation that they use, taking into account their objectives.

Co-design sessions are a special type of activity within the design process that bring with them additional challenges for the creation and use of design representations. Co-design participants typically range from internal team members, to clients and end users. Note that we use the term ‘clients’ to refer to people that work for the organisation, or part of an organisation, that has commissioned the design activity, whereas ‘end-users’ are the people that actually use the product. Differences in the professional backgrounds and levels of design expertise amongst the various types of co-design participants can result in communication barriers and misunderstandings.

‘X-reality’ technologies, including Virtual Reality (VR), Augmented Reality (AR), and Spatial Augmented Reality (SAR), offer new functionalities that may help to overcome some of the challenges that practitioners face when creating and using design representations within co-design sessions. In this paper, we explore the potential for a SAR-based system for the creation of design representations for use within co-design sessions. Requirements for the system are defined through an iterative process leading to the development of a prototype SAR system. Preliminary trials with this prototype SAR system are used to further refine the requirements. Two research questions are addressed:

1. What challenges do design practitioners face when making and using design representations within co-design sessions?

2. What requirements do design practitioners have for a SAR system that supports co-design sessions?

The application scope of the study is limited to product and packaging design. The research leads to general findings that are relevant to:

• Researchers investigating the use of co-design sessions in industry

• Researchers investigating the role of design representations within co-design sessions

• Design practitioners that manage/facilitate co-design sessions (in-house or consultancy)

• Computer scientists that develop ICT tools for product development.

2. Literature review

Design representations play an important role in co-design sessions because they enable a group to develop a shared vision of the product that is being designed (Özçelik Buskermolen and Terken 2012) and support idea generation (Van Der Lugt 2005). However, design representations can also introduce challenges, particularly for co-design activities. Below we describe some of those challenges identified from the academic literature.

2.1. Challenges design practitioners face when making and using design representations within co-design sessions

2.1.1. Reducing the time and cost to make design representations

Reducing time to market and development costs are perennial concerns for product development in general, and this is also true for the preparation of design representations. Some factors that influence the time and cost required to produce design representations are the level of fidelity (i.e. the level of detail represented) and workmanship (i.e. the quality/professionalism of the finish) (Hallgrimsson 2012). The challenge for designers is to produce a design representation that serves the intended purpose as cheaply and quickly as possible. Lim, Stolterman, and Tenenberg (2008) have expressed this more formally as the ‘economic principle of prototyping’, which they define as, ‘the best prototype is one that, in the simplest and most efficient way, makes the possibilities and limitations of a design idea visible and measurable’.

2.1.2. Selecting the right type of design representation to support idea generation and review

A number of researchers have looked into the impact that the type of design representation has on different aspects of the design process. Concerning idea generation, Häggman et al. (2015) found that designers using blue foam as prototyping medium were able to generate ideas faster than those using CAD or sketching and were able to generate more ideas that were highly rated in terms of novelty. The authors also found that the concepts created using CAD were not rated highly in terms of novelty of geometry. This led Häggman to concur with an earlier finding by Robertson and Radcliffe (2009) that the use of CAD too early in the design process can sometimes lead to premature limitations on design space exploration, resulting in reduced novelty of ideas.

Another aspect that has been investigated is the impact of design representation type on the problem of ‘fixation’. ‘Fixation’ is the psychological phenomenon in which designers inadvertently carry over specific and unhelpful features from a previous example when they are designing something new (Crilly 2015). Concerning this topic, Atilola, Tomko, and Linsey (2016), compared the use of sketches and function trees as means of presenting an example solution to groups of student designers. They found that fixation occurred more frequently in the groups presented with sketches and concluded that, in comparison to sketches, function trees help to reduce fixation and lead to higher-quality ideas. Viswanathan and Linsey (2011), found that using design representations that require more time, effort or cost to create was correlated with higher levels of fixation and lower novelty in their design experiments.

As well as idea generation, researchers have also looked at the role of design representations in design review activities. Hannah, Joshi, and Summers (2012) presented groups of engineering students with solutions to a design task using a variety of design representations. Those teams that were presented with high-fidelity prototypes were most confident in their assessment of the solutions and scored the most correct answers.

These findings suggest that quick, cheap, and low-fidelity design representations may be most effective in supporting ‘divergent’, idea generation activities, whilst high-fidelity design representations are required for ‘convergent’ review and filtering activities.

2.1.3. Avoiding misinterpretations

Design representations are incomplete and approximate representations of a potential final product, which can lead to misinterpretations by stakeholders. For instance, during user testing of a mobile phone interface, Lim et al. (2006) found that building mock-ups of the interface on a PC screen or out of paper, rather than on a mobile phone screen, led to misunderstandings of how the interface should be used.

Hudson and Mankoff (2006) suggest that one of the challenges for user-interface design is that designers often have to choose between design representations that ‘…look like the final product (either physically, e.g. using media such as foam mock-ups, or visually, e.g. using tools such as Flash™) or work like the interactions envisioned for the final product (e.g. provide equivalent interaction via an on-screen simulation)’. They suggest that the separation of these two aspects of the design representation ‘…results in a less fluid process with potentially slower and weaker communication and iteration’ within the design process.

In summary, there are a number of challenges that design practitioners face when creating and using their standard types of design representations. These challenges have been identified through studies that have focused on specific issues related to design representations but it is interesting to note that there has not been a general study of the challenges faced by design practitioners when making and using design representations for co-design sessions. Hence, it is not evident that the challenges identified above represent a comprehensive and exhaustive summary of the challenges faced by practitioners when preparing design representations for co-design sessions. The study presented here aims in part to addresses this gap. In the following section we consider the role that X-reality technologies might play in overcoming some of the known challenges.

2.2. The role of X-reality technologies in co-design sessions

2.2.1. Introduction to X-reality technologies

At the highest level, X-reality technologies can be decomposed into VR, in which the user is completely immersed in a world that is entirely fabricated and AR, in which digital imagery is superimposed over the existing physical world, ‘augmenting’ it (Milgramand and Kishino 1994).

VR systems are typically implemented through a Head Mounted Display (HMD) or through a ‘CAVE’ system, which makes use of a cube-like space in which images are displayed by a series of projectors and are updated based on the orientation of the user’s head to ensure a realistic virtual perspective. Within AR, the two most common approaches are the use of an optical HMD (OHMD) or a handheld device (HHD), which allows digital content to be overlaid on the user’s view of the world. SAR is a less common version of AR in which digital content is projected on to the surface of a physical prototype. The projected content can be used to represent:

• the intended colours, materials and finishes of the surface of the product

• logos, images and text (see Figure 1(a))

• screens, buttons, dials, or other user-interface elements

• hidden details and visualisations of engineering analysis.

Figure 1. (a) A SAR model and graphical user interface. (b) SAR model with tracking.

When the SAR system features motion-tracking technology, the user can pick up and handle the design representation and the projected digital content will remain correctly aligned with the physical prototype (Figure 1(b)).

2.2.2. How could X-reality technologies address these challenges?

X-reality technologies offer the potential to overcome some of the challenges in the creation and use of design representations highlighted previously. For instance, the time and cost challenge might be addressed by using a single physical prototype, onto which multiple alternative concepts can be rendered digitally (using AR or SAR technology), thereby reducing the number of physical prototypes required and the associated materials costs. Further examples of how X-reality technologies could help to address the challenges identified in the previous subsection are provided in Table 1.

Table 1. Examples of how the challenges for the creation of design representations for co-design sessions could be addressed by X-reality technologies.

Challenge	How could X-reality help?
Time and cost of preparing design representations	Eliminate material costs (VR) or reduce materials costs reduced by presenting multiple concepts using one basic model (AR/SAR), although also need to consider initial hardware costs for X-reality technology
Limiting creativity by conforming to the characteristics of the tool	Not helped by X-reality technology
Avoid fixation phenomenon	Provide quick methods to make live modification of concepts
Deciding correct level of fidelity for design review	Level of fidelity can be easily modified according to the scope/objectives of the session
Avoid misunderstandings/misinterpretations	Can show the context of use (VR) and provide tangible prototype to avoid misinterpretation of scale or form (AR/VR)
Creating design representations that look like and work like the envisioned final product	Potential to quickly and cheaply integrate multimedia content and logic to show user interface/interaction aspects within virtual (VR) or physical (AR/SAR) prototypes

2.2.3. Technical challenges for the use of X-reality technologies in co-design sessions

Along with the potential advantages identified in Table 1, X-reality technologies also bring with them a new set of technical challenges. For example, Park and Moon (2013) have identified the lack of haptic feedback, the discomfort caused by wearing heavy (O)HMD devices for long periods, and loss of realism due to hand occlusions (where the projected digital content appears on the user’s hands instead of the product) as some of the technology-specific challenges. Other challenges include the need to provide an unlimited field of view (Billinghurst et al. 2003), the need to provide a shared view of the design representation to support deixis in interaction (i.e. when making reference to ‘This part here…’) (Hindmarsh and Heath 2000), the need to allow participants to see each other in order to support non-verbal and gestural communication between participants (Reid and Reed 2007), and the need to support tangible methods for interacting with the design representation (Ware and Rose 1999). The performance of VR, AR, and SAR technologies with respect to these challenges is discussed in Table 2.

Table 2. Performance of VR, AR, and SAR technologies against the technical challenges associated with co-design sessions.

Challenge	VR	AR	SAR
Unrestricted field of View	Limited if using HMD. Natural field of view of human eye for VR CAVE	Limited by field of view of OHMD or handheld device camera.	Natural field of view of human eye
Provide a shared view of the design representation	No – individual viewpoint	No – individual for OHMD or glasses. Yes – shared for handheld device	Yes
Support non-verbal/gestural communication between participants	Gestures cannot be seen by other participants if using a HMD	May be limited by field of view OHMD or need to maintain hold on handheld device	Unrestricted
Ensure user comfort	Potential for discomfort and nausea from wearing HMD or head tracking equipment.	Potential for discomfort from wearing OHMD or holding handheld device.	No impact on user comfort.
Provide a convincing visual realism in design representation	Potential occlusions possible with CAVE	Some occlusions or loss of tracking possible	Some occlusions or loss of tracking possible
Support tangible methods for interaction	No – unless physical props are used	Yes	Yes
Provide haptic feedback	Can be simulated to some extent through haptic glove/arm/suit	Yes – from touching real prototype	Yes – from touching real prototype

From Table 2 it becomes clear that SAR technology has the potential to overcome many of the challenges that are present when using either VR or AR technologies. This potential is being further investigated as part of the ‘SPARK’ project – a 3-year, EU-funded research and development project. The results presented in this paper are helping to inform the development of the SPARK system and could be applied in the development of other X-reality (AR/VR/SAR) technologies for design and product development. The following section reviews the previous research on the use of SAR technology for co-design.

2.2.4. Examples of the use of SAR technologies for co-design

Verlinden (2014) has proposed the ‘IAP-M’ design methodology, which features the use of interactive SAR prototypes for design review purposes. He concluded that the SAR technology and the IAP-M methodology were useful in developing a shared understanding amongst stakeholders of the design and helped with the early identification of errors and flaws in the design.

Porter et al. (2010) report on the testing of a SAR-based system that incorporates finger tracking for use in the detailed design phase of product development. Despite some challenges with ‘button press’ actions, many participants felt that SAR provided a good visual representation of the concept and 88% of participants agreed that SAR technology would be useful as a design tool.

Irlitti and von Itzstein (2013) report on the development of the ‘SARventor’, which combines SAR with three tangible user-interface ‘tools’. The system was presented to three experts from architecture and industrial design. Challenges noted by the reviewers included the lack of a visible toolkit and the inability to manipulate volume (3D geometry). Despite this, the reviewers felt there was ‘…a strong case towards [SARventor] being used as a collaborative tool for use in feedback sessions between designers and stakeholders’.

Akaoka, Ginn, and Vertegaal (2010) introduce ‘DisplayObjects’ as ‘a new category of future everyday computational objects with fully interactive skins’. The DisplayObjects workbench uses SAR technology to project graphics and user-interface elements on to physical prototypes – see Figure 2. Testers liked the interactive, hands-on approach and the ability to change elements quickly. However, they did find problems with hand occlusions and found it difficult to create good digital models from 3D scans (as the model ‘cleaning’ process was time-consuming).

Figure 2. ‘DisplayObjects’ SAR system (Akaoka, Ginn, and Vertegaal 2010), also showing an example of hand occlusion.

Other studies of SAR technology in design have contributed to specific technology development aspects, most notably on improving the accuracy of colour rendering (Menk and Koch 2013; Park et al. 2015)

One thing lacking from most of the technology development activities discussed in this section is a detailed description of how the requirements for the system were specified. Verlinden (2014, 80–81) does provide a set of requirements for a SAR system for co-design but the focus of the research is on design review activities rather whereas we understand co-creative design sessions to include both idea generation and review activities. Sage (2017) reports on work to identify requirements for enterprise AR systems but these do not specifically cover SAR technology and they refer to scenarios of usage such as warehouse picking, object assembly, maintenance, and repair operation rather than co-design sessions. There is therefore a lack of information in the academic literature concerning the design practitioner’s needs and requirements for X-reality technology-based design representations for co-creative design sessions – this study begins to address this gap.

3. Methodology

The aims of this study were to understand practitioners’ challenges in creating and using design representations for co-design sessions and to define the requirements for a SAR-based system for co-design. An iterative approach was used to support the identification of challenges and refinement of the requirements. The first iteration of the requirements was based on discussions between technical development staff from within the SPARK consortium and designers from the two industry partners of SPARK: Stimulo – a product design consultancy; and Artefice – a packaging and brand design agency. This was complemented with technical meetings with potential suppliers of the projector and tracking technology necessary for a SAR system.

The second iteration of the challenges and requirements was developed based on an interview study with design practitioners from a range of product and graphic design organisations. These interviews were also used to identify the common challenges that practitioners face in the preparation and use of design representations within co-design sessions.

The third iteration of the requirements was developed based on feedback from practitioners that had participated in trial co-design sessions with an early prototype of the SAR technology.

This iterative approach was necessary as users will often struggle to describe their requirements for a new technology if they are not familiar with that technology (Von Hippel 1986). It would have therefore been very difficult to identify a significant number of design practitioners with the necessary experience of SAR technology to be able to define a comprehensive set of requirements for SAR-based design representations without seeing some kind of demonstration of the technology. The iterative approach has therefore enabled the requirements to be defined and refined to a sufficient level of detail to support the key technical design decisions and feature prioritisation activities at each stage of the technology development process. Further details of the methodology for the interview study and the trials with early prototypes are presented in the following subsections.

3.1. Methodology for interview-based study

The criteria for selecting companies to participate in the study were that they should have significant product or graphic design activities and be based at a site in Europe. The practitioners were required to have experience of creating and using design representations and experience of conducting co-design sessions which use those representations. Suitable companies and practitioners were identified and contacted by email. They were provided with a brief introduction to the SPARK project and objectives of the interview.

Fifteen design practitioners from 11 different companies participated in the interviews. Table 3 provides an overview of the main characteristics of the companies and the practitioners involved. The companies agreed to participate in the research on the condition of anonymity, hence no identifying features of the companies or participants are provided. Companies A to G were based in the UK and were interviewed by two researchers from the University of Bath. Companies H to K were based in Belgium and were interviewed by a researcher from Antwerp Management School.

Table 3. Overview of the participating companies and practitioners interviewed.

Company background	Typical products	Interviewees job title
Company A Global consultancy specialising in product design of consumer goods and branding	Fast-moving consumer goods Packaging	Creative Director
Company B In-house consultancy providing design services to various business units within a large manufacturer of FMCG packaging as well as external clients	Structural Packaging for the FMCG sector	Designer
Company C Small- to medium-sized research, design, and innovation consultancy with experience of product design and product development	Packaging Medical equipment Military equipment Consumer goods	Design Director Head of User Experience Design
Company D Small industrial design consultancy	Professional equipment Industrial machinery	Industrial Designer (Owner)
Company E Small product innovation consultancy with a strong focus on the front-end innovation activities	Consumer goods Packaging	Designer FMCG Designer
Company F Small consultancy specialising in structural packaging design	Packaging	Creative Director Designer
Company G Large manufacturer	Household electronics	Principal Industrial Design Engineer Principal Design Engineer Product Design Engineer
Company H Small design consultancy	Toys Consumer goods	Design manager Senior designer
Company I Medium-sized consultancy offering research, design and innovation support services	Professional equipment Furniture Display equipment	Co-owner
Company J Large manufacturer	Luggage	Design Director – Europe
Company K Small- to medium-sized consultancy offering design and innovation services	Industrial machinery Furniture Electronic equipment	Head designer/Owner

The protocol for the interviews was developed with the following objectives in mind:

1. To understand more about the current practices of companies in terms of preparing and using design representations for co-design sessions.

2. To understand the challenges practitioners face when preparing and using design representations for co-design sessions (research question 1).

3. To provide a basic introduction for the participants to SAR technology.

4. To understand the practitioners’ requirements for SAR-based design representations for use in co-design sessions (research question 2).

The protocol developed featured four main sections, corresponding to the objectives listed above.

In the ‘current practices’ section (Obj 1), interviewees were asked to sketch out the main phases of their design process on a large sheet of paper. They were then asked about the types of co-design sessions that occur during the process and the type of design representations used in each of those sessions. This information was noted on the same diagram as the design process overview. To facilitate discussion and avoid misunderstandings a ‘design representation chart’ was created that showed 14 types of commonly used design representation, based on the work of Pei (2009). For each design representation a name, an example, and a brief description is provided – an example is shown in Figure 3.

Figure 3. Examples from the design representation chart (adapted from Pei (2009)).

In the ‘challenges’ section (Obj 2), interviewees were asked to describe any challenges they had experienced in preparing or using design representations within co-design sessions. In the ‘introduction to SAR’ section (Obj 3), the interviewees were shown a 2.5 min video that included a compilation of examples of SAR technology being used with objects such as shoes, a car seat, and packaging, as well as for user-interface research. The participants were then given an opportunity to ask any questions about SAR technology and the examples they had seen. In the ‘SAR requirements’ section (Obj 4) the interviewees were asked ‘If you were to start using SAR technology what would be your most important requirements?’. Prompting questions were also used if the interviewees were struggling to think of potential requirements. The second part of this section also involved the interviewees ranking the importance of a set of eight pre-defined requirements related to: the accurate rendering of materials, colours and finishes; the ease of set-up and use of the system; the visibility of the SAR model from various vantage points; the projection response speed in relation to movement (latency); the projection resolution; the requirements for the room in which the system is to be set-up; and the cost of the system. These requirements were defined based on the version 1 requirements that had previously been identified.

An audio recording was made of each interview with the permission of the participants. The audio data from each interview was first transcribed and then coded using the qualitative data analysis software package NVivo 10.

Using an approach based on content analysis/thematic analysis, a coding scheme was developed by reviewing the aims of the interview along with topics and questions considered most interesting by the partners involved in the SPARK system development. This led to a first iteration of the coding scheme. Modifications were made to the coding scheme throughout the early stages of coding. Changes included adding new types of design representation mentioned by the interviewees for which no suitable category was available. The final coding scheme is provided in the appendix along with a summary of the coding completed.

The coded data were analysed using several complementary approaches. First, by simply examining the number of statements that fall under each theme and the number of companies that have made reference to that theme at least once. Secondly, by reviewing the content coded against a particular theme and trying to identify common topics from across the companies. Thirdly, by making use of compound queries. For example, if the aim was to find instances when an interviewee mentioned challenges in obtaining suitable feedback from the client, then the transcripts were searched for content coded against both ‘Obtaining feedback’ and ‘Customers or client’.

3.2. Methodology for trials with prototype SAR system

An early prototype SAR system was created after the completion of the interview-based study. The main components and information flows of the system are depicted in Figure 4. The preparation for a co-creative design session starts with the preparation of the digital assets (logos, images, icons etc.) that the designer might want to display during the session. These digital assets are uploaded to the Information System (IS) server using a web interface. A digital 3D model of the prototype must also be created to represent the surface of the prototype on which the digital content will be projected – which is known as a ‘UV map’. A PC computer is connected to the IS server as well as a single, high definition projector (Hitachi CP-WU8600) and an interaction tablet (Samsung Galaxy Tab S2 9.7”). Using a specially developed user interface installed on the interaction tablet, the designer is able to select any of the digital assets that were previously uploaded to the IS server and have them appear on the surface of a physical, plain white prototype, thereby creating a ‘mixed prototype’. The user interface enables the user to modify the digital assets in terms of their position, size and orientation. The digital assets can include items relevant for packaging design (e.g. logos, product images, marketing claims etc.), and can also include items relevant for product interaction design (e.g. buttons, dials, LED lights etc.). Background colours can also be applied to the mixed prototype. Further details of the technical implementation of the SPARK SAR system can be found in Morosi et al. (2018).

Figure 4. SPARK platform architecture: main components and information flows.

Four trial sessions were organised with the industrial partners of the SPARK project. The first two sessions were conducted in the laboratories of Politecnico di Milano, whilst the second two sessions were completed at the laboratories of Grenoble INP, with an almost identical SAR installation and very similar room layout. The realism of the sessions was maintained through the involvement of professional designers, working on real, live projects, and involving real clients were possible. Where it was not possible to involve real representatives from the client organisation, people were recruited to participate in the sessions as representatives of end users. The sessions completed with Artefice designers were packaging design sessions, focused on the layout and colour scheme of food product packaging, whilst the sessions completed with Stimulo designers were product design sessions, focused on the selection of colour, materials and finishes as well as the layout of key user-interaction elements for small electronic devices. Further details of the products, tasks and participants for each of the sessions are presented in Table 4, whilst Figure 5 shows examples of the packaging and product design concepts developed during the sessions. Video recordings of each of the trial sessions are provided in the supplementary materials for this article.

Table 4. Summary of the participants, products, and tasks for the trials with early prototype.

Session	Participants	Product and task
Session 1 (POLIMI)	3 Artefice designers 1 Artefice client manager 2 Clients (real)	Two baby food products Tasks: Define the layout and colour scheme of the packaging
Session 2 (POLIMI)	2 Stimulo designers 1 End user (not real)	Smart fitness product to monitor performance when using gym equipment Tasks: Define the colours, materials, and finish of the main housing.
Session 3 (GINP)	1 Artefice designer 1 Facilitator 3 End users (not real)	Tomato cooking sauce Tasks: Define the layout and colour scheme of the packaging
Session 4 (GINP)	3 Stimulo designers 1 End user (not real)	Hand held device for assessment of human exposure to electromagnetic fields Tasks: Define the colours, materials and finish of the main housing. Define the layout of the main user-interaction features (lights, buttons, and speakers)

Figure 5. Examples of concepts generated during the packaging design sessions (left) and product design sessions (right).

After each session, semi-structured interviews were completed with the representatives from Artefice or Stimulo to obtain their feedback on the session and their experience of using the SAR technology. Audio recordings were made of the interviews and selectively transcribed to identify the SAR technology requirements that were mentioned.

4. Results and discussion

In the following sections the results from the interview-based study and the trials of the prototype SAR system are presented, in relation to the two research questions.

4.1. What challenges do design practitioners face when creating and using design representations within co-design sessions?

In the following subsections we present five important challenges that practitioners face when creating and using design representations within co-design sessions that were identified from the interview-based study.

4.1.1. Time and cost

The time and cost to create design representations were two related challenges that five of the companies mentioned. One of the contributors to cost identified by the participants is the number of similar design representations that have to be created in order to show variations in colour, material and finish options.

Significant costs are incurred to produce very high quality, detailed design representations such as photo-realistic renders, appearance models and pre-production prototypes. Two companies stated that they sometimes spend in the region of €5,000–10,000 to produce one appearance model. Company G stated that they spend around €5.5 million on creating models for product development, including around €500,000 per year on producing appearance models. These findings emphasise the relevance and importance of the aforementioned ‘economic principle of prototyping’ (Lim, Stolterman, and Tenenberg 2008).

4.1.2. Misinterpretation of design representations

Five companies mentioned the problem of misinterpretation of design representations by clients or other stakeholders. For example, judging the size and scale of a product that is represented through sketches or virtual models appears to be a common challenge and source of misinterpretations.

Another frustration mentioned was that clients and end users will often evaluate design representations as if they are the final product and expect all aspects of the design representation to be a perfect representation of the final product, even though it may be an early-stage design representation.

4.1.3. Selecting the fidelity level for the design representation

Another challenge identified was selecting the level of fidelity of the design representation. Interviewees at two companies identified that a trade-off often exists when making this decision. Clients sometimes struggle to provide feedback on design representations that do not look or function exactly like the final product, which implies a need to create very detailed, accurate models (high fidelity) in order to obtain good-quality feedback. On the other hand, presenting this type of very detailed model early in the process can limit creativity according to more than one interviewee. This challenge links to the finding of Söderman (2005) that the level of fidelity of a design representation can impact on the level of understanding of the concept that is portrayed by the representation.

4.1.4. Communicating how products function

One key limitation of virtual representations noted by two companies was the ability to communicate to clients how products function and move, and how they are assembled. They felt that this issue could be addressed more easily with physical design representations.

4.1.5. Obtaining feedback on user interfaces

A final challenge that was noted by three companies was the difficulty in creating design representations that would allow for feedback on user interfaces and points of user interaction (such as buttons and dials).

Previous work has attempted to address the challenge of user interaction with X-reality prototypes. These include Barbieri et al. (2013), who have shown how AR technology can be used with reconfigurable physical prototypes that feature real dials and switches to offer more haptic realism for usability studies.

In conclusion, the interviewees were able to identify a number of challenges that they currently encounter when creating and using their normal design representations. The significant cost of creating design representations, particularly ‘appearance models’, was an interesting observation. This topic does not appear to be discussed in great detail elsewhere in the academic literature. This suggests that there may well be a case for technology or tools that can help to reduce the cost of creating design representations for co-design sessions.

4.2. What requirements do practitioners have for SAR technology?

The first iteration of the requirements for SAR-based design representations for use in co-design sessions were defined through discussions with the industrial partners within the SPARK consortium and with potential suppliers of the projection and tracking technology. The requirements, presented in Table 5, relate to the basic functions of the system (visualisation and tracking of the SAR model), the practical constraints (size of the room required for the set-up), and user safety and comfort (light source safety, projector noise). The subsequent iterations of the requirements are discussed below.

Table 5. Summary of the requirements SAR system requirements identified – later versions include all the requirements listed in earlier versions.

Type of requirement	V1 requirements	V2 requirements	V3 requirements
Visualisation	Accurate colour rendering Sufficiently high image resolution to enable legible 11pt fonts Sufficient image brightness to be visible in normal office ambient lighting Good viewing angles Projection volume of at least 0.5m^3	-	Realistic radiance from SAR prototype Ability to render black on SAR prototype Realistic rendering of transparent/translucent materials Realistic rendering of surface finish (matte, glossy, metallic) Higher projection resolution required
Tracking	Low latency Good stability	-	-
Installation and set-up	Be easy to set-up	Portable system	-
User interface	-	Specification of colour by RGB/Pantone value Direct manipulation of assets on the SAR model	Support precise spatial placement and alignment of elements Support rotational placement of elements at common angles (0, 45, 90 degrees) More efficient selection and manipulation of elements (fewer clicks) Maintain/control scale of textures when applied over a large area Revert to an earlier version saved during the session Efficient selection and manipulation of elements (fewer clicks) Copy-paste function Undo function
Miscellaneous	System hardware cost under 15,000 euro Low system noise (no more noise than a standard projector/laptop fan) Safe lighting source	Achieve realistic surface finishes in the preparation of the physical model Support for design of user interfaces	Easy comparison of screenshots taken during the session A plug-in for standard graphic design software that allows you to create the graphical elements for use in SPARK

4.2.1. Results of the interview-based study

The second iteration of the requirements was defined through the interview-based study with design practitioners from 11 companies. From the analysis of these interviews, it was found that the most commonly mentioned category of requirement was to do with the accurate rendering of colours, materials, and finish. Many participants felt that this was the most important requirement, going as far as to say that they would not buy a SAR system if it could not perform this function satisfactorily. Accurate colour rendering is a major challenge for SAR projection technology. It will also require some user-interface feature to input and output colour selections as several designers mentioned that they specify colours using colour systems, such as the RGB, CMYK, and Pantone® Colour Matching System.

Projection resolution was often closely linked to accurate rendering of colours, materials and finishes as both are required to give a high-quality rendering.

Five of the companies stated that it was important to minimise the latency – i.e. how quickly the system can update the projected content in response to movements – to ensure a realistic experience for the user. A similar requirement has also been noted by Verlinden (2014), who suggested a system refresh rate of at least 10Hz to maintain the realism of the SAR rendering.

User-interface and user-interaction design was a major part of the design activity for six of the companies interviewed. For these companies, the possibility to display elements of a user interface on a physical, three-dimensional model seemed exciting.

For the companies that focus less on user interaction and more on the aesthetic qualities of the product, their interest was in being able to obtain feedback on the texture and feel of the model surface. This requirement would be important to consider if developing SAR technology for use in the later stages of the product development process, when aesthetic models and pre-production prototypes are constructed with the aim of offering realistic materials and finishes.

Of the general system requirements, the sales price of the system was the most frequently discussed. However, interviewees found it difficult to rate the importance of price without having any idea of the final specification of the SAR technology or what the retail price of a system might be. Most companies did not place any limit on the system cost but would be looking to save either development time or cost in comparison to their existing approach. For some interviewees there was also an expectation (or hope) that using the SPARK system would reduce the costs of creating physical models.

Within discussions about the system set-up, preparation and usability, the topic of portability often emerged. Ten out of 11 companies stated that they sometimes hold co-design sessions somewhere other than their own offices. Of those, six companies went on to express an interest in having a portable system that could be taken to a client’s site. This finding is aligned with the ‘ease of mobility’ requirement proposed by Verlinden (2014).

Another common requirement was to ensure good usability of the system. An important aspect of usability was the speed and ease of being able to modify colours, materials and finishes or switch between pre-defined complete concepts with a single button click.

Some interviewees liked the idea of being able to make changes to design concepts through direction manipulation of the SAR model, for example, changing the position of an image by dragging it across the surface of the model. There were several comments suggesting that the system should be very simple to use so that a wide variety of people, including clients and end users, could use the system without the need for training.

4.2.2. Results of requirement ranking activity

The final part of the interview-based study required the interviewees to place a list of requirements (based on the first iteration of requirements) into rank order of importance. A summary of the ranking by each of the companies is provided in Figure 6. A ranking of ‘1ʹ indicates the highest importance. In some cases, interviewees placed two or more requirements at the same level of importance. In those cases, the points were distributed evenly amongst those requirements (i.e. two requirements placed in the 4^th rank position receive (4 + 5) /2 = 4.5 points each). The ‘Product’ column shows the mathematical product of the rankings from each company for that particular requirement. The ‘Overall rank’ column is based on the product score, where a low product score means it is more important. Company F did not participate in the ranking activity as the interviewees wanted to see the first prototype system before providing feedback.

Figure 6. Ranking of requirements with break down by company and sorted by overall ranking.

The result that accurate rendering of materials, colours, and finishes received the highest overall ranking is consistent with the qualitative analysis and confirms the importance of the ongoing technology development activities in these areas through studies such as Menk and Koch (2013) and Park et al. (2015). Whilst ‘projection resolution’ was often discussed in the same context as ‘accurate rendering of materials, colours and finishes’, the former has been ranked significantly lower (6^th position). This may be because the interviewees were not sure about the distinction between these two requirements or because the availability of ‘high-definition’ projectors means that they do not expect resolution to be a problem.

The high ranking of ‘ease of set-up and use’ seems to be based on the assumption that the SAR technology would be available as a portable system, as several of the participants mentioned the importance of being able to set-up the SAR equipment quickly and reliably at a client’s office. This concurs with the ‘installation efficiency’ proposed by Verlinden (2014) in which he suggested that the set-up of the system should require no more than 15 min.

The visibility of a model from various vantage points came third in the overall rank, although there are significant variations in the individual rankings by the companies ranging from one to seven. This appears to be due to different interpretations of how the system would be used.

4.2.3. Results of the trials with prototype SAR system

Analysis of the interviews completed with the designers after the trials with the prototype SAR system identified a variety of new requirements for the SAR system. Table 5 shows how the complete set of requirements developed iteratively. As a reminder, Version 1 of the requirements was based on discussions between technical development staff from within the SPARK consortium and designers from the industry partners of SPARK. Version 2 shows the new requirements identified from the interview-based study with design practitioners external to the SPARK consortium. Finally, Version 3 shows the new requirements identified from the trials with the prototype SAR system.

Most of the new requirements identified in Version 3 relate to the quality of the SAR visualisation and the user interface. Concerning visualisation, some designers complained that the SAR prototype was too bright and requested more realistic radiance and lighting effects. The work of Radkowski and Linnemann (2009) concerning the use of a ray tracing method to enhance AR model lighting effects might help to address this issue. There were also requests for better rendering of surface finishes (e.g. matte vs glossy) and translucent materials. The work of Park et al. (2015), who describe a function that enables more accurate rendering of a material’s optical properties, could help to address this issue.

The requirement to render black on the SAR prototype is not technically possible with SAR technology but the appearance of ‘black’ is enhanced by having very low ambient light levels. Unfortunately, this would conflict with the V1 requirement for the SAR model ‘to be visible in normal office ambient lighting’.

The issue of projection resolution re-emerged within the V3 requirements, having been given a relatively low importance rating by the participants in the interview-based study (who had not seen the prototype system). This demonstrates the value of the iterative approach to requirement definition as it has allowed the system requirements to be refined as the potential users have developed a better understanding of their own requirements.

Concerning the user interface, a total of eight new requirements were identified, mostly related to the need for greater control and efficiency in the placement of digital assets on the SAR model. Several designers suggested that the user interface should mimic the behaviour of some of the commonly used CAD and graphic design software packages.

Finally, two new requirements emerged related to the IS of the SPARK platform. The first was to be able to easily compare screenshots taken during the session. This requirement is related to the ‘recording capabilities’ requirement proposed by Verlinden (2014) but is more focused on quick comparison of the main concepts to emerge from the session rather than reviewing a recording of the whole session. The second was for a plug-in to standard graphic design software in order to automate the process of exporting digital assets and preparing them for use with the SAR system.

Beyond these specific requirements identified from the interviews, the general feedback from the product designers (Stimulo) was that they see potential in using SAR technology for creating design representations as, in their opinion, it encouraged a much broader exploration of ideas and led to some surprising new ideas. However, they were frustrated by limitations of the user interface and the lack of support for user-interaction design. For the packaging designers (Artefice), there was again strong interest in the SAR technology as they felt that it could significantly reduce the number of iterations in the design process linked to the client review and approval activities. However, they noted that the accuracy of the colour rendering was not sufficient for their requirements and they were frustrated by the user interface of the system. Some level of user frustration is of course inevitable with a prototype system. The feedback and new requirements identified will be used to inform future developments of the SPARK system.

5. Conclusions

This study set out to address two research questions related to the use of design representations within co-design sessions. The first research question was, ‘What challenges do design practitioners face when creating and using design representations within their current co-design sessions?’ Four main types of challenge (or opportunity) were identified and are summarised here.

5.1. The time and cost of preparing design representations

The significant cost of creating current design representations, particularly ‘appearance models’, identified within this study should be encouraging for SAR technology developers as there will be significant commercial interest in technologies that can significantly reduce prototyping costs. This is a useful insight into a topic that has received little attention within the academic literature and yet appears to be a significant contributor to the overall time and cost of the product development process.

5.2. Misinterpretation of design representations by clients or other stakeholders

As co-design sessions will often involve participants with limited familiarity with design representations and knowledge of design (such as clients and end users), the scope for misinterpretations is likely to be higher than when dealing with a group of design professionals. If developers of AR and SAR systems for co-design want to help reduce this type of problem then the recommendation from this study and the literature is to create high-fidelity representations, as this has been shown to reduce misinterpretation errors (Hannah, Joshi, and Summers 2012). However, this may conflict with other requirements for the system, such as encouraging creativity – as discussed in the following point.

5.3. Level of fidelity

The finding that stakeholders struggle to provide feedback on low-fidelity design representations is consistent with the previously mentioned studies by Hannah, Joshi, and Summers (2012). The feeling from designers that using high-fidelity design representations early in the design process can limit creativity is consistent with a study by Robertson and Radcliffe (2009), who suggested that ‘…when a detailed CAD model is displayed, it can convey an illusion of completeness that tends to discourage creative thought in a group situation’. This implies that practitioners should carefully select the level of fidelity of their design representations according to the purpose of the session (i.e. use low-fidelity design representations for sessions where the goal is idea generation and reserve high-fidelity design representations for sessions where the goal is concept evaluation, validation and filtering). Getting the level of fidelity right for creative sessions seems to be a particularly difficult challenge. The requirement for AR or SAR technology developers would be to support both low- and high-fidelity representations so that the practitioner can select what is most appropriate for each individual co-design session.

5.4. Communicating principles of operation and obtaining feedback on graphical user interfaces

Explaining how products work and how they will be used was identified as one of the challenges of co-design. Designers were interested in the potential of a SAR system to enrich this type of discussion within co-design sessions through the use of annotation, animation, revealing of hidden details etc. Akaoka, Ginn, and Vertegaal (2010) have already begun to investigate the potential for SAR prototypes within interaction design.

The second research question was, ‘What requirements do design practitioners have for a SAR system that supports co-design sessions?’ Through three activities (internal discussions, interviews with practitioners, and trials with a prototype SAR system) the requirements have been iteratively developed and refined. Trials with a prototype SAR system showed that this technology is of interest to both product and packaging designers and identified a significant number of new requirements at a level of granularity that had not emerged from earlier activities. Whilst there was some frustration amongst the designers related to the user interface and the lack of certain features, the general response was positive and is a promising sign for the future of SAR technology in this co-design application. Whilst the focus of this study was on the product and packaging design domains, it is likely the requirements will be common to other design domains. There will of course be some differences in the SAR system requirements for other domains. For example, in vehicle design the need for object tracking may be redundant, but there will be extra demands in terms of the projection volume of the system if a 1:1 scale model of a complete vehicle is to be created.

It was noted earlier that much of the research in which design-representation technology is developed tends to describe the new functionalities/capabilities of the system being developed but without providing details of the user requirements or needs that the system is trying to fulfil. The requirements presented in this paper help to address this gap and will therefore be of interest to technology developers (including our own SPARK project team) working on systems to support co-design sessions.

Supplemental material

Supplemental data for this article can be accessed at https://doi.org/10.1080/15710882.2018.1546319

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the H2020 LEIT Information and Communication Technologies [688417].

Appendix

The following table provides a summary of the coding scheme used for the interview-based study. The ‘Companies’ column indicates the total number of companies in which a particular theme was identified at least once. The ‘References’ column indicates the number of individual references made to that theme across all companies.

Table

Name of topic headings and individual themes	No. of companies mentioning a theme	Total no. of references made to that theme
Characteristics of potential users
Designers and engineers	7	13
Knowledge of design and design process	4	6
Marketeers	4	5
Other functions	6	12
Sales people	2	5
Senior management	3	6
Characteristics of the environment
Lighting	9	14
Own site or other site	11	31
Size	6	9
Creating design representations
Cost to create a design representation	11	28
Process to create a design representation	10	37
Time to create a design representation	10	21
Potential users and stakeholders
Consultancy or in-house team	10	28
Customers or Client	9	55
End users	7	36
Number of users in a session	7	11
Value chain partners	3	7
Properties and affordances of design representations
Colour, Material and Finish	9	28
Graphics and text	3	3
How the product works or is assembled	4	9
Shape and form	3	3
Size, volume, or mass	5	8
User interaction or user interface	8	12
Requirements
Doubts and concerns about SAR technology	11	28
Information management	11	26
Interaction	7	20
Portability	7	9
Preparation time or effort	7	17
Price	10	12
Projection	10	35
Recreate the context and environment of the product	1	1
Safety	0	0
System noise	7	8
Target object	10	19
Tracking	8	12
Usability of system	8	12
Scenarios of use
Evaluate and filter	8	22
Generate ideas	9	16
Obtain feedback	11	47
Other specific applications of SAR technology	8	33
Types of design representation
3D print	9	19
3D renderings	10	29
Appearance model	5	12
Augmented Reality or Virtual Reality	2	4
CAD Drawings	9	28
Concept drawing	8	14
Development sketch	9	26
Explanatory sketch	3	3
Functional concept model	4	10
Interface mock-up	3	10
Mood boards	4	5
Movie or animation	3	5
Multi-view drawing	1	1
Post-it note	2	2
Pre-production prototype	5	9
Scenarios and storyboards	4	10
Shape model	10	27
Shelf mock-up	2	3
Simulation	3	5
Technical sketch	1	1
Working prototype	2	2
Value of design representations
Challenges and costs	11	78
Opportunities and benefits	9	33