The development of a Design Research Canvas for data practitioners

Abstract

This paper sets out the development of a Design Research Canvas through engagements with data and decision support practitioners. This is in direct response to the lack of tools for practitioners in crossing the practice research divide. Designed, built and evaluated within the context of an Executive Education programme, version 1 of the canvas is demonstrated. Initial results depict positive utility and effectiveness in completing a Design Research project. In addition, the evaluation of version 1 guided the development of an improved version 2 of the canvas. Moreover, the canvas itself provides a useful aid for practitioners, researchers and teachers involved in Design Research. Furthermore, being itself an example of Design Research, this study highlights some key insights on the process, such as early stage evaluations.

Keywords:

• Design research
• Practitioner-researcher
• Design Research Canvas

1. Introduction

This paper details the development of a Design Research Canvas using Design Research. With the aim of fulling the needs of all IS community members (practitioners and researchers) the first version of the Canvas focuses on data practitioners at an executive level. Within this context, the paper describes: (i) the motivation for developing the artefact, (ii) the first iteration of design, build and evaluation of version 1, (iii) the learnings and contributions from the first iteration, and (iv) initial design of version 2 of the canvas.

2. Literature review

It has been argued that ‘IS research needs to get closer to wealth creation rather than reporting’, where wealth creation or value creation is the creating of a consumable product/service (Desouza, El Sawy, Galliers, Loebbecke, & Watson, 2006, p. 348). In the past so much effort was placed in debating the rigourous merits of IS research through topics such as: qualitative versus quantitative methods (Lee & Hubona, 2009), or positivist versus interpretivist approaches (Fitzgerald & Howcroft, 1998; Weber, 2004) the relevance of IS studies to practice was seen as a secondary importance (Benbasat & Zmud, 1999). As a result, the balance between rigour and relevance in IS research was heavily weighed on the side of rigour. A number of other factors added to this imbalance, such as: (i) the lack of a cumulative research tradition, (ii) the fast paced dynamic environment of IS, (iii) the limited practical exposure of IS academics in the areas they research, and (iv) the restrictions on academics undertaking action in their research (Benbasat & Zmud, 1999). However, in an effort to redress this imbalance IS academics have explored a spectrum of solutions that range from: including an additional step to existing research approaches (Rosemann & Vessey, 2008), to sourcing industry sponsored research (Desouza et al., 2006) to implementing new research approaches such as action research (Susman & Evered, 1978) or design research (Hevner, March, Park, & Ram, 2004). While the first two solutions provide sound guidance in becoming more relevant, they are but incremental changes to the current way of thinking and operating for IS researchers. The last solution sets out more substantive changes, as the approaches enable the researcher to become more involved in the practical nature of their areas of expertise and also provide operationalization guides which put real world problems at the focal point of academic research (Davison, Martinsons, & Kock, 2004). While these approaches have met with a huge degree of popularity they still only view the solution of becoming more relevant as a function of the academic. In addition, the utilisation of these approaches by academics as a structuring mechanism for studies rather than a problem solving protocol have meant the initial bridging of the research practice divide are primarily superficial. As a result, there has been a clear call to action in understanding ‘how to more effectively structure and shape the way that practitioners participate in IS research’ (Desouza et al., 2006, p. 343).

However, given the fact that this call for research was made a decade ago, it has been the direct experience of the authors, that it has not been sufficiently fulfilled. This experience incorporates directing an executive education programme of which a key focus of the programme is to enable the participants become Practitioner Researchers through Design Research. Nonetheless, outside of Mathiassen and Sandberg (2013), the amount of guidance or tools in achieving this objective are yet to be developed. With this primary motivation and key driver, the focus of the paper to is to detail the development of such a tool (Design Research Canvas), with the aim of supporting our objective of enabling the participants of the executive programme to become Practitioner Researchers. The benefit of doing so will not just apply to the context of the executive education course but also to the Decision Support community and researchers at all levels.

Facilitating Decision Support practitioners in becoming Practitioner-Researchers is a worthwhile objective, as practitioners are in a better position to identify relevant problems that are difficult to solve and have real organisational impact. In contrast, with academic researchers primarily abstracted from the real world, it has been advised they qualify their research objectives with practitioners to ensure relevance (Rosemann & Vessey, 2008). Moreover, as highlighted by Davison et al. (2004, p. 68) ‘the researcher seldom has complete control over interventions’. In comparison, practitioners will always have areas where they possess a certain degree of control to dictate the course of a project and guide its implementation to completion. This is a major advantage as it provides bounded areas that are a rich bed for conducting research with full direct access. These bounded areas also provide protection from the risk of losing control over the environment underpinning the research (Davison et al., 2004). In addition, immersed in the realities of their work, practitioners experience relevant and wicked problems that need to be solved. Due to their stubborn nature these wicked problems are a rich source for valuable research for both the academic and research community (Buchanan, 1992).

While the Practitioner-Researcher is not a novel concept, it is only recently that the IS domain have begun to make some progress on the topic (Mathiassen & Sandberg, 2013; Taylor, Artman, & Woelfer, 2012). More specifically, previous to these studies the focus has been on making the researcher more relevant through methodologies such as Action Research and Design Research. In contrast, Mathiassen and Sandberg (2013) is the most explicit in its examination of the issue and provide a robust template for uncovering key insights into how a practitioner can bridge the academic/practice divide through Collaborative Practice Research. Yet, given the experience of the authors there is a need for more tangible tools if practitioners are expected the bridge the divide between practice and research.

In describing the development of the Design Research Canvas, this paper outlines the executive programme which provides the context and motivation of the research as well as the source of the data that is analysed. From this, the design of the canvas is detailed starting with an outline of the challenges faced by practitioners. A description of the development of the canvas proceeds in direct response to these challenges and is followed by the evaluation/utilisation of the canvas and corresponding results and contributions.

3. Method

The focus of this study is the development of a Canvas. The methodology implemented is Design Research and follows the four step process outlined in Table 1. The backdrop to this study is the MSc in Data Business, which provides practitioners with the capabilities to have new conversations within the business around data. Further detailed in (Figure 4), the MSc is split into two parts: Part 1 - 30 credits over six months and Part 2 – 60 credits over 12 months. The programme focuses on IS areas such as data modelling, agile development, innovation management and enterprise data management. At the end of Part 1 a blueprint is required in which the practitioners apply the knowledge gained throughout the programme to challenge their use of data, to highlight solid opportunities for improvement and propose a plan of action for those opportunities within a realistic timeframe.

Table 1. Phases of Design Research and research approach utilised by the practitioners.

Research Approach	Design Guidelines (Hevner et al., 2004)	Design Science Process (Peffers et al., 2006)	Action Design Research Process (Sein et al., 2011)
Problem	Problem Relevance	Problem Motivation and Relevance	Problem Formulation
		Objectives of Solution
Design and Build	Design as an artefact	Design and Development	Building, Intervention, and Evaluation
	Design as a Search Process
Evaluation	Design Evaluation	Demonstration
	Research Rigour	Evaluation
Contributions	Research Contributions	Communication	Reflecting and Learning
	Communication of Research		Formalisation of Learning

Starting six months after the completion of Part 1, the objective of Part 2 is aligned with Part 1 but focuses on developing the practitioners’ research capability in conjunction with their ability to execute data projects. In particular, the practitioners learn how to implement the applied research methodology of Design Research; their skills in literature reviews and academic writing are developed, and they are exposed to research seminars around interesting topics in the data domain. In addition, the practitioners further develop their capability to implement value driven/agile projects as they internalise the value of design thinking and data driven design. Finally, throughout Part 2 the practitioners utilise their applied research skills to solve a problem for their organisation, while also making a contribution to research, which is documented in a 16,000 word final project report and poster of the project.

Through this process the authors of the paper have experienced at first hand the challenge participants face in implementing Design Research, which in-turn became the key motivation for developing a tool to facilitate the journey to becoming a Practitioner Researcher. More specifically, the data collected to detail and support the problem and research gap was primarily sourced from the first cohort of the MSc.

This included: the analysis of three key assignments (Information Supply Chain, Blueprint, and Final Project Report) which totalled over 25,000 words of data (for each participant) all relating to the implementation of Design Research. In addition, a survey was also completed by the participants after they had finished the programme with the aim of supplementing their submissions with key insights about their academic journey and research capability. Finally, the participants engaged on a one-to-one basis with the mentors outside of class periods. The focus of these engagements was primarily on the applied research project. In these discussions all aspects of the Design Research methodology were covered, with particular emphasis of how they applied it to their project. These were an excellent insight into how the practitioners were progressing with the programme, but also into how they were coping with becoming a Practitioner-Researcher. The mentors met with all of the practitioners, which amounted to over 100 h of one-to-one sessions and was documented in terms of written notes, pictures of white-boarding sessions and follow-up emails. With a strong understanding of the challenges, a canvas was developed at the end of the first cohort’s tenure. The evaluation of the canvas was later focused on the second cohort. The evaluation came from the first 8 months of their Part 2 journey. During that period the participants were exposed to the canvas but were not explicitly told to utilise it as part of the course. A key evaluation date occurred in January 2016, when each of the participants presented the progress of their projects to the class and mentors. This gave the authors an opportunity to measure the adoption of the canvas from a number of aspects.

4. The design research canvas

With the objective of developing a tool to enable practitioners become Practitioner-Researchers through Design Research the first decision to be made was what type of tool should be developed. The outcome of this decision was to develop a canvas. The medium of a canvas has been popularised by the Business Model Canvas (Osterwalder & Pigneur, 2010) and has been a key tool in developing a common language around what a business model entails. The objective of the Design Research Canvas aligns with the success of the Business Model Canvas but in the area of Design Research. Tsoukas and Chia (2002, p. 579) view such tools as discursive templates ‘which enables a novel way of talking and acting’ and ‘make it possible for organisational members to notice new things, make fresh distinctions, see new connections, and have novel experiences’. This form of artefact is also aligned with ‘Design Thinking’ and ‘Visual Thinking’ (Ware, 2010) in enabling its users to create mental models that can be shared, communicated or collaborated with peers.

4.1. Format and structure of the artefact

Having decided the format of the actual artefact, its underlying objective of facilitating the participants in understanding Design Research needed to be incorporated in its development. From the first cohort of the MSc, one of the issues they had was the multiple definitions of Design Research and differing steps to each definition. In response to this need Table 1 provides a synthesis of Design Research methodologies in the literature and highlights a simplified four stage Design Research process which becomes a core structuring component in the canvas (see Figures 1 and 3 for versions 1 and 2 of the canvas).

Figure 1. The Design Research Canvas - version 1 (Iteration 1).

Figure 2. Examples of canvas photos used in the presentation.

Figure 3. Version 2 of the Design Research Canvas.

Figure 4. Timeline of the first cohort for the MSc including key events, assignments, and an aggregate view of practitioners’ Practitioner Researcher journey.

4.2. Incorporating dual role of the Practitioner-Researcher

Another key issue to be solved by the canvas was the difficulty in understanding the difference between the practitioner and researcher aspects in the process. To ensure the users of the canvas would be clearly able to internalise the dual role of the Practitioner-Researcher and also differentiate between researcher and practitioner tasks; each of the four stages of Design Research was split in two. One side dedicated to the practitioner with the other dedicated to the researcher. In addition, the focus of the practitioner side was further focused with a label for the artefact (title) and study title for the researcher side. Further facilitating the participant’s journey in becoming a Practitioner-Researcher, it was deemed that leading questions were needed for each section of the canvas. These questions are outlined in the proceeding text.

4.2.1. Problem

The first step in the Design Research process is the most important step in defining the relevance of the research. As noted by March and Storey (2008) the key focus of Design Research is to improve the performance of the organisation which, translates to a focus on organisational problems that are worth solving (Peffers et al., 2006). In addition, during this phase the need to describe the problem through a clear definition is also a requirement, as well as making sure you have reached the root cause of that problem. Moreover, when completing a Design Research study there needs to be a demonstration that the problem is also worth researching. This can be achieved by demonstrating the lack of existing solutions to the problem in the extant knowledge-base (March & Storey, 2008). This demonstration can come from a literature review highlighting a paucity of knowledge, or indeed a call for research on the problem. In summary, from the practitioners’ aspect the problem must be worth solving from an organisational perspective, and (iii) worth researching from an academic perspective.

4.2.2. Design and Build

During the Design and Build phase the key focus is to build an artefact to solve the defined problem. An artefact can take the form of a: (i) construct, (ii) model, (iii) method, or (iv) instantiation (March & Smith, 1995). The need to focus on the artefact is already incorporated above. Yet, when building the artefact there is always a need to ensure that the design and building of the artefact is well organised. In contrast, from a researchers’ perspective the focus is more on the need to document the process, which will provide the necessary data to theorise. As a result, the practitioner is prompted with the question ‘Well organised?’ and the researcher with ‘Well documented?’.

4.2.3. Evaluation

Just as the Design and Build phase links directly back to the Problem phase, the Evaluation phase focuses on the: (i) utility, (ii) quality, (iii) validity, and (iv) efficacy of the artefact developed (Helfert, Donnellan, & Ostrowski, 2012; Hevner et al., 2004). In Peffers et al. (2006) definition of Design Research, there is a preceding step to the Evaluation phase that focuses on the act of demonstrating or implementing the artefact. The data arising from this demonstration is then evaluated to highlight how well the artefact solves the problem and if more iterations are needed. Detailing the types of evaluation Hevner et al. (2004) lists observational, analytical, experimental, testing and descriptive. In the canvas the direct evaluation of the artefact is located on the side of the practitioner and incorporates a high-level prompt to articulate the role of the artefact in problem solution. However, from a research perspective the value of the work may not be in the success of the artefact but the learnings that were created through the process. To capture this on the canvas the user is prompted to reflect on the process and record any learnings experienced.

4.2.4. Contribution

Gregor and Hevner (2013) identify three levels of contributions ranging from: Level 1 - very specific and situated implementations of artefacts, Level 2- nascent design theory or knowledge as operational principles/architecture, Level 3 - well-developed design theory about embedded phenomena. Moreover, in developing the concept of Practice Research, Goldkuhl (2012) notes that a study that conducts a situational inquiry, driven by a local problem will generate situational knowledge (contribution to local practice). Nonetheless, theorising the results of the situational inquiry generates two other contribution types. Contributions to the general practice community can be made along with contributions to the scientific body of knowledge. In formatting the contributions for dissemination, Gregor and Hevner (2013) outline a template for publishing Design Research projects to research stakeholders, but in line with Goldkuhl (2012) communication of contributions can also be made to: (i) the general practice community in the form of a white paper or trade press articles, and (ii) to local practice in the form of internal presentations or meetings. To simplify this for the users of the canvas, one question is posed on both sides: So What? This gets the users to think about the impact they have had as they complete an iteration.

4.3. Connecting the dual roles

Having explicitly depicted the dual roles on the canvas, it is still possible to overlook how the two aspects connect or integrate. With this in mind a third column was added to canvas and placed centrally between the practitioner and researcher column. Within this column, a key activity for each Design Research stage was listed. For the Problem stage, a Problem Statement is required. Complimented with the left and right side of the canvas it provides a very clear focus as to what the objective of the research is. For the Design and Build stage, the need to map both the research approach to the project management approach is outlined. From the first cohort, this was primarily achieved by creating a timeline that linked the Agile methodology to Design Research. Completing such an activity makes it abundantly clear how much of a fit there is between the two methodologies. Another key reason for this activity is that it gets the user to think about the number of iterations they have in the study. For the Evaluation stage, there is a need to outline how the artefact is demonstrated. Again this gets the user of the canvas to plan the context of their evaluation(s) and visualise its alignment with the artefact. Finally, for the Contributions stage, the form in which the contributions are communicated need to be outlined. From a practitioner perspective, this could include white papers or indeed a presentation to the local organisation, or an industry body. From a researcher perspective this would include the type of academic conference or journal which was used to communicate the contributions.

4.4. Design Research Canvas – version 1

Incorporating all the design aspects outlined the previous section, Figure 1 outlines version 1 of the canvas. The only addition made to canvas that was not previously discussed is the circular image at the centre. This is to visually represent the iterative and cyclical nature of the canvas. Without the image, the process can be interpreted as being very procedural or waterfall in nature. It is envisioned that the preferred process in populating the canvas is through the use of Postits that can be easily positioned and repositioned. Through this process the users can collaborate with their academic/industry peers as well as communicate their progress with supervisors.

5. Canvas evaluation

The evaluation of the canvas took place with the second cohort of the MSc. Initially the group of 15 were given an A1 copy of the canvas in September/October 2015. At that point in the time the participants had completed Part 1 of the MSc (6 months) and were 5 months into Part 2 of their journey, which is 12 months long. The canvas was not central to any assignment and was not made compulsory for the class. This left the decision fully up to each participant on whether they would adopt the canvas or not (based on its merits). The key period of evaluation took place on the 7th of January 2016 when each participant¹ presented the progress of their project to the class and mentors. This was a graded presentation and typically lasted 15 min with 5 min Q&A. No set format or structure was prescribed for the presentation beyond the need to detail their progress.

Of the 13 that presented 7 participants used the canvas in their presentation, leaving 6 that did not. This indicates over 50% adoption of the canvas. It is worth noting that the use of the canvas varied slightly across the presentations. Uses included:

•	at the start to provide context to how much progress was made
•	at the end to show what were the next steps
•	through-out the presentation and used as a structure for the presentation and progress report

Moreover, there were different methods in which the canvas was presented. In some cases, a photo snapshot of the canvas was used (see Figure 2). From these photos it was possible to see that some used different coloured Postits while other used all one colour. There was no meaning associated with the different colours but it did provide the option of having another dimension to the canvas.

Others recreated the canvas as a table and presented in that format rather than using the template above. This made it much easier for them to structure the presentation around the canvas and was seen as being more appropriate for PowerPoint. Finally, some used a screen grab of the canvas and overlaid text to describe their project. This points to a need to have different versions that are suitable for different forms of communication.

The A1 sheet is primarily suited to collaborative or personal use, which was the key focus during the design. However, one decision that needs to be made for the next iteration is whether a more horizontal version should be created to facilitate ease of communication through presentations.

Indicating the effectiveness of the canvas the grades were compared between the group that used the canvas and the group that did not. The results showed that the participants that used the canvas were on average 9% higher than those that did not.

Finally, during the evaluation of the canvas it has become clear that its ability to handle iterations needs to be improved. Iterations are a key concept of Design Research and while it was not overlooked in the first version of the canvas it was thought that a canvas could be used for each iteration. Having seen the canvas in action, it is now evident that such a process will detract from the effectiveness of visualising the entire project on one sheet.

It must be noted that while the data for the first evaluation may be viewed as being minimal, it is in-line with the type of evaluations undertaken for early artefact versions. For these versions, the need to observe the utility and validity of the artefact is of key importance. Later versions can be more rigorously evaluated with the confidence that the artefact does indeed solve the problem. In addition, these later evaluations can be focused more on the quality and efficacy of the artefact.

6. Contributions

Having gone through the first iteration of the canvas a number of contributions have been made. As the Design Research Canvas focuses on the facilitation of Practitioner-Researchers, the contributions from the process hits both practitioners and researchers. For instance, the primary contribution is the artefact (the canvas itself). This is the first Design Research Canvas to be developed and provides a useful discursive template for users to discuss and collaborate on all aspects of the Design Research process. While, the key focus of the canvas has been on the practitioner it also provides a platform for their research mentors/supervisors to communicate with them. Moreover, there is no reason why novice researchers cannot also use the canvas in guiding their Design Research studies. Indeed, this could be incorporated in the evaluation of the version 2 of the canvas.

In addition, not only is the canvas the first of its kind to be developed, it is also effective in achieving its key objective in enable researchers complete the dual role required when implementing a Design Research project. The Design Research Canvas can be used the compliment the frameworks outlined by Hevner et al. (2004), Peffers et al. (2006), and Sein, Henfridsson, Purao, Rossi, and Lindgren (2011). There is also no reason why the canvas cannot be used as a teaching aid and describing past Design Research projects.

Other contributions include the considerations that need to be taken into account when creating a canvas. From the evaluation it was found the canvas was appropriated in a number of different ways. In certain cases, the format of the canvas was not suitable and had to be modified. For instance, while the key focus during the design and build of the canvas was to facilitate Practitioners become Practitioner-Researchers, it was assumed that the collaborative aspect of the artefact would have been the primary method in achieving that objective. Yet, communication more than collaboration was seen as the key appropriation. In comparing this result to the Business Model Canvas (Osterwalder & Pigneur, 2010), it is now clear why there were many different versions (e.g. powerpoint, A4, A3, iPad app) and why the canvas needs to flexible enough to transfer across the key communication formats, such as: documents, presentations, workshops and the web.

This study being itself a Design Research project is also another contribution. The steps taken in the Design Research process as well as the format of its dissemination will be of interest to both academics and practitioners. Moreover, the inclusion of the educational offering (for data practitioners) highlights this paper as a key example of how academics can blend practice, research and teaching. In addition, while the context of the study is based in the data/decision support domain the contributions are valid to researchers and practitioners of the wider IS community.

Finally, taking into account the learnings from the evaluation, version 2 of the canvas was created (see Figure 3). While no formal evaluation of the canvas has taken place informal evaluations from experienced users of Design Research have provided positive feedback. In this version, all of the possible improvements that were identified have been included. With the confidence of knowing the canvas works, more focus was placed on improving the quality, utility and the efficacy of the artefact. Besides the inclusion of more questions within each section, the major difference is the move of Contribution section to the sides of the canvas. Now it becomes clearer that contributions can come at any time during the entire Design Research process. In addition, the labels of the sections have changed to being ‘Business Impact’ and ‘Academic Impact’. This further clarifies the focus of the contributions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Two participants were unable to present on the day.