Design principles for learning analytics information systems in higher education

ABSTRACT

This paper reports a design science research (DSR) study that develops, demonstrates and evaluates a set of design principles for information systems (IS) that utilise learning analytics to support learning and teaching in higher education. The initial set of design principles is created from theory-inspired conceptualisation based on the literature, and they are evaluated and revised through a DSR process of demonstration and evaluation. We evaluated the developed artefact in four courses with a total enrolment of 1,173 students. The developed design principles for learning analytics information systems (LAIS) to establish a foundation for further development and implementation of learning analytics to support learning and teaching in higher education.

ACCEPTING EDITOR:

• Pär Ågerfalk

ASSOCIATE EDITOR:

• Jonny Holmstrom

KEYWORDS:

• Design Science Research
• learning Analytics
• information Systems
• higher Education

1. Introduction

Advances in educational technologies and the digitalisation of education have generated increased interest in utilising learner behaviour data to provide process-oriented information to enhance learning and teaching (Nguyen et al., 2017; Noroozi et al., 2019). Researchers and educators have noted the potential use of big data and data analytics in higher education (Chaurasia et al., 2018; Daniel, 2015; Picciano, 2012). Hathaway (1985) suggested that “the main barrier to effective instructional practice is lack of information”. By applying data analytics, we can now obtain useful information about the learner and learning process to aid instructional practice. For instance, Wise and Jung (2019) reported the university instructors’ use of a LA dashboard to inform their teaching. A model of instructor analytics was constructed based on the findings to propose useful categories of activities for future study and support.

Although previous studies have recognised different practical implications of learning analytics (hereafter LA) related to user behaviour and engagement modelling, predictive analysis, personalisation and adaptive learning (Krumm et al., 2014; Nguyen et al., 2017), the design and implementation of data analytics in education involve complex processes, and the widespread adoption of LA will require sustained efforts (Chatti et al., 2014; Daniel, 2015). The development and implementation of LA in higher education are often ad-hoc and lacking the replication and improvement capabilities (Nguyen et al., 2020). In this paper, we argue that depicting LA information systems (hereafter LAIS) as a class of information systems and conceptualising its design theories would establish a fundamental infrastructure to promote the development and implementation of LA.

For decades, information systems (IS) have been a powerful tool supporting and transforming education to meet the increasing demands of society (Leidner & Jarvenpaa, 1995). Institutions have applied IS to assist different educational stakeholders, such as students, teachers, and institutional administrators, in learning and teaching activities, administrative tasks, and decision-making (Goldstein & Katz, 2005; Leidner & Jarvenpaa, 1995). IS have also become an inseparable part of modern education. The application of educational IS has continuously improved the effectiveness of learning and teaching (Dahlstrom et al., 2014; Leidner & Jarvenpaa, 1995). In the context of higher education, IS enhance learning and teaching by allowing for distance and self-paced learning, data-driven instruction, and automation of pedagogic activities. The use of IS in education led to novel research domains, theories, and principles that sought to address challenges facing education. As such, research in educational IS has addressed the demands for more effective tools regarding both administration, education and research required for increased masses of students (Becker et al., 2017; Henderson et al., 2017; Lacity et al., 2018; Pucciarelli & Kaplan, 2016). We argue that the theorisation of LAIS would open new opportunities for the educational IS research agenda to respond to the increasing demands from a widespread digitalisation in education.

The balance of both technology and learning aspects is crucial for the realisation of LA (Daniel, 2017; Dawson et al., 2015; Reimann, 2016; Zhang et al., 2018). Previous studies often examined ad-hoc or one-off applications of LA and there is a lack of standardised design knowledge to guide LA development. Recently, a few attempts have been made by connecting LA and IS research through the role of design science and design-based research (Nguyen et al., 2020; Reimann, 2016). The recognition of LAIS as a class of IS would bring together the forces of IS and learning sciences researchers to offer effective LA solutions.

In this paper, we also respond to recent calls to consider how LA should be applied to support learning and teaching activities in higher education (Daniel, 2017; Zhang et al., 2018). This study seeks to establish a set of design principles that guide the development and implementation of LAIS. The research question for this study is as follows: How to design underlying information systems that support LA in higher education?

To address the research question, this study employed a design science research methodology (hereafter DSRM) (Peffers et al., 2007) to develop and evaluate a set of design principles for LAIS as a type of IS applied in the field of education. DSRM focuses on the development of a research artefact that would likely be a system or an object intended to support system development. DSRM is also well suited for DSR, which’s objective is to form design principles to support systems development (Tuunanen & Peffers, 2018). The design principles are prescriptive statements that constitute the basis of design actions (Baskerville & Pries-Heje, 2010; Chatterjee et al., 2017). In this study, we conceptualised an initial set of design principles grounded in the literature and then revised these via demonstration and evaluation of an operational prototype. The conceptualisation of our design principles was both action and materiality oriented (Chandra et al., 2015). We sought to prescribe what an artefact should allow users to do and what it should comprise. Furthermore, we developed and demonstrated a fully functioning prototype as a design instantiation of LAIS that illustrates the established design principles. Prior research has recognised the role of an instantiation of an IS design theory as an expository or representational tool that is embodied within it (Gregor & Jones, 2007). Accordingly, we revised the principles through the development process of the system instance, its demonstration, and evaluation. Design science researchers can match our design principles with a particular application scenario and translate them into specific design requirements for LAIS (Chandra Kruse et al., 2016).

The following section reviews the literature on LA and the design, development, and implementation of LAIS in higher education. Then, we conceptualise and formulate the initial design principles. The design principles were applied to a technological architecture for LAIS. Thereafter, we demonstrate the proposed design principles through an operational LAIS prototype. Later, for evaluation, we used the artefact in four undergraduate courses with a total of 1,173 students. In the evaluation, we initially inspected server log data to assess the utility and efficacy of the artefact. We also conducted lecturer interviews and a student survey for further evaluation. After presenting the DSR process, we discuss the implications of our study, and we conclude by discussing its limitations and future research directions.

2. Theoretical foundations

2.1. Learning analytics: opportunities and challenges

Over the past decade, rapid developments in the field of big data and analytics have offered opportunities to discover useful insights from massive volumes of educational data (Baker & Inventado, 2014; Ifenthaler et al., 2018). Some research has reviewed and analysed the features and applicability of data analytics to support learning and teaching (Arnold & Pistilli, 2012; Nguyen et al., 2017, 2018; Nistor & Hernández-Garcíac, 2018). As noted, these attempts to apply data analytics in education have emerged as a new discipline called LA. In general, LA refers to the application of data analytics methods and techniques in learning and teaching. The widely adopted definition of LA is “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (Siemens & Long, 2011, p. 32). In other words, the aim of LA is to process educational data to offer meaningful information related to learner profiles, learning materials, and the learning context. It can perform descriptive modelling and predict learning constructs on a scheduled or real-time basis.

LA offers remarkable benefits to different educational stakeholders, including lecturers and students. For instance, it can provide updated information about learning activities and student engagement, which could be used to construct a model of successful student behaviours (Nistor & Hernández-Garcíac, 2018; Siemens & Long, 2011). Instructors could use this model to revise learning activities and remove those that are unrelated to the course objectives. For example, Brown (2020) reported the use of LA dashboard to monitor student learning at scale in large lecture courses. Although the teachers recognised the efficiencies that the LA dashboard facilitated, they are frustrated with how the displayed data undermined their existing pedagogical strategies. They faced difficulties in sensemaking about the reported data. Likewise, Vieira et al. (2018)’s systematic review of visual LA of educational data noted that far too little attention has been paid to how to deliver LA information to users in classroom settings. Moreover, while LA research mainly focusses on the analytics and learning facets, there are only a few studies that investigate LA from an IS perspective to promote its practical applications to support learning and teaching in higher education.

Although the literature has shown that LA has a promising impact on learning and teaching, the implementation of LA in practice has faced several challenges (Daniel, 2015; Peña-Ayala, 2018). For instance, most of the data generated and stored in institutional IS are interoperable, but it is difficult to integrate data from disparate sources without data loss (Daniel, 2015, 2017). Moreover, as LA is an interdisciplinary area of research, there is still a divide between those who understand the methods and techniques of data analytics and those who know how data analytics can be used to produce useful outcomes (Daniel, 2015). Furthermore, the systematic implementation approach of LA in higher education is still lacking. The importance and originality of this study are that it explores LAIS as a class of IS and offers insights into its design.

2.2. Development and implementation of learning analytics information systems

A considerable amount of literature has been published on the use of LA as a method to gain insights into learners and their learning (Greller & Drachsler, 2012; Peña-ayala, 2018; Saarela & Kärkkäinen, 2017). Although these studies have extended the understanding in the domain of education, ad-hoc analyses were most common, and they were usually conducted to answer a specific research question, not gain information that could be used for widespread application.

Other studies have proposed a variety of LAIS for practical implementation (Bodily et al., 2018; Leony et al., 2012; Ruiz et al., 2014; Siemens et al., 2014). For instance, Leony et al. (2012) presented a web-based visualisation platform called GLASS (Gradient’s Learning Analytics System). This system offers a simple workflow to visually present information related to students and their learning process in the form of widgets on a canvas on a visualisation dashboard. Another example is Gavriushenko et al. (2017)’s system architecture towards the development of an automated system for the academic advising process. This system architecture allows for determining the study profiles and recommending the proper study path to the learners.

While some prior studies propose that LAIS are an extension of existing educational IS (Ruipérez-Valiente et al., 2015), other research describes them as distinct, standalone systems (Bodily et al., 2018; Dyckhoff et al., 2012; Siemens et al., 2014). We argue that standalone LAIS with less dependency better address the challenges related to data integration. As such, LAIS can benefit from the flexibility of connecting to data from different sources to offer useful insights (Järvelä et al., 2019; Noroozi et al., 2019) and from the scalability. Accordingly, this indicates the need to investigate the underlying characteristics of LAIS as a new class of IS.

Most studies in the field of LA have developed and implemented LA for a specific application (Bodily et al., 2018; Nguyen et al., 2020). For instance, Rubio‐Fernández et al. (2019) proposed a LA tool designed and implemented specifically for recommending actions to be performed to enhance a specific type of flipped classroom. Although previous research has made sustainability contributions in exploring the development and use of LA, they often neglected to present the general design principles and implementation of LAIS for further developments in the field. To our knowledge, there is only a limited account of research on LA from IS perspectives and no studies have synthesised and conceptualised design principles for LAIS. This study seeks to fill this gap in the literature. To do so, the research process involves iterations of three research activities: 1) formulation of design principles, 2) design and development and 3) demonstration and evaluation.

3. Design science research approach

The main goal of this study is to develop a set of empirically and theoretically grounded design principles for systems that support LA in higher education. This set of design principles is an information technology (IT) meta-artefact which establishes a general solution by depicting a class of technologies and can be instantiated into concrete IT artefacts (J Iivari, 2015, 2017).

We adopted Peffers et al. (2007)’s iterative research process that allowed for the development of both design principles and system artefact. The system artefact, as an information technology with certain material properties, is utilised to demonstrate and evaluate the proposed design principles. Our study began with the identification and formulation of the problem and objectives (Phase 1) then went through three iterative phases: 2) Conceptualisation of Design Principles; 3) Design and Development; and 4) Demonstration and Evaluation.

In the identification and formulation of the problem and objectives (Phase 1), the problem of lacking guidance in the design and development of systems supporting LA is identified through the literature review and practical experience. Digitalisation has influenced and changed how education functions and is administered. While a massive amount of educational data is generated every minute, most institutions have not gathered and utilised the data effectively. Although LA has demonstrated potential benefits to educational stakeholders (Nguyen et al., 2017; Peña-Ayala, 2018), there are several challenges facing the implementation of LA in higher education (Chatti et al., 2014; Daniel, 2015; Dawson et al., 2014). As an interdisciplinary field, LA involves different expertise from different disciplines such as IS, computer sciences, and education (Dawson et al., 2014). Consequently, the development and implementation of an effective LAIS in institutions require extensive resources, skills, and in-depth knowledge. Even though different sets of challenges have been conceptualised by previous studies (Daniel, 2015; Slade & Prinsloo, 2013), there remains a paucity of guidance on the development and implementation of LAIS in higher education. As a result, this study aims to construct a set of design principles that describe a class of systems that are a means to the purpose of supporting LA in learning and teaching.

In the conceptual development phase (Phase 2), we formulated the design principles based on a process-oriented view of LA, using intervention theory as kernel theories. Drawing on the principles of intervention theory, we identified the prominent affordances required in the LA process and material properties to provide those affordances.

In the design and development phase (Phase 3), the design principles were translated into technical architecture and an operational prototype, that established the base for the subsequent phase of demonstrating and evaluating the artefact. The system prototype and its architecture acted as an expository or representational tool and was designed with the design principles embodied. It was a theory-ingrained artefact (Sein et al., 2011) and embedded the research contribution in its design (Peffers et al., 2007). The system prototype was developed with state-of-the-art web application frameworks and technologies and hosted on Amazon Web Services (AWS) for operating in production.

In demonstration and evaluation (Phase 4), we validate the utility and efficacy of the proposed design with the prototypical implementation. The operational prototype of a LAIS was implemented to support learning and teaching at a university. We conducted two rounds of demonstration and evaluation of the artefact. The design principles and prototype were revised after each round. In the first round, we conducted a pilot study to demonstrate the prototype to the users and gathers their feedback for potential improvement. In the second round, we used three data sources including usage data (server logs), survey data from the students, and interviews with the lecturers, to formally evaluate the design principles and its artefact by adopting the evaluation criteria suggested by Venable et al. (2012). The analysis of usage data and survey data allowed us to observe whether affordances were indeed enacted and thus justify the utility and efficacy of the artefact. The teacher interview reveals potential side effects and undesirable consequences of using the artefacts for the improvement of the design principles.

4. Design principles for learning analytics information systems

The conceptualisation of design principles was informed by the literature on learning analytics (LA) (Greller & Drachsler, 2012; Nguyen et al., 2017) and LA processes (Ifenthaler & Widanapathirana, 2014; Siemens, 2013), and based on kernel theory provided by prior theory (Gregor & Jones, 2007). The conceptualisation focuses on the activities that the system should afford for the LA process including measuring, collecting, analysing and reporting of data. In line with the fundamentals of design science research methodology (Hevner et al., 2004; Sein et al., 2011; Walls et al., 1992), the requirements derived from kernel theories were used as the foundation for articulating the design principles. Particularly, the intervention theory (Argyris, 1970) was used as a kernel theory to formulate the design principles for guiding the development of LAIS.

Prior research implies that the main applications of LA centre around providing insights to support decision making, aid or perform necessary interventions in learning and teaching (Dyckhoff et al., 2012; Nguyen et al., 2017). Beyond collecting and analysing educational data to offer useful insights, LA should consider the learning and teaching intervention to effectively support the learning and learning design process (Ifenthaler et al., 2018; Xing et al., 2015). In conjunction with this point, we argue that LAIS is only effective by empowering teachers and students as key users. Accordingly, the intervention theory is the best fit for this purpose, because of its emphasis on allowing the user, the receiver of the intervention, to be autonomous. Argyris (1970)’s Intervention Theory and Method suggest that “an intervenor, in this view, assists a system to become more effective in problem-solving, decision making and decision implementation in such a way that the system can continue to be increasingly effective in these activities and have a decreasing need for the intervenor” (Argyris, 1970, p. 15). As a result, we derive the constraints from the design of interventions to establish LAIS design principles.

According to the intervention theory (Argyris, 1970), there are three fundamental principles guiding the design of interventions: leveraging valid and useful information, allowing free informed choice by the user, and fostering internal commitment. The first principle emphasises on the use of valid and useful information. Valid information is “that which can be verified and has been shown to affect the phenomena the intervenor is seeking to influence” (Piccoli et al., 2019, p. 3) whereas useful information allows the users to “control their destiny” (Argyris, 1970). To support effective interventions, LAIS should offer both valid and useful information to the users. In consonance with this requirement, the literature on LA also highlights the central role of actionable insights in LA applications (Dawson et al., 2015; Rl & Gynther, 2018; Siemens & Long, 2011). The actionable insights refer to valid and useful information that can be concerned with the potential for practical action and influence user behaviour. We argue that LAIS must be able to report actionable insights for the users to perform and evaluate necessary interventions to the process of learning and teaching:

DP1. Principle of actionable information: LAIS should have features that allow for the reporting of actionable information about learners and their learning.

The first principle reflects the main purpose of LAIS: to provide insightful information that can support users in decision-making associated with learning and teaching activities (Ifenthaler et al., 2018; Mangaroska & Giannakos, 2017). To perform scheduled or real-time modelling and predict learning, LAIS collects static and dynamic data about learner profiles, learning content, and the learning context (Greller & Drachsler, 2012; Nguyen et al., 2018). In this way, LAIS can provide useful metrics that provide teachers insight into learners and their learning behaviour. In addition, LAIS facilitates the evaluation of learning design with real-time and predicted information (Bakharia et al., 2016; Persico & Pozzi, 2015). The metrics may have different levels of aggregation specified to the needs of the users to help them plan learning interventions accordingly. For instance, a LAIS may digest the learning behaviour data and historical records to offer predictive information on the student performance in a short-term (e.g., semester results) or long-term (e.g., degree completion) so that teachers can conduct early interventions and provide additional support. This principle also reflects the usefulness of LAIS.

Beyond providing insightful information, timing can influence the validity and usefulness of the information and affect the potential interventions (Butler et al., 2007; Shimada et al., 2018; Te’eni, 1991). Up-to-date information is needed for appropriate decision-making in dynamic contexts in which data change over time and information needs to be frequently updated (Ifenthaler et al., 2018). Nevertheless, real-time reporting of information is not always effective. For instance, Butler et al. (2007) reported that tactically delayed feedback with the right timing led to improved final test performance relative to immediate feedback. As a result, we argued that LAIS should deliver reports in a timely manner to effectively support end-users:

DP2. Principle of information timeliness: LAIS should generate reports in a timely matter.

Information timeliness refers to the ability to provide information at the appropriate time for its maximum impact. Apart from the technical constraints, the time latency between the data collection and reporting should be designed based on findings from the learning sciences and decision sciences. The second and third principles guiding the design of intervention refers to free informed choice by the users and fostering their internal commitment subsequently (Argyris, 1970). Once the users are provided with valid and useful information, LAIS should enable users to take control of the information and interventions. Furthermore, we argued that the least required efforts would best motivate the system usage for effective interventions. The ease of use would encourage positive attitudes towards technology and promote its usage. As a result, we initially posited that LAIS should allow for availability and interoperability.

LAIS should be available and accessible to both data clients and data subjects at any time, as server shutdowns may discourage users from using the system and affect their tasks. LAIS should also respond to users when they request expanded exploration of the data in real-time or almost real-time (Ifenthaler et al., 2018; Nguyen et al., 2017).

Moreover, it is important for LAIS to connect with the existing information systems in the institutions. Continuous data integration enables one to capture and load data with different schemas from multiple sources to generate meaningful information. As a result, LAIS should have a mechanism to collect and integrate data from different system environments (Chatti et al., 2014; Siemens, 2013):

DP3: Principle of availability and interoperability: LAIS should be available and accessible to both data clients and data subjects at any time and be able to interoperate with any learning and teaching system, including virtual learning environments (VLEs), and allow the integration of different data sources.

This principle addresses the challenge of data integration when developing and implementing LAIS (Chatti et al., 2012; Daniel, 2015). Institutions in higher education do often have several non-integrated information systems that store different educational data including student profile, learning and teaching resources, institutional finance, etc. The design and development of LAIS should concern multiple data sources to establish an integrated data pipeline that can offer a comprehensive picture of the learners and the learning process. The next section proposes a LAIS architecture encapsulating the initial conceptualised design principles, that guides the development and demonstration of an operational system.

5. Learning analytics information systems architecture

Based on the design principles, a LAIS architecture was designed using service-oriented architecture (SOA) approach, which is popular in system design because of its high flexibility and extensibility (TM Nguyen et al., 2005). An SOA system consists of multiple discrete components with a set of defined functionalities, and each unit can operate and be updated independently. This makes the SOA approach best suited for the purpose of actionable reporting and information timelines as different services can be designed for specific interventions and groups of users. Furthermore, this approach allows for high availability and interoperability.

We designed a LAIS architecture as an overall infrastructure for a group of LA services that support learning and teaching activities. Each LA service was designed and developed based on learning theories related to specific pedagogic activities (see DP1 and DP2). The SOA approach allows for parallel or even collaborative operation of different LA services to support different phases of learning and teaching processes. The actionable insights can be generated and provided to the users for different intervention scenarios without many dependencies. In addition, it features rapid extensibility and reusability.

Figure 1 proposes a LAIS architecture that allows for real-time interactions between learning and teaching systems and LA services.

Figure 1. Learning analytics information system (LAIS) architecture

Learning and teaching systems support authoring, content delivery, learning design, and learning processes. A typical example is a learning management system, which is intended for course delivery and administration. However, learning and teaching systems can also include authoring systems, communication systems, and LD tools. Both static and dynamic data are collected for analytical processes. A batch extract-transform-load (ETL) module collects and transforms static data that do not change over time, such as data from the enrolment system concerning learners’ backgrounds and course information from the learning management system.

The event transformation service module uses dynamic events from logging services and event trackers via event adapters as inputs. It normalises events from multiple sources and records them in an intermediate database. A real-time data cache can be retrieved by LA services for further analytics processing (see DP2). The outcomes of LA are delivered to end-users by reporting and response services. The user can also directly query LA data from the databases via a data application programming interface (API) such as the Experience API (xAPI). APIs are a set of subroutine definitions and communication protocols, and the xAPI is a new specification for learning technology that enables the collection of data about a wide range of experiences a learner has within online and offline learning environments. Only authorised parties have access to this data and the UI supports interactions between the end-users and the system.

The LAIS architecture allows for parallel development and simultaneous implementation of different LA while maintaining comprehensive interoperability between services and different educational IS (see DP3). The independence between the services allows for minimum downtime during new development and implementation thus maximise the availability of the system and its services. In addition, it allows for independent connections between specific services and existing educational IS. This would be also extremely useful in the scenario that institutions may have a number of different IS that may or may not contain usable interfaces for integration. Educational data from different systems would be extracted and put through a data pipeline that transforms and loads raw data into an integrated schema. This LAIS architecture does not only address the interoperability with the existing systems but also enables accelerated LA development informed by the needs of learning and teaching. Furthermore, evaluation of LA services based on pedagogic outcomes provides useful feedback for the development and implementation of future LA services.

We defined the main LA services and their functionalities based on learning management questions (Wiggins & Mctighe, 2005) and the main applications of LA (Ifenthaler & Widanapathirana, 2014; Nguyen et al., 2017, 2018). Table 1 shows these services in relation to the questions. Each LA service interacts with other systems or services via four groups of parameters: events, metrics, situations, and actions. Figure 2 shows an abstraction of these interactions.

Figure 2. Key learning analytics (LA) services

Table 1. The functionality of learning analytics (LA) services

LD Phase	Learning Management Question	LA Service
1. Profiling	What does the learner already know?	Learner Profiling: • Prior knowledge sets • Personal aspirations • Preferred learning styles Prediction: • Performance
	Where does the learner need and want to be?
	How does learner best learn?
2. Strategising and Delivery	What resources do I have at my disposal?	Decision Making: • Appropriate instructional resources and/or strategies Prediction: • Scenario planning
	What will constitute the learning journey, and therefore, what is the best context for learning?
	Who will do what?
3. Ascertainment and Reporting	How will I check whether the learner has achieved the learning outcomes?	Situation Detection: •Notification of at-risk situations Evaluation: • Information on learning engagement Decision Making: • Recommended interventions
	How will I inform the learner and others about the learner’s progress?

6. Demonstration and evaluation of the design principles of learning analytics information systems

Next, we demonstrate how we applied the design principles described above to develop a LAIS prototype in the context of higher education. Furthermore, we present a formal evaluation of the proposed design principles, and we discuss how this evaluation validates the use of the design principles and facilitates revision of them.

Evaluation of the DSR methodology (DSRM) comprises two main sub-activities: demonstration and evaluation (Peffers et al., 2007; Tuunanen & Peffers, 2018). Demonstration indicates the ability of the artefacts to solve the identified research problems (Venable et al., 2012). In other words, it is a proof-of-concept evaluation that seeks to demonstrate that artefact is viable for fulfiling its design purpose in a particular context. The evaluation aims to formally validate the effectiveness of the research artefacts for addressing the problem (Peffers et al., 2007).

In this study, we demonstrate and validate the utility of LAIS design principles through operational LAIS prototyping (Basili et al., 2005; Davis, 1992). The principles were then evaluated by qualitative assessment involving observation, participant feedback, and semi-structured interviews (Tremblay et al., 2010; Tuunanen & Peffers, 2018). We adopt the evaluation criteria suggested by Venable et al. (2012):

• Evaluate the artefact to establish its utility and efficacy for achieving its stated purpose.

• Evaluate the artefact to identify weaknesses and areas of improvement.

• Evaluate the artefact to identify side effects or undesirable consequences of its use.

6.1. Demonstration of an operational learning analytics information system prototype

The demonstration phase of the study was conducted at a large university in New Zealand in the first semester of 2016. The semester started at the beginning of March and ended in early July. The phase involved three operations management courses: two stage-II undergraduate courses (79 and 58 students) and one postgraduate course (14 students). Among the undergraduate students, 24 were taking both courses. In total, there were 127 potential data subjects. The courses ran for twelve weeks, and the final exam period occurred after the twelfth week. All the courses applied a blended learning approach, combining online teaching and learning materials with traditional classroom methods (i.e. face-to-face lectures were incorporated with computer-mediated activities for content delivery) (Porter et al., 2014). In this case, a lecture theatre recording (LTR) service was used in the courses to allow students to take control over the time, place, path, and pace of their learning.

We applied the operational prototyping approach to demonstrate the use of LAIS design principles in the context of higher education. Operational prototyping (Davis, 1992; Tuunanen et al., 2008) has been a popular approach to demonstrate the use of system design in the process of IS development (Davis et al., 2006; Flink, 2014). A prototype is a partial implementation of an IS intended to validate its utility as a solution to the identified problem as well as learn about potential issues with the design. The creation of prototypes has been standard practice in IS development for many decades (Beynon-Davies, 1998; Davis, 1992; Davis et al., 2006).

Operational prototyping integrates the two traditional approaches: throwaway prototyping and evolutionary prototyping (Davis, 1992). A throwaway prototype is built as fast as possible to verify poorly understood requirements and then is discarded, whereas an evolutionary prototype is constructed with well-understood parts to discover unknown requirements and then evolve the design. Throwaway prototyping is ineffective due to the lack of understanding about critical requirements, and evolutionary prototyping is ineffective as it does not produce information about the poorly understood requirements (Davis, 1992; Davis et al., 2006). Operational prototyping balances these limitations and enables comprehensive demonstration by building throwaway prototypes on top of evaluation prototypes.

6.1.1. An operational prototype of a learning analytics information system

We sought to develop and demonstrate an operational LAIS prototype based on the set of design principles described above. During a departmental meeting at University A, we conducted a presentation on how LA can be applied to support teachers in learning and teaching and introduced our research project. Some lecturers expressed interest in testing the prototype. With these lecturers, we discussed constructing a theory-based objective for LAIS prototyping.

As a proof of concept, the LAIS architecture and design principles were implemented to support teachers in evaluating students’ engagement with LTR, an expected service in blended classes using a lecture-based instructional approach. This information allows lecturers to evaluate in-class activities and ensure that their design engages not only the students that were present at the lecture but also the students who use the recordings. Since recent studies recognised the increasing use of lecture LTR as a substitution for attending classes (Wieling & Hofman, 2010; Williams et al., 2012), the consideration of LTR usage would inform lecturers about student engagement with learning.

Previous research suggests that students’ engagement with learning has a significant impact on their performance (Carini et al., 2006; Kahu, 2013). By frequently monitoring student engagement with a course, a lecturer could perform necessary interventions and provide additional support to certain students in a timely manner. This could moderate and improve student engagement and, in turn, improve learning and achievement (Klem & Connell, 2004). There is evidence showing that using LTR as a supplement when developing a knowledge base significantly improves learning performance (Bos et al., 2016). Using observations of in-class attendance as a complement to information about students’ interactions with LTR would provide lecturers a more complete picture of students’ engagement with lectures. Accordingly, we identified relevant events, metrics, and the potential use of LTR for operational prototyping (Table 2).

Table 2. Design of the learning analytics information system prototype

Data Captured

Reported Information

Events: • load_video • play_video • pause_video • seek_video • speed_change_video • stop_video. Metrics: • Course information • Video metadata (e.g., title, length.)

Metrics: • Number of active students in each week • Average video views each week • Number of students who watched a particular video (unique viewers) • Number of replays at any point in the video (replayed segments) Potential Use: • To identify the parts that students most frequently skip or replay → Adjustment of potentially problematic parts of the learning materials • To monitor students’ engagement with learning → Early interventions if there is a lack of engagement → Evaluation of the effectiveness of learning design improvements on subsequent student engagement

An operational prototype was built based on the Open edX platform (“Open edX,” 2017). Among several LA platforms, Open edX was selected because it appeared to meet our requirements. The prototype captures defined events and metrics to provide analytical reports of students’ engagement with LTR.

To ensure that they include actionable information about learners and their learning (see DP1), the reports were designed to take into account potential pedagogical actions. A report was sent to the lecturers each week via email (see DP2). The report included multiple graph visualisations with a user-friendly design. Figure 3 shows an example of the graphs.

Figure 3. An example of a graph reported to lecturers

The graphs show the total number of views on a video and the points on the video timeline to which the views correspond (light blue area). Thus, they show which parts students skipped most frequently. For instance, most students skipped through the first minute of the video represented in Figure 3 after viewing a few seconds. This result could be explained by the fact that viewers usually ignore the introduction slide and jump to the main content. It is interesting to note that the number of views slightly decreased near the end of the video. This indicates that a proportion of students gave up on learning from the LTR while viewing it. These results can be used to objectively determine the optimum length of lectures and determine the interest in a particular lecture.

The reports also provide information about segments that were replayed (dark blue area). This information may indicate potential cognitive difficulties concerning a specific part of the LTR, such as the peaks at the fifth and seventh minutes shown in Figure 3. Furthermore, several replayed segments in a lecture might suggest that students found the lecture to be difficult to understand, and the instructor might need to revise the content in later lectures or provide additional resources to improve students’ understanding.

The LTR system is available for the students (data subjects) and teachers (data clients) to access at any time (see DP3). The information reported to lecturers is generated by analysing the data collected and integrated from several sources (see DP3). The system prototype tracks student inputs posted to the server and records them to the tracking logs. In particular, the system captures emitted events regarding students’ interactions with the video player while watching a recording. It also gathers static data from multiple databases. Course information is retrieved from the MongoDB database, and user details are collected from the MySQL database in the learning management system. Then, the application passes all events and state data to the analytics pipeline for data analysis.

6.1.2. Learnings from the demonstration of the prototype: utility and efficacy of the artefact

The objective of the demonstration phase of the study was to evaluate the implementation of the LAIS to establish its utility and efficacy for achieving its stated purpose (Venable et al., 2012). Next, we present how lecturers could use the LAIS prototype to support their teaching. By inspecting the parts that students most frequently skipped or replayed, the lecturers can identify potentially problematic parts of the learning materials and make appropriate adjustments, including revision of difficult content and inclusion of missing knowledge in the following lecture. Parts that were most commonly problematic featured poor audio quality and unclear explanations of concepts. In addition, the inspection of the most frequently skipped parts indicated that some content was counterproductive and should be eliminated. To optimise the blended learning settings, the lecturers attempted to edit the original LTR to remove ineffectual fragments, reorganise the recordings into shorter videos and classify them into themes for better navigation and reduced cognitive workload. Prior studies suggested that these interventions would be able to promote learning and teaching by leveraging student engagement with learning materials (Lawless & Brown, 1997; Mayer, 2008). For example, a well-designed navigation tool can leverage student engagement with learning content and promote learning performance (Mertens et al., 2004).

By regularly monitoring students’ engagement with learning, the lecturers can perform necessary interventions to counteract a lack of engagement in a timely manner. For instance, learning procrastination often occurs in the early weeks of the semester that does not include any assessments. Although this behaviour can be expected, evidence can encourage lecturers to perform early interventions. Attempting to engage students in constant learning could improve students’ performance and enhance learning outcomes (Carini et al., 2006; Kahu, 2013). Figure 4 illustrates an example of the LAIS reporting dashboard that indicates a lack of engagement with LTR.

Figure 4. The learning analytics information system’s dashboard indicates a lack of engagement in a timely manner

The lecturers attempted to encourage students to engage with the lecture content early in the semester whenever they observed low engagements with recent recordings. Two lecturers introduced quizzes at the beginning of the following lecture to assess students’ knowledge and recall.

The lecturers observed a high level of student engagement with activity-based learning (ABL) exercises, as shown in Figure 5. This finding aligns with previous education science literature findings that ABL exercises engage students better than other types of exercises and hence promote active learning and improve academic performance (Oigara et al., 2014; Roehl et al., 2013). Consequently, the lecturers adjusted their syllabi to include ABL exercises.

Figure 5. The learning analytics information system’s dashboard shows the most engaging activities

6.1.3. Learnings from the demonstration of the prototype: weaknesses and areas of improvement

The lessons learned while developing and implementing an educational application provide insights and guidelines regarding the current understanding of specific educational technologies. Continuous evaluation and adaptation are important in software development to deliver better value to the end-users. In order to learn about possible weaknesses and areas of improvement (Venable et al., 2012), we performed qualitative assessments through observation and user feedback (Boudreau et al., 2001; Tremblay et al., 2010). We constantly communicated with two lecturers using the LAIS prototype throughout the semester. Students gave feedback via email regarding any operational issues or failures. Furthermore, the field notes taken by the researchers were examined to identify observed problems and potential improvements.

During the demonstration phase of the study, the two lecturers expressed interest in receiving information about students’ engagement with LTR via an analytics dashboard that they could access when needed. Also, issues regarding multiple logins were observed, and students asked for the opportunity to sign on to the learning platform, which was isolated from the university learning management system, a single time. Many students demanded improvements in the ease of accessing the system. These results indicate the need for a single sign-on service (SSO) to eliminate the need to manage two different accounts.

We revised the design principles to address the issues experienced in the demonstration phase of the study. We found that, together with system availability, users found the availability of reporting information useful. As a result, it is important to store LA reports on the system so that the data clients (i.e. teachers) can access it when needed. Moreover, a dashboard with visualisations can be applied to display LA reporting information. LA reports can be practically supported by visualisations to deliver more meaningful information to the users (Duval, 2011; Leony et al., 2012; Nguyen et al., 2017). The benefit of visualisations is to better communicate large amounts of complex data to identify trends, patterns, correlations, and key issues. Furthermore, we observed that integration of the LAIS into the existing IS infrastructure at the university may lead to issues or complications. When developing the LAIS, we should consider and address these issues to optimise the utility of the system. Accordingly, we revised DP2 and DP3 as Principle of information delivery and Principle of interoperability to include these properties. We also saw a need to form a new DP4, Principle of information availability, based on the original DP3 and our findings from the demonstration phase. The design principles are defined as:

DP2. Principle of information delivery: Should generate responses and information that visualise learning and teaching behaviour and performance.

DP3: Principle of interoperability: Should interoperate with any LA and/or educational IS, including VLEs, and enable integration with different data sources without resulting in any discernible issues or complications.).

DP4. Principle of information availability: LAIS should be available and accessible to both data subjects and data clients at any time. LA reports should be stored on the system and accessible to the data clients at their convenience.

The system prototype was revised and updated to embrace the new design principles. The analytics dashboard module was updated so that the lecturers could access the reported information at any time (DP4). Moreover, an SSO was developed and implemented to allow for seamless integration with the existing learning management system (DP3).

6.2. Evaluation of the learning analytics information system

To formally evaluate the developed artefact, we adopted a subsequent case-study approach. This allowed us to examine the use of LAIS in the context of higher education, evaluate the artefact’s utility and efficacy for achieving its stated purpose, and investigate possible side effects or undesirable consequences of its use (Venable et al., 2012). The case-study approach to evaluation has been widely used in design science research on IS (Boudreau et al., 2001; Hevner et al., 2004; Peffers et al., 2007). In this case study, we evaluated LAIS design principles and their implementation in a system via server log data, student surveys, and semi-structured interviews with the lecturers.

At University A, the developed LAIS prototype was examined in four undergraduate courses with a total enrolment of 1,173 students. These courses included one large first-year course (n = 966), and three second-year courses (n = 207). The large first-year course is compulsory for all students aiming to obtain a Bachelor of Commerce (an undergraduate degree in business administration) from the university. The course introduces students to the field of IS and explores how IS and operations management help organisations to innovate, optimise, and deliver value. Two of the second-year courses are mandatory for operations management majors, and one is an optional course on data management.

6.2.1. Analysing server log data and student surveys: Utility and efficacy of the artefact

We used server log data to evaluate the utility and efficacy of the LAIS for achieving its stated purpose. As the artefact was designed to provide actionable information that the lecturers could use to monitor students’ engagement and perform necessary interventions, we looked at students’ engagement with each LTR over time. Figure 6 illustrates the total number of interaction events performed by students for each LTR (marked with different colours) in the first half of the semester.

Figure 6. Total number of interaction events for each video in the large first-year course over time

The analysis shows the effect of interventions on students’ engagement with a particular video. Since students mainly use LTR as an alternative to attending lectures in person (Williams et al., 2012), students’ engagement with an LTR is usually expected to peak on the day it is released or the following day. Nevertheless, learning procrastination often delays student engagement, making it necessary for the lecturer to intervene. By monitoring student engagement, the lecturer can perform any necessary interventions to increase student engagement early in the semester to reduce the effect of procrastination and enhance student learning. A drill-down analysis of LTRs that involved lecturer intervention validated the impact of the lecturer’s actions using the artefact.Figure 8

Figure 7 illustrates the change in the total number of interaction events with a particular LTR (LTR.A) over time since the day of its release. The students only performed 153 interaction events with LTR.A on the day of release. The number of events increased to 606 in the following two days before decreasing after that. The lecturer intervened to increase students’ engagement with LTR.A, increasing the total number of interaction events to 1,987. The more than 300% increase in the total number of interaction events demonstrates the effect of the lecturer’s pedagogic actions on engagement with this particular LTR. This observation analysis of server log data demonstrates the use of the LAIS to support teaching and enhance students’ engagement with learning. Hence, the system design was validated to match the purpose. Rather than directly performing an action, the LAIS allowed for free and informed choice with valid and useful information. Accordingly, the design reflects the principles of intervention theory (Argyris, 1970).To further evaluate the artefact, we conducted a student survey. The survey was sent to students using the LAIS. The seven-point Likert scale (see Appendix 1) was designed to collect information about the extent to which they support teachers’ use of LA (see DP1, 2 and 4), their perceptions of the system’s quality to clarify the interoperability without resulting in any discernible issues or complications (see DP3) as well as potential ethical issues and concerns they identified (see DP4).

Figure 7. An example of the effect of the lecturer’s intervention on student engagement

Figure 8. Thematic analysis process for interviews with the lecturers

The results support teachers’ use of LA. The students agreed that the educational IS should inform lecturers about students that are at risk of failing (87.3% of respondents), learning content that students appear to find difficult to understand (96.0%), the learning progress of each individual (81.0%) and the whole class (90.5%) and visualisation of individuals’ (78.6%) and the whole class’s (85.7%) learning activities. Previous research has proposed different LAIS designs to address these applications of LA. For instance, Arnold and Pistilli (2012) demonstrated the Course Signal system that provided prediction on students’ performance thus informed about students that are at risk of failing. Furthermore, prior research showed the use of LA dashboard to monitor the learning progress and keep track of learning activities (Arnold & Pistilli, 2012; Bodily et al., 2018; Verbert et al., 2013). The results did not show any significant conflicts of interest regarding the principle of actionable reporting (DP1). They revealed that, from the students’ perspective, educational IS should have LA capabilities to support learning and teaching. Previous studies also reported positive student attitudes towards the use of LA in higher education (Pontual Falcão et al., 2019; Roberts et al., 2016).

The survey reported positive results regarding students’ perception of system quality of the artefact. Most students found the system consistent (84.5%) and the response time reasonable (84.5%). This validated that the artefact had improved since the demonstration phase of the study and was seamlessly integrated into the current learning management system without causing significant difficulties to the students (see DP3 and 4). In fact, recent research suggests that, from the students’ perspective, LA should not cause any difficulties in using learning management systems but improving its usage (Pontual Falcão et al., 2019). This result indicates that our design principles are beneficial for designing LAIS and can satisfy the requirements of both lecturers and students.

Last, a majority of students (78.1%) expressed concern about ethical issues regarding how the lecturers would use the information provided by LA. They were also concerned about the transparency of the process (82.5%), data security (73.7%), and data storage (82.5%). The findings not only validate the principle of information availability (DP4) but also provide evidence supporting concepts concerning students’ perspectives on ethical issues suggested in the literature (Pardo & Siemens, 2014; Slade & Prinsloo, 2013).

6.2.2. Interviews with the lecturers: Identifying side effects or undesirable consequences of using the artefact

Semi-structured interviews were conducted with the four lecturers who used the system in their courses. The interview questions were developed based on the guidelines by WEISS (1995) and included questions related to the usage of the reporting dashboard, perceived usefulness, difficulties, and concerns. In accordance with the definition of semi-structured interviews, the interviewer asked open-ended questions and then followed the participants’ lead with follow‐up questions so that they were given the opportunity to clarify or elaborate upon their answers until no additional information was discovered. During the interview, the interviewee was able to interact with the LA dashboard to demonstrate the usage of the IS or illustrate their statements. The interviews were recorded on digital files and transcribed by the researchers. Each interview lasted between 25 and 60 minutes, and the total length of the transcript was 37 pages. To analyse the data, we used thematic analysis, as described by Braun and Clarke (2006) (see Appendix 2). For confidentiality, pseudonyms were used, and identifying details were altered in the transcripts. In addition, some parts of the answers were omitted as they were unrelated or extraneous. These omissions are denoted by ellipses in brackets. Our findings reveal two main themes within the interview data: perceived usability and usefulness and subjective interpretation of the reported information.

The interviews explored the lecturers’ use of the LAIS and its perceived usefulness. All the lecturers reported that the LAIS was beneficial and supported them in monitoring students’ engagement with learning through the semester:

“It is useful to see how many students watched the lecture recordings” (Lecturer 4).

“I was able to be alerted that a particular recording at a particular time has a lot of replays” (Lecturer 3).

“So, it’s actually quite helpful to see at which moment when the peaks occurred. […], well, for short videos it’s very useful because I can just check a few peaks” (Lecturer 2).

The LA information helped the lecturers identify potentially problematic parts of the lecture content. For instance, if students experienced difficulties due to poor sound quality and missed a piece of essential information, this would produce unusual activity at that point in the video, indicating to the lecturer that there is a problem. The lecturer could then revisit that piece of information in the following lecture.

The lecturers also reported that being allowed to observe the content that is most often replayed by the students was useful:

“[I]t’s potentially very useful. I think, for me, the usefulness is being in reinforcing that doing activities in class is worthwhile” (Lecturer 1).

They commented that this insight would help their decision-making when designing the course for the following semester. In particular, the lecturers observed that students spent more time performing fill-in-the-blank exercises, in which students have to fill in missing words in a text, than engaging with the rest of the LTR. As a result, the lecturers aim to use these kinds of exercises to emphasise important knowledge in future courses.

Overall, the LAIS allowed lecturers to evaluate and analyse lecture content online via LTRs. The artefact provided the lecturers with information about the impact of the current lecture design, which will be useful for creating lecture content in the future. However, although the lecturers perceived the LA to be useful, they did report a lack of granularity regarding the visualisations of the aggregated analytics; the total number of replayed segments or completed viewing might not represent students’ actual engagement. For example, it is common for students to skip the title and copyright slides, which leads to a lower completion rate, even if students watch all of the body of the LTR. This negatively influenced the perceived usability and usefulness of the artefact:

“I think maybe the problem is the granularity. To be honest, I found that the difference between completed and not completed is misleading because students are never going to finish, actually complete the video, because there’s a gap on the end, right? In the beginning, there’s a copyright warning, and at the end, there’s a dead space. And I was, generally speaking, too lazy to edit the videos, to make them fully complete” (Lecturer 1).

“[…] the different graph there would be harder to analyse.” (Lecturer 2).

Despite the challenge of providing sufficient information without causing information overload (Edmunds & Morris, 2000; Ware, 2012), the findings indicate that both aggregated information and highly granular information improve the usefulness of reports for lecturers’ decision-making. LAIS should support customised ad-hoc analyses of learning with different units of analysis (e.g., individual learners or a whole class) and time frames (e.g., information aggregated by the day, week, month, or year). Correspondingly, LAIS should report information with a flexible granularity that can deliver satisfying usability and efficacy:

DP1. Principle of actionable information: Should provide reports of actionable information about learners and their learning with flexible granularity in reporting.

The interview also indicated that lecturers could have these biases that affect their behaviour in lots of different ways. Therefore, the finding highlights the need for anonymity in the set of ethical principles for LA (Pardo & Siemens, 2014; Slade & Prinsloo, 2013). If the anonymity is secured, it may prevent the following scenario reported by the lecturer:

“[…] might get quite angry about students not doing things, and not necessarily deliberately penalise them but indirectly, they might think “Well, they didn’t watch that, I’m going to make my whole exam about that” (Lecturer 1)

In addition to anonymity, transparency of the process and data security are the main aspects of ethical principles for LA (Pardo & Siemens, 2014; Slade & Prinsloo, 2013). The data privacy was identified as the key challenge for the implementation of LA at institutions of higher education (Daniel, 2015). Concerns have been raised about recording student activities on the system and profiling student learning. Gregor and Jones (2007) suggest that institutional executive offices are likely to be concerned about privacy and security issues when the system is up and running. We noticed that the third principle was quite generic and not reflected the nature of data sensitivity in using LA. According to these findings, the principle of information availability (DP4) is revised to address data anonymity, transparency, and security:

DP4. Principle of information anonymity and protection: LAIS Should provide anonymity for personal and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorised disclosure or access.

6.3. The final set of design principles for learning analytics information systems

The proposed set of design principles for LAIS was developed through the DSR process of theory-inspired development, demonstration with operational prototyping, and case-study based evaluation. Table 3 provides an overview of the final set of design principles for LAIS that support learning and teaching in higher education.

Table 3. LAIS design principles

#	Design principle	Design principle specification
DP1	Principle of actionable information	Should provide reports of actionable information about learners and their learning with flexible granularity in reporting.
DP2	Principle of information delivery	Should generate responses and information that visualise learning and teaching behaviour and performance.
DP3	Principle of information interoperability	Should interoperate with any LA and/or educational IS, including VLEs, and enable integration with different data sources without resulting in any discernible issues or complications.
DP4	Principle of information anonymity and protection	Should provide anonymity for personal and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorised disclosure or access.

7. Discussion and implications for research and practice

In the following, we discuss our contributions and implications in light of the literature on LA and educational IS as well as the practice of teaching with IS.

As an emerging field of research, LA has been defined as a process of measuring, collecting, analysing and reporting education data (Siemens, 2013). We demonstrated how LAIS could automate the LA process, so it has a greater impact on higher education. Although the literature has recognised the potential of LA for supporting and transforming institutional activities, such as educational decision-making, knowledge creation and enhancement of the learning experience (Nguyen et al., 2017; Nistor & Hernández-Garcíac, 2018), little attention has been paid to the design and development of IS with LA capabilities. This work links previous system designs for LA (Bodily et al., 2018; Ruipérez-Valiente et al., 2015; Siemens et al., 2014) to provide comprehensive guidelines for developing and implementing LAIS in higher education. Effective design and implementation of LAIS would offer great values to higher education institutions that are operating in an increasingly complex and competitive environment (Daniel, 2015).

Our study introduces LAIS as a new class of educational IS. Although such systems may be considered part of other types of educational IS, such as learning management IS (Daniel, 2015; Ruipérez-Valiente et al., 2015), the purpose of their design is different from and independent of existing systems. Furthermore, the theories regarding LA have indicated a need to examine LAIS as a new genre of educational IS to maximise its impact in different educational settings. Thus, we created a set of design principles for developing and implementing LAIS. Intervention theory (Argyris, 1970) as kernel theories governed our design requirements for LAIS. Through demonstration and evaluation, we show the feasibility of realising this type of IS in the context of higher education.

The theorisation of LAIS as a new class of IS can encourage educational IS researchers to join the force with the research communities in learning sciences to push the LA agenda forward. Furthermore, the conceptualisation of underlying design principles for LAIS would establish the essential foundations for further development and implementation of such systems. There is a significant body of IS literature pertinent to information systems design theory (Baskerville & Pries-Heje, 2010; Walls et al., 1992). IS design principles as design theories offer simple and elegant functional explanations for generalised solution components of a specific class of IS by the related generalised requirements (Baskerville & Pries-Heje, 2010; Gregor & Jones, 2007).

The goal of LA is to understand and optimise learning and teaching (Nistor & Hernández-Garcíac, 2018; Siemens, 2013). The proposed design principles are consistent with this view and our study demonstrates how information systems can deliver certain material properties that allow for the realisation of LA in higher education. Previous research suggested that the success of LA rests on its capability to provide actionable insights from educational data (Clow, 2013; Dawson et al., 2015; RL & Gynther, 2018). The computational aspects of LA must be well integrated with learning sciences to make a sustainable impact on the research and practice of learning and teaching (Dawson et al., 2015). Accordingly, the proposed design principles emphasise on generating actionable responses and information that visualise learning and teaching behaviour and performance with flexible granularity in reporting. The actionable insights could be appraised by their ability to encourage users to take necessary actions to improve learning and teaching. For instance, LA dashboard may report at-risk situations from students’ learning activities for lecturers to perform early interventions to improve student retention (Dietz-Uhler & Hurn, 2013; Wolff et al., 2013). Accordingly, the purpose of LAIS as a new class of IS was described by our design principles as to provide reports of actionable information about learners and their learning with flexible granularity in reporting. This sets out a clear goal for the design and development of LAIS to enhance learning and teaching.

Although the development and implementation of LA may show promising to automate several measurements and predictions about learning and teaching, the singular concentration on learning outcomes and performance, as the principal target of LA, without consideration of learning and teaching processes can have unfavourable consequences (Dawson et al., 2015; Van Leeuwen, 2019). Our study shows an example of how LA dashboard can assist lecturers in monitoring student engagement with learning materials to improve teaching and learning. Furthermore, we report empirical evidence supporting that LA should consider flexible granularity in reporting for maximising its effectiveness. (Gašević et al., 2016; Schumacher & Ifenthaler, 2018; Siemens et al., 2014). A lack of attention to instructional conditions can lead to ineffective use or exploitation of LA (Gašević et al., 2016). The flexible granularity in reporting can allow the LA users for inspecting and adopting actionable insights in accordance with instructional conditions. Correspondingly, the proposed set of design principles highlights the importance of information granularity in the LAIS development.

With regards to the confidential nature of the personal data used by LA, our study shows that for successful LA implementation it is necessary to provide anonymity for personal and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorised disclosure or access. For instance, the use of personal data and LA process can only be undertaken once they are approved by ethical committees and with consents from the data subjects. As the analysis of data relating to students and their activities is the foundation of LA process, the collection and use of these data face several ethical issues and dilemmas (Slade & Prinsloo, 2013). Our study reports that lecturers could have behavioural biases as the outcomes of using LA information. Prior studies indicated a need to contemplate how these ethical issues must be addressed from the early stages of the LA deployment (Pardo & Siemens, 2014). Our focus on information anonymity and protection is consistent with the ethical and privacy principles for LA (Greller & Drachsler, 2012; Pardo & Siemens, 2014). The adoption of information anonymity and protection would promote trust and accountability to use LAIS.

With regard to the fragmented nature of educational data and information systems, our study indicates that it is essential for LAIS to incorporate with any educational IS and enable integration with different data sources without resulting in any discernible issues or complications. As it is challenging to gather and combine unstructured data from several sources (e.g., learning materials, financial information, etc.) in a single data model (Daniel, 2015), we propose a service-oriented architecture of LAIS that allows for flexible parallel connections with different sources. The proposed design would reduce the dependencies (Arsanjani, 2004) between distinct LA services and lessen the risks of systematic errors and disruptions in the operation. Furthermore, our empirical data show that, for successful LAIS implementation, the integration with existing IS should not result in any discernible issues or complications.

All design principles proposed in this paper imply material properties that are expected to be embraced for the realisation of LAIS in higher education. Although our proposed design principles are either comprehensive or objectively superior to any other viable meta-requirement, they serve as the basis for LAIS implementation and evaluation. Moreover, following the iterative process of DSR, the set of design principles is grounded by both theories and empirical evidence that reflects the genuine user requirements.

The diversity of DSR has progressively developed over the past decade in different facets such as purpose, methodology, philosophical grounding, and mental models (Nguyen et al., 2019). Peffers et al. (2018) document five DSR genres in IS research: DSRM, IS Design theory (ISDT), Design-oriented IS research (DOIS), Explanatory design theory (EDT), and Action design research (ADR). J Iivari (2015) suggests two strategies for IS design science research: 1) designing an IT meta-artefact as a solution to a general class of problems and 2) solving a specific problem for a certain group of users and generalising a bundled solution generalised that addresses a class of problems. While ADR approach proposed by Sein et al. (2011) is more appropriate for the second strategy, DOIS and DSRM share more mutual elements with the first strategy focusing on the design of an IT meta-artefact as a solution to address a general class of problems. Nevertheless, Österle et al. (2011) note that DOIS “is not a non-judgemental scientific discipline; rather it is normative, in the sense that the construction of artefacts is guided by the desire to yield a specific benefit and to satisfy certain objectives” as this approach does not recognise theory building as a DSR activity but as “valid cause-effect relations” that provide “foundations for choosing desirable ends, i.e., normative actions” (Winter, 2008). Among the five DSR genres, DSRM (Peffers et al., 2007) is the best fit to our study as it emphasises on the design and development of applicable artefacts which could have contributions to both theory and practice. IS research artefacts include but do not limit to systems, applications, frameworks, design theories and methods (Gregor & Hevner, 2013; Hevner et al., 2004; Tuunanen & Peffers, 2018).

Peffers et al. (2018) further argue that the DSR genres must define “their standards, values, and beliefs flexibly to accommodate innovation and evolution” and “if authors describe and justify their objectives, methods, and results with good, appropriate rationale, their arguments should be given due consideration, even though they do not fit prior patterns” (Peffers et al., 2018, p. 136). DSRM artefacts imply generalisability in practice, yet the concept of design theory is not frequently found in the DSRM articles (Peffers et al., 2018). Our study intends to go further towards presenting design theories (Gregor & Hevner, 2013) than DSRM studies that focus on building an IS. Building on Peffers et al. (2007)’s DSRM we contend that the design, implementation, and evaluation of the set of design principles and its instantiation are substantial research contributions (Baskerville et al., 2018).

Design artefacts and design theories have been the two dominant types of DSR contributions (Gregor & Hevner, 2013). The literature highlights the importance of coexisting artefact and theory contributions in DSR but also recognises challenges to achieve both (Gregor & Jones, 2007; Hevner et al., 2004). Recently, Baskerville et al. (2018) clarify the importance of balancing the technical and scientific contributions of a DSR project. Accordingly, our DSRM approach to the study attempts to accomplish both these two types of contributions. To do so, we first conceptualised the initial set of DP using the principles of intervention theory as kernel theories then instantiated it into the technical architecture and system prototype for demonstration and evaluation (Gregor & Jones, 2007). Through the implementation and evaluation of the prototype, the proposed DPs were revised with the insights drawing from the authentic environment and end-users. As such, we both theoretically contribute a set of design principles and practically deliver the system architecture and prototype.

Our theory-inspired and practically applicable design principles attempt to address the problem of the lack of guidance in the design and development of LAIS. The development of DPs is rationalised with references to the findings of prior studies in LA and based on the IS kernel theories while the implementation and evaluation of the instantiated system in a real context enhances the practicability of the set of DPs since it is refined with empirical evidence. The design principles are recognised as contributions in the form of nascent design theory (Gregor & Hevner, 2013; Gregor & Jones, 2007).

J Iivari (2020) criticised that kernel theories borrowed from reference disciplines often lack technological substance and substantive technological theories should be design-oriented. However, we argue that merely technological theories may not lead to an effective and optimal solution for the problem in the domain of reference. We suggest that a balance of both substantive technological theories and situated knowledge drawn from the reference domain is needed for producing useful design and evolving design knowledge. Likewise, Brocke Vom et al. (2020) suggest that it is essential to engage DSR in the problem and solution spaces to maximise its impact. Our study attempted to demonstrate a DSRM approach incorporating both design-oriented substance and kernel theories from the reference domain to offer useful solutions to the identified problem. Our DSRM approach supports the study to provide both design knowledge and artefacts situated in both problem and solution spaces of LA in higher education. As a result, our study also delivers the technical architecture which can be adopted for developing LAIS that embraces the proposed design principles. Furthermore, previous discussions surrounding DSR indicate that a situated implementation of an artefact can be considered as a sufficient knowledge contribution (Baskerville et al., 2018; Gregor & Hevner, 2013). Our case study offers insights into the realisation of LA in the context of higher education.

Higher education has been subject to a series of major challenges in the past decade (Daniel, 2015; Pucciarelli & Kaplan, 2016). The institutions are required to improve their capabilities for the three main missions: teaching, research, and public service. Previous research showed the huge potential of utilising educational data to support institutional activities (Nguyen et al., 2017; Wagner & Ice, 2012). Nevertheless, a systematic understanding of how to apply LA is still lacking. It is also hoped that this work will generate fresh insight into LAIS as a new class of educational IS and contribute to a deeper understanding of underlying design theories for LAIS. Consequently, this study seeks to support higher education institutions to design, develop, and implement LAIS as means to improve their capabilities in teaching, research, and public services.

Dawson et al. (2015) suggested that the design and development of LA should be “better integrated into existing educational research and note the implications for LA research and practice” (p. 65). Our DP1 and DP2 clearly supported this point by calling for affordances to provide theory-inspired actionable insights with the timing that can maximise the effectiveness of reported information. The actionable insights can allow a corrective procedure, or feedback loop, to be established for a set of pedagogic actions (RL & Gynther, 2018). LAIS guided by our DPs would improve learning and teaching by offering these actionable insights to the end-users. For instance, the actionable insights enabling feedback loop would trigger the self-regulated hence enhance learning performance (Yamada et al., 2017).

Our findings also contribute to the literature on educational technology, which has recognised LA as a promising technology in education transformations (Peña-ayala, 2018). The study demonstrates that to successfully use LA, it is necessary to consider the conflicts of interest between different groups of education stakeholders. Information granularity benefits data clients, but it causes ethical concerns for the data subjects. Our study provides empirical evidence confirming the conceptualised sets of ethical issues proposed by Pardo and Siemens (2014) and Slade and Prinsloo (2013).

From the lecturers’ perspective, the findings indicate that the design and development of LAIS should consider the importance of the end-user experience (i.e. the ease of use and access as well as the appropriateness and flexibility of the delivered information). The DSR paradigm and literature encourage scholars to report successes and failures when planning future research (Lobato et al., 2015). Perhaps DSR researchers, as practitioners of teaching in higher education and as scholars, should consider their successes and failures to avoid failures and leverage successes in their future endeavours to apply LA in higher education and, especially, develop LAIS.

The ability to report insightful information has been identified as a salient feature of LA. While this element is conceptualised in our principle of actionable reporting (DP1), we offer a more detailed description of the main affordances of LAIS and explicit guidance for designing such IS, which are essential to support the development and implementation of LAIS and utilise LA in higher education.

8. Concluding remarks

This study proposed LAIS as a class of educational information systems and aimed to establish its design principles. We developed and implemented a fully functioning prototype as an instance of such systems to illustrate the proposed design principles. Through the DSR process of developing and evaluating a set of design principles for LAIS, this study makes both practical and theoretical contributions to the fields of educational technology, IS, and DSR. As e-learning has become a fundamental part of the learning experience in higher education, LAIS presented great opportunities for learning and teaching. The development and implementation of an operational LAIS prototype and the case-study based evaluation show that the proposed LAIS design has the potential to provide teachers with useful LA information. With timely and accurate information about learning and teaching in their classes, lecturers can adjust their pedagogical activities and make appropriate decisions. The LAIS design principles were evaluated and improved from both the lecturer and student perspectives to avoid potential conflicts of interest. Thus, the proposed LAIS design can serve as a guideline for further development and implementation of LA to support learning and teaching. Also, this study provides useful information about the LA services and functionalities that lecturers appreciate to commercial stakeholders, IS developers, and engineers.

This study has some limitations. First, rather than focusing on the fundamental purpose of LA and the requirements from the stakeholders, the design principles of LAIS could be conceptualised differently. Second, although we demonstrated and evaluated our set of design principles, additional rounds of demonstration and evaluation could be conducted in a different context to revise the design principles to ensure their generalisability. Third, we believe that, given a year’s experience with LA, the lecturers will ask for new features and services beyond those that automatically inform them about student engagement. Finally, further research could examine LAIS design principles in the context of different LA modules in higher education.

Step 1: Familiarisation with the research data

The initial step of the analysis involved repeatedly reading of the interview transcripts in an active manner to become immersed in and intimately familiar with the data. While reading and re-reading the manuscripts, we actively looked for potential key patterns and meanings in the interviews.

Step 2: Coding

After becoming familiar with each interview, the transcripts were coded line‐by‐line for specific themes. In accordance with the objective of the interview, we used a deductive approach to develop the coding and themes, and this process was initially directed by existing notions about the two initial themes at the highest level:

(1) The perceived usability and usefulness; and

(2) Difficulties and concerns related to the use of LAIS.

Step 3: Searching for themes

We inspected the codes and collated data to check for patterns of variability and consistency across all transcripts. Furthermore, significant broad patterns of meaning were used to identify any additional potential themes. We also identified more specific subthemes for each candidate theme.

Step 4: Reviewing themes

We examined and refined the candidate themes against the dataset to determine whether they present underlying meanings of the data and address the objective of the interviews. We also reviewed the themes to ensure that the coded extracts of participants’ accounts formed a coherent pattern.

Step 5: Defining and naming themes

We analysed the revised themes in detail and determined the scope and focus of each theme to explore the story of each one. In this step, we developed an informative name for each theme:

• Perceived usability and usefulness and

• Subjective interpretation of reported information.

Step 6: Writing up

We created the analytic narrative and themes and then contextualised the analysis in relation to relevant literature. The results were written and reported in this research paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

Appendix 1. Student Survey Questions and Results

Table

Constructs	Based on	Code	Question
Use of learning analytics by lecturers/teachers (UT)	Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information?
	NGUYEN et al. (2017)	UT01	Early alerts of students that are at risk of failing.
		UT02	Notifications about learning content that students appear to find difficult to understand.
		UT03	A summary of individuals’ learning progress, including when and how a particular student learned.
		UT04	A summary of the overall learning progress of the class, including when and how the class learned.
		UT05	A visualisation of individuals’ learning activities over time.
		UT06	A visualisation of the entire class’s learning activities over time.
Ethical concerns (EC)	Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching?
	PARDO and SIEMENS (2014) Ifenthaler & Mauriello (2016) SLADE and PRINSLOO (2013)	EC01	I am concerned about data anonymity in my courses.
		EC02	I am concerned about data privacy in general.
		EC03	I am concerned about data security.
		EC04	I am concerned about the transparency of the process.
		EC05	I am concerned about the period of time for which data and outcomes will be stored.
		EC06	I am concerned about how the staff could use the information provided by learning analytics.
System quality (SQ)	Regarding the system for lecture recordings, do you think
	Lwoga (2014)	SQ01	The response time of the system is consistent.
		SQ02	The response time of the system is reasonable.
		SQ03	The speed of the Internet connection is acceptable.

Table

UT01. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? Early alerts of students at risk of failing.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	1	.7	.8	.8
	Disagree	2	1.4	1.6	2.4
	Somewhat disagree	1	.7	.8	3.2
	Neither agree nor disagree	12	8.5	9.5	12.7
	Somewhat agree	21	14.8	16.7	29.4
	Agree	45	31.7	35.7	65.1
	Strongly agree	44	31.0	34.9	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT02. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? Notification about learning content that students appear to find difficult to understand.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Neither agree nor disagree	5	3.5	4.0	4.0
	Somewhat agree	9	6.3	7.1	11.1
	Agree	49	34.5	38.9	50.0
	Strongly agree	63	44.4	50.0	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT03. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A summary of individuals’ learning progress, including when and how a particular student learned.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Neither agree nor disagree	4	2.8	3.2	3.2
	Somewhat agree	2	1.4	1.6	4.8
	Agree	5	3.5	4.0	8.7
	Strongly agree	13	9.2	10.3	19.0
	Total	25	17.6	19.8	38.9
Missing	System	41	28.9	32.5	71.4
Total		142	36	25.4	28.6

Table

UT04. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A summary of the class’s learning progress, including when and how the class learned.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	1	.7	.8	.8
	Somewhat disagree	1	.7	.8	1.6
	Neither agree nor disagree	10	7.0	7.9	9.5
	Somewhat agree	25	17.6	19.8	29.4
	Agree	45	31.7	35.7	65.1
	Strongly agree	44	31.0	34.9	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT05. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A visualisation of individuals’ learning activities over time.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.6	1.6
	Disagree	4	2.8	3.2	4.8
	Somewhat disagree	5	3.5	4.0	8.7
	Neither agree nor disagree	16	11.3	12.7	21.4
	Somewhat agree	25	17.6	19.8	41.3
	Agree	35	24.6	27.8	69.0
	Strongly agree	39	27.5	31.0	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT06. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A visualisation of the entire class’s learning activities over time.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.6	1.6
	Disagree	2	1.4	1.6	3.2
	Somewhat disagree	1	.7	.8	4.0
	Neither agree nor disagree	13	9.2	10.3	14.3
	Somewhat agree	27	19.0	21.4	35.7
	Agree	38	26.8	30.2	65.9
	Strongly agree	43	30.3	34.1	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

EC01. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data anonymity in my courses.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	2	1.4	1.8	1.8
	Somewhat disagree	2	1.4	1.8	3.5
	Neither agree nor disagree	19	13.4	16.7	20.2
	Somewhat agree	24	16.9	21.1	41.2
	Agree	28	19.7	24.6	65.8
	Strongly agree	39	27.5	34.2	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC02. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data privacy in general.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	2	1.4	1.8	1.8
	Somewhat disagree	1	.7	.9	2.6
	Neither agree nor disagree	14	9.9	12.3	14.9
	Somewhat agree	24	16.9	21.1	36.0
	Agree	29	20.4	25.4	61.4
	Strongly agree	44	31.0	38.6	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC03. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data security.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	4	2.8	3.5	3.5
	Somewhat disagree	3	2.1	2.6	6.1
	Neither agree nor disagree	13	9.2	11.4	17.5
	Somewhat agree	19	13.4	16.7	34.2
	Agree	34	23.9	29.8	64.0
	Strongly agree	41	28.9	36.0	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC04. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about the transparency of the process.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	3	2.1	2.6	2.6
	Somewhat disagree	2	1.4	1.8	4.4
	Neither agree nor disagree	18	12.7	15.8	20.2
	Somewhat agree	22	15.5	19.3	39.5
	Agree	31	21.8	27.2	66.7
	Strongly agree	38	26.8	33.3	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC05. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about the period of time for which data and outcomes will be stored.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.8	1.8
	Disagree	6	4.2	5.3	7.0
	Somewhat disagree	7	4.9	6.1	13.2
	Neither agree nor disagree	15	10.6	13.2	26.3
	Somewhat agree	23	16.2	20.2	46.5
	Agree	28	19.7	24.6	71.1
	Strongly agree	33	23.2	28.9	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC06. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about how the staff could use the information provided by learning analytics.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	3	2.1	2.6	2.6
	Somewhat disagree	4	2.8	3.5	6.1
	Neither agree nor disagree	18	12.7	15.8	21.9
	Somewhat agree	22	15.5	19.3	41.2
	Agree	35	24.6	30.7	71.9
	Strongly agree	32	22.5	28.1	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

SQ01. Regarding the system for lecture recordings, do you think … The response time of the system is consistent.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.7	1.7
	Disagree	3	2.1	2.5	4.2
	Somewhat disagree	3	2.1	2.5	6.7
	Neither agree nor disagree	18	12.7	15.0	21.7
	Somewhat agree	23	16.2	19.2	40.8
	Agree	41	28.9	34.2	75.0
	Strongly agree	30	21.1	25.0	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Table

SQ02. Regarding the system for lecture recordings, do you think … The response time of the system is reasonable.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.7	1.7
	Somewhat disagree	2	1.4	1.7	3.3
	Neither agree nor disagree	15	10.6	12.5	15.8
	Somewhat agree	22	15.5	18.3	34.2
	Agree	49	34.5	40.8	75.0
	Strongly agree	30	21.1	25.0	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Table

SQ03. Regarding the system for lecture recordings, do you think … The speed of the Internet connection is acceptable.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	1	.7	.8	.8
	Disagree	1	.7	.8	1.7
	Somewhat disagree	7	4.9	5.8	7.5
	Neither agree nor disagree	10	7.0	8.3	15.8
	Somewhat agree	22	15.5	18.3	34.2
	Agree	50	35.2	41.7	75.8
	Strongly agree	29	20.4	24.2	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Appendix 2. Process for Thematic Analysis of Interviews with the Lecturers

As the main objective of the interviews was to evaluate the use of LAIS and explore any side effects or undesirable consequences, the thematic analysis is constructionist in that it explores how a certain reality is created by the data. We analyse the interview using a recursive process consisting of six steps: