Design principles for learning analytics information systems in higher education

Figures & data

Figure 1. Learning analytics information system (LAIS) architecture

Figure 2. Key learning analytics (LA) services

Table 1. The functionality of learning analytics (LA) services

LD Phase	Learning Management Question	LA Service
1. Profiling	What does the learner already know?	Learner Profiling: • Prior knowledge sets • Personal aspirations • Preferred learning styles Prediction: • Performance
	Where does the learner need and want to be?
	How does learner best learn?
2. Strategising and Delivery	What resources do I have at my disposal?	Decision Making: • Appropriate instructional resources and/or strategies Prediction: • Scenario planning
	What will constitute the learning journey, and therefore, what is the best context for learning?
	Who will do what?
3. Ascertainment and Reporting	How will I check whether the learner has achieved the learning outcomes?	Situation Detection: •Notification of at-risk situations Evaluation: • Information on learning engagement Decision Making: • Recommended interventions
	How will I inform the learner and others about the learner’s progress?

Table 2. Design of the learning analytics information system prototype

Data Captured

Reported Information

Events: • load_video • play_video • pause_video • seek_video • speed_change_video • stop_video. Metrics: • Course information • Video metadata (e.g., title, length.)

Metrics: • Number of active students in each week • Average video views each week • Number of students who watched a particular video (unique viewers) • Number of replays at any point in the video (replayed segments) Potential Use: • To identify the parts that students most frequently skip or replay → Adjustment of potentially problematic parts of the learning materials • To monitor students’ engagement with learning → Early interventions if there is a lack of engagement → Evaluation of the effectiveness of learning design improvements on subsequent student engagement

Figure 3. An example of a graph reported to lecturers

Figure 4. The learning analytics information system’s dashboard indicates a lack of engagement in a timely manner

Figure 5. The learning analytics information system’s dashboard shows the most engaging activities

Figure 6. Total number of interaction events for each video in the large first-year course over time

Figure 7. An example of the effect of the lecturer’s intervention on student engagement

Figure 8. Thematic analysis process for interviews with the lecturers

Table 3. LAIS design principles

#	Design principle	Design principle specification
DP1	Principle of actionable information	Should provide reports of actionable information about learners and their learning with flexible granularity in reporting.
DP2	Principle of information delivery	Should generate responses and information that visualise learning and teaching behaviour and performance.
DP3	Principle of information interoperability	Should interoperate with any LA and/or educational IS, including VLEs, and enable integration with different data sources without resulting in any discernible issues or complications.
DP4	Principle of information anonymity and protection	Should provide anonymity for personal and protect data against accidental or unlawful destruction or accidental loss, alteration, unauthorised disclosure or access.

Table

Constructs	Based on	Code	Question
Use of learning analytics by lecturers/teachers (UT)	Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information?
	NGUYEN et al. (2017)	UT01	Early alerts of students that are at risk of failing.
		UT02	Notifications about learning content that students appear to find difficult to understand.
		UT03	A summary of individuals’ learning progress, including when and how a particular student learned.
		UT04	A summary of the overall learning progress of the class, including when and how the class learned.
		UT05	A visualisation of individuals’ learning activities over time.
		UT06	A visualisation of the entire class’s learning activities over time.
Ethical concerns (EC)	Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching?
	PARDO and SIEMENS (2014) Ifenthaler & Mauriello (2016) SLADE and PRINSLOO (2013)	EC01	I am concerned about data anonymity in my courses.
		EC02	I am concerned about data privacy in general.
		EC03	I am concerned about data security.
		EC04	I am concerned about the transparency of the process.
		EC05	I am concerned about the period of time for which data and outcomes will be stored.
		EC06	I am concerned about how the staff could use the information provided by learning analytics.
System quality (SQ)	Regarding the system for lecture recordings, do you think
	Lwoga (2014)	SQ01	The response time of the system is consistent.
		SQ02	The response time of the system is reasonable.
		SQ03	The speed of the Internet connection is acceptable.

Table

UT01. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? Early alerts of students at risk of failing.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	1	.7	.8	.8
	Disagree	2	1.4	1.6	2.4
	Somewhat disagree	1	.7	.8	3.2
	Neither agree nor disagree	12	8.5	9.5	12.7
	Somewhat agree	21	14.8	16.7	29.4
	Agree	45	31.7	35.7	65.1
	Strongly agree	44	31.0	34.9	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT02. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? Notification about learning content that students appear to find difficult to understand.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Neither agree nor disagree	5	3.5	4.0	4.0
	Somewhat agree	9	6.3	7.1	11.1
	Agree	49	34.5	38.9	50.0
	Strongly agree	63	44.4	50.0	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT03. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A summary of individuals’ learning progress, including when and how a particular student learned.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Neither agree nor disagree	4	2.8	3.2	3.2
	Somewhat agree	2	1.4	1.6	4.8
	Agree	5	3.5	4.0	8.7
	Strongly agree	13	9.2	10.3	19.0
	Total	25	17.6	19.8	38.9
Missing	System	41	28.9	32.5	71.4
Total		142	36	25.4	28.6

Table

UT04. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A summary of the class’s learning progress, including when and how the class learned.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	1	.7	.8	.8
	Somewhat disagree	1	.7	.8	1.6
	Neither agree nor disagree	10	7.0	7.9	9.5
	Somewhat agree	25	17.6	19.8	29.4
	Agree	45	31.7	35.7	65.1
	Strongly agree	44	31.0	34.9	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT05. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A visualisation of individuals’ learning activities over time.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.6	1.6
	Disagree	4	2.8	3.2	4.8
	Somewhat disagree	5	3.5	4.0	8.7
	Neither agree nor disagree	16	11.3	12.7	21.4
	Somewhat agree	25	17.6	19.8	41.3
	Agree	35	24.6	27.8	69.0
	Strongly agree	39	27.5	31.0	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

UT06. Would you agree that the learning system (e.g., Canvas) should provide lecturers the following information? A visualisation of the entire class’s learning activities over time.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.6	1.6
	Disagree	2	1.4	1.6	3.2
	Somewhat disagree	1	.7	.8	4.0
	Neither agree nor disagree	13	9.2	10.3	14.3
	Somewhat agree	27	19.0	21.4	35.7
	Agree	38	26.8	30.2	65.9
	Strongly agree	43	30.3	34.1	100.0
	Total	126	88.7	100.0
Missing	System	16	11.3
Total		142	100.0

Table

EC01. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data anonymity in my courses.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	2	1.4	1.8	1.8
	Somewhat disagree	2	1.4	1.8	3.5
	Neither agree nor disagree	19	13.4	16.7	20.2
	Somewhat agree	24	16.9	21.1	41.2
	Agree	28	19.7	24.6	65.8
	Strongly agree	39	27.5	34.2	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC02. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data privacy in general.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	2	1.4	1.8	1.8
	Somewhat disagree	1	.7	.9	2.6
	Neither agree nor disagree	14	9.9	12.3	14.9
	Somewhat agree	24	16.9	21.1	36.0
	Agree	29	20.4	25.4	61.4
	Strongly agree	44	31.0	38.6	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC03. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about data security.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	4	2.8	3.5	3.5
	Somewhat disagree	3	2.1	2.6	6.1
	Neither agree nor disagree	13	9.2	11.4	17.5
	Somewhat agree	19	13.4	16.7	34.2
	Agree	34	23.9	29.8	64.0
	Strongly agree	41	28.9	36.0	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC04. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about the transparency of the process.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	3	2.1	2.6	2.6
	Somewhat disagree	2	1.4	1.8	4.4
	Neither agree nor disagree	18	12.7	15.8	20.2
	Somewhat agree	22	15.5	19.3	39.5
	Agree	31	21.8	27.2	66.7
	Strongly agree	38	26.8	33.3	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC05. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about the period of time for which data and outcomes will be stored.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.8	1.8
	Disagree	6	4.2	5.3	7.0
	Somewhat disagree	7	4.9	6.1	13.2
	Neither agree nor disagree	15	10.6	13.2	26.3
	Somewhat agree	23	16.2	20.2	46.5
	Agree	28	19.7	24.6	71.1
	Strongly agree	33	23.2	28.9	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

EC06. Are you concerned about ethical issues regarding the use of learning data to improve learning and teaching? I am concerned about how the staff could use the information provided by learning analytics.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Disagree	3	2.1	2.6	2.6
	Somewhat disagree	4	2.8	3.5	6.1
	Neither agree nor disagree	18	12.7	15.8	21.9
	Somewhat agree	22	15.5	19.3	41.2
	Agree	35	24.6	30.7	71.9
	Strongly agree	32	22.5	28.1	100.0
	Total	114	80.3	100.0
Missing	System	28	19.7
Total		142	100.0

Table

SQ01. Regarding the system for lecture recordings, do you think … The response time of the system is consistent.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.7	1.7
	Disagree	3	2.1	2.5	4.2
	Somewhat disagree	3	2.1	2.5	6.7
	Neither agree nor disagree	18	12.7	15.0	21.7
	Somewhat agree	23	16.2	19.2	40.8
	Agree	41	28.9	34.2	75.0
	Strongly agree	30	21.1	25.0	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Table

SQ02. Regarding the system for lecture recordings, do you think … The response time of the system is reasonable.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	2	1.4	1.7	1.7
	Somewhat disagree	2	1.4	1.7	3.3
	Neither agree nor disagree	15	10.6	12.5	15.8
	Somewhat agree	22	15.5	18.3	34.2
	Agree	49	34.5	40.8	75.0
	Strongly agree	30	21.1	25.0	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Table

SQ03. Regarding the system for lecture recordings, do you think … The speed of the Internet connection is acceptable.
		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Strongly disagree	1	.7	.8	.8
	Disagree	1	.7	.8	1.7
	Somewhat disagree	7	4.9	5.8	7.5
	Neither agree nor disagree	10	7.0	8.3	15.8
	Somewhat agree	22	15.5	18.3	34.2
	Agree	50	35.2	41.7	75.8
	Strongly agree	29	20.4	24.2	100.0
	Total	120	84.5	100.0
Missing	System	22	15.5
Total		142	100.0

Nguyen, A., Gardner, L., & Sheridan, D. (2017) A multi-layered taxonomy of learning analytics applications. In Pacific Asia Conference on Information Systems (PACIS) 2017 Proceedings, Langkawi, Malaysia. Available at: http://aisel.aisnet.org/pacis2017/54/.

 Google Scholar

Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438–450. https://doi.org/10.1111/bjet.12152

 Web of Science ®Google Scholar

Slade, S., & Prinsloo, P. (2013). Learning analytics ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. Available at: https://doi.org/10.1177/0002764213479366

 Web of Science ®Google Scholar

Lwoga, E. (2014). Critical success factors for adoption of web-based learning management systems in Tanzania. International Journal of Education and Development using ICT, 10(1).Chicago. Available at: https://www.learntechlib.org/p/147447/

 Google Scholar