Development and evaluation of a visualization system to support meaningful e-book learning

ABSTRACT

This study presents an ontology-based visualization support system for e-book learners which promotes both meaningful receptive learning and meaningful discovery learning. To examine the system effectiveness, two learning modes are used: (a) reception comparison mode, where at the outset learners are shown complete versions of expert-generated topic maps; and (b) “cache-cache comparison mode,” where at the first stage of learning all information concerning relations is concealed, and at the second stage learners are encouraged to actively create those relations before comparing the learner-generated and expert-generated relations. The 50 control group participants studied in reception comparison mode while the 146 experimental groupparticipants studied in cache-cache comparison mode. Differences in learning perception and achievement between the two groups are examined, as is the effect of learner expertise level on learning mode effectiveness. Although the control group reported significantly more pressure and less satisfaction than the experimental group, no significant learning achievement differences were found between the two groups. However, in cache-cache comparison mode, the performance of learners with low prior knowledge increased more than that of learners with high prior knowledge; on the other hand, for learners with high prior knowledge, no significant effect of learning mode on learning achievement was found.

KEYWORDS:

• Meaningful learning
• discovery learning
• ontology
• topic map
• e-book

1. Introduction

The hierarchical nature of knowledge has been elucidated in many studies (Ausubel et al., 1978; Bruner, 1961; Rand, 1990). Evidence from numerous studies suggests that learning approaches that support the organization of knowledge in a hierarchical cognitive structure significantly increase the learning performance of learners (Bransford et al., 1999; Tsien, 2007). The assimilation learning theory (Ausubel, 1963; Ausubel et al., 1978) defines “meaningful learning” as the non-arbitrary substantive incorporation of new concepts or propositions into the existing hierarchical framework of cognitive structure. The process of meaningful learning involves the learner understanding the potential meaning of new learning material, connecting it with relevant concepts or propositions that he/she possesses, and finally expanding, reorganizing or reconstructing his/her existing cognitive structure (Ausubel, 1963; Ausubel et al., 1978). The findings of Ausubel et al. (1978) also indicate that rote learning (mechanical learning), which mainly involves memorization by repetition and checking without recognition of the meaning, contributes little to the modification of cognitive structure and is distinctly different from meaningful learning. Nevertheless, Novak (1993) points out that since learners with different quantity and quality of prior knowledge may realize different number and quality of associations between new learning material and their existing prior knowledge, and that this leads to the occurrence of rote learning (little association of new knowledge with existing cognitive structure) or different levels of meaningful learning. In that sense, rote learning and meaningful learning can be seen as distinct points on a continuum. Therefore, encouraging the learner to identify significant associations between new concepts or propositions (underlying new learning materials) and his/her existing cognitive structure is essential for facilitating high levels of meaningful learning.

In recent years, e-books have gradually replaced traditional hard-copy textbooks in education due to certain advantages such as cost-saving and greater portability (Shepperd et al., 2008; Shin, 2012; Yin et al., 2018). Many universities now support daily classroom teaching with e-book systems used in tandem with learning management systems (LMSs, such as Moodle) or other e-learning systems. These combinations of e-book systems and e-learning systems provide a platform where instructors can easily upload teaching materials that the learners can conveniently view and annotate or comment. These systems can also record learning behaviour and report the results to the instructors. However, these e-learning systems do not support the learner in identifying the knowledge he/she possesses before and after a learning activity. Furthermore, existing e-book systems and LMSs (even other e-learning systems) do not encourage the learner to compare new knowledge with relevant previously acquired or learned knowledge (Wang et al., 2014), thus they cannot effectively support the learner in the construction of personal knowledge structures – and as a result cannot foster high levels of meaningful learning. In other words, learners who use e-books superficially can still easily fall into the rote learning that is often seen in learning with traditional textbooks.

To address the problem of superficial e-book use, Wang et al. (2017) present a framework intended to provide a meaningful learning environment for e-book learners using an ontology driven engine. Based on this framework, we develop here a visualization support system (VSSE) for assisting the e-book user in identifying and understanding concepts or propositions appearing within any page range of an e-book and in relating them to his/her cognitive structure. VSSE provides both a meaningful reception learning environment (in which the e-book learner is provided at the outset with expert-generated topic maps of the target e-book content) and a meaningful discovery learning environment (in which the e-book learner is encouraged at the outset to seek hidden relations in knowledge topic maps of the target e-book content).

Independent of meaningful learning-rote learning dimension, which focuses on how knowledge is incorporated into the learner’s existing schema, “reception learning” and “discovery learning” are two fundamentally different types of learning in another dimension related to how the knowledge is presented to the learner (Ausubel et al., 1978; Novak & Cañas, 2008). In reception learning, knowledge is presented in its complete form, therefore the learner only needs to internalize it; on the other hand, in discovery learning, the learners have to discover knowledge themselves before internalization. Bruner (1961, 2009) indicates that learners are more likely to remember knowledge that they discover on their own than that which is presented directly in receptive instruction mode. He (Bruner, 1961) defines discovery learning as an inquiry-based, constructivist learning theory-based process that takes place in problem solving situations where the learner draws on his or her own experience and acquired knowledge to discover facts, relations or new truths. It is claimed that this application process, which encourages active engagement, can foster the development of creativity and problem-solving skills, and promote learning motivation (Bruner, 1961, 2009). However, many researchers (Alfieri et al., 2011; Ausubel et al., 1978; Mayer, 2004) have cautioned that unassisted discovery learning without sufficient prior knowledge and guidance may easily lead to misconceptions and cause additional cognitive load. As well, timely guidance is needed in discovery learning to avoid learner confusion and frustration (Kirschner et al., 2006). Learners need to gain confidence in their ability to complete tasks, given the requisite knowledge; on the other hand, when confronted with failure they also need to be encouraged to learn from that and thus have sufficient motivation to continue learning.

In the VSSE meaningful discovery learning environment, the relations among knowledge points are identified autonomously by the learner, whereas in the reception learning environment, relations are provided explicitly in the map for transmission to the learner. It is worthy of note that implementation of discovery learning does not ensure meaningful learning (Ausubel et al., 1978). Unless the learner possesses at least a rudimentary conceptual understanding of the nature of an investigation task, a discovery learning activity may contribute little or nothing to the construction of the learner's knowledge framework (Novak & Cañas, 2008). In that light, this study examines the influence of meaningful learning environments (meaningful reception learning and meaningful discovery learning) and individual differences in prior knowledge on learning effectiveness.

2. Mapping and ontology technique

In the interest of encouraging meaningful learning, maps consisting of nodes (key concepts) and links (relationships between nodes) can provide scaffolding to help learners to organize knowledge and structure their own knowledge frameworks (Novak & Cañas, 2008). In the knowledge presentation literature, three main types of maps are used: concept maps, knowledge maps and topic maps (Lee & Segev, 2012).

Notably, concept maps are constructed with reference to a focus question. “Concept” is defined as “a perceived regularity in events or objects, or records of events or objects, designated by a label” (Novak & Cañas, 2008). Concept maps are normally designed for the organization in map form of knowledge pertaining to some situation or event, so as to facilitate the understanding of that knowledge. The main difference between knowledge maps and concept maps is the deliberate use in knowledge maps of a common set of labeled links that connect ideas (O’Donnell et al., 2002).

Some empirical studies on concept mapping (Lee & Nelson, 2005; Lim et al., 2009) report that learners studying in the full learner-generated map significantly outperformed those studying in the expert-generated map. They claim that more cognitive engagement in reflection on the relations between knowledge concepts will lead to increased generation of meaningful learning (Nesbit & Adesope, 2013). However, the target learning content in those experiments was generally rather easy, so the participants did not have much difficulty in understanding the discovery task required for making the concept map. For example, the target learning content in the experiment of Lee and Nelson (2005) is simply printed materials including five basic concepts about instruction planning and design; and the target learning content in the experiment of Lim et al. (Lim et al., 2009) is a 1900-word expository text describing the human heart, accompanied by corresponding visual images. There is a need for further investigation and discussion of learner performance when studying content of moderate difficulty in concept map.

In this study, the meaningful learning environment provides topic maps in which the learner associates corresponding content elements of learning materials, including definitions and explanations of knowledge (from the topic maps) and e-book content, with the knowledge structure provided. Different from concept maps (Chu et al., 2011; Novak & Cañas, 2008) and knowledge maps (Lee & Segev, 2012; O'Donnell et al., 2002), which are used as learning materials in knowledge representation, topic maps are mainly used as metadata of learning materials (Wang et al., 2014).

Since ontology is flexible in terms of description of map structure and enabling the merging of different sources, it affords a viable means of modelling a hierarchical knowledge network in which nodes represent concepts and arcs or arrows represent the relations between those concepts. “An ontology is a formal explicit specification of a shared conceptualization” (Gruber, 1993). Common vocabularies can be usefully defined by ontology for users (such as instructors, learners and researchers) who need to share information in a domain (Noy & McGuinness, 2001).

Several knowledge-based systems (Chu et al., 2011; Dicheva & Dichev, 2006; Wang et al., 2014; Zhong et al., 2015) have utilized ontology techniques to support knowledge mapping. The concept map learning system of Chu et al. (2011), intended to help develop learner reflection and establish fully detailed knowledge structure, is implemented based on ontology technology; to access a learner’s mastery of knowledge, the geometric intelligent assessment system of Zhong et al. (2015) employs a hybrid cognitive assessment method that considers not only the declarative knowledge described in a disciplinary ontology but also the procedural knowledge described in a problem solving process; TM4L (Dicheva & Dichev, 2006), a specialized environment for creating, maintaining and using “topic map-based” learning repositories, also is structured around an ontology-based engine. The innovative contribution of the ontology-based visualization support system in this study to the above literature array is the provision of personalized topic maps based on the learner's e-book log and search interests.

3. A visualization learning support system for E-book users

To construct a demonstration unit, first the ontology design method described by Wang and Mendori (2012) and Wang et al. (2014) was adjusted and applied to the development of a course-centered ontology for an existing computer science course (entitled COCS). The ontology consists of about one hundred knowledge points and twenty kinds of relations, extracted and defined based on an analysis of the content of all the e-books in this computer science course. In this study, a knowledge point (KP) is defined as “a minimum learning item which can independently describe the information constituting one given piece of knowledge in the content of a specific course.” The learner can understand a KP via its own expression or can acquire it through practice. As to relations in COCS shown in Table 1, they are designed to indicate e-book location of KP or upper concepts, dependences between KPs, and similarities and contrasts between KPs.

Table 1. Frequency and design notes of all the relations of COCS.

Function of relation	Relation name	Frequency	Type	Design notes
Indicate location information	isLocatedAfter/isLocatedBefore	28	Inverse	Used to show the relative e-book location of the two upper concepts.
	include /locatedIn	531	Inverse	Used to record the KPs appearing in one e-book page or the location information of one KP.
Indicate dependences between concepts	hasNecessaryPrior	20		Used in A to B orientation when the understanding of concept A requires the acquisition of concept B
	isRelatedTo	4	Symmetric	Used between a proposition or theorem and a KP when one or more KPs are combined to generate a proposition or theorem
	comprise	6		Used when the representation of a KP consists of several parts represented by other KPs
	isATypeOf/ hasType	7	Inverse	Used to group several KPs by linking those KPs with one basic KP.
	isAnInstanceOf /hasInstance	7	Inverse	Used to indicate a specific case of the representation of a KP.
	encode/canBeEncodedBy	5	Inverse	Used when the representation of one KP defines the standardized code of the representation of another KP.
	isPrescribedBy/prescribe	3	Inverse	Used to indicate a definition described by an approach or a standard or a schema.
	canBeCalculatedFrom/canBeUsedToCalculate	1	Inverse	Used to indicate the input and output of a calculation.
Indicate similarities and contrasts between concepts	isCompatibleWith	3	Symmetric	Used to show the compatibility between two approaches or schema.
	Extend/hasExtension	2	Inverse	Used between two KP of the same type when one representation of one KP is modified based on the representation of the other KP.

To support e-book users in effectively constructing their own knowledge structures, a visualization learning support system using COCS as engine was implemented. Two learning modes are provided: (a) reception comparison mode, in which the learner is provided with complete versions of topic maps at the outset; and (b) cache-cache comparison mode, where at the outset the learner is encouraged to actively create the relations between knowledge concepts.

3.1. Reception comparison mode in VSSE

In the reception comparison mode, VSSE provides learners with topic maps extracted from an expert-generated ontology to elucidate the knowledge concepts and the relations among them. Those maps are diagrams in which labeled nodes represent concepts and the labeled links that connect the nodes represent relationships between the concepts. Those pre-constructed maps with specifications of concepts and relationships are automatically extracted from the expert-generated ontology COCS.

Since a single node represents a concept, relations that refer to the same concept are always visually close. This feature enables use of these node-link-node structures to allow the learner to easily compare a given concept with all its related concepts simultaneously. When expert-generated ontology maps are provided, first all the multiple occurrences of a concept in e-book learning materials are merged into a single labeled node; second, related concepts are placed closer and connected by labeled links so that learners can readily compare them; finally, the number of links connected to one concept indicates the concept’s importance to the learner.

The three main functions provided in the reception comparison mode of VSSE are described below.

3.1.1. Ubiquitous visualization of knowledge

The first main function provided in the reception comparison mode of VSSE is the display of a topic map containing the KPs appearing within any page range of any e-book, along with the upper concepts of those KPs. As can be seen in Figure 1, users of the e-book system can select a specific e-book and input interested page range at the top of the VSSE interface. VSSE will display all the KPs appearing in the searched pages along with their related KPs.

Figure 1. The topic map when searching page 9–11 of e-book A-08.

An example is shown in Figure 1, which displays: red nodes, which represent the KPs that appear in pages 9–11 of e-book A-08; blue nodes, which represent related KPs that do not appear in those pages but have essential relations with the KPs represented by the red nodes; and pink nodes, which represent the upper concepts of the KPs represented by red or blue nodes. When the user hovers the mouse arrow over any node in this topic map, the essential properties (such as definition and explanation, represented by the data properties of one individual in COCS) of that KP will be listed, while for every arc in the topic map, a statement of the relation represented by that arc will be displayed (for example, the displayed relation axiom “can be used to calculate” from “Uniquely Decodable Codes” to “Prefix Codes” in Figure 1). Therefore, using this visualization map users can conveniently find the essential properties of any given KP and all of its related KPs. All of that information is extracted automatically from COCS.

3.1.2. The visualization of any KP and its related KPs

The second main function provided in the reception comparison mode of VSSE is catalogue searches of the information about each KP included in the e-books. As shown in the left part of Figure 2, all the learning contents included in the e-books are organized in a tree structure. The user can open all the concepts level by level until he/she reaches the KP being sought. When the user double clicks one leaf (which represents one KP), the right relations panel will display that KP and all its related KPs linked by relations defined in COCS.

Figure 2. Hovering the mouse over “prefix codes” after clicking “optimal codes”.

For instance, in Figure 2, after the item representing the KP “Optimal Codes” is clicked from the catalogue, the right panel shows that there is reference to KP “Optimal Codes” on pages 1, 13, 14, 15, 16, 17, 18, 19 and 24 of E-book A08 and that “Optimal Codes” has 2 related KPs (“Average Codeword Length” and “Prefix Codes”). Also, “Average Codeword Length” and “Prefix Codes” are linked to “Optimal Codes” in a visual topic map. As is the case for the previous function, with this visualization map users can conveniently browse the essential properties of every KP by hovering the mouse over the node (for instance, the definition of KP “Prefix Codes” is displayed in Figure 2 when hovering the mouse over its representation) or get the statement of its relations by hovering the mouse over arcs.

3.1.3. Visualization of knowledge learned during any period

The third main function provided in the reception comparison mode of VSSE is e-book user search of the knowledge accessed during any learning period, using the e-book system log data. When a e-book learner input a time period, VSSE will display a topic map containing all the KPs included in the e-book pages which have been read during that given period along with their related KPs (Wang et al., 2017).

3.2. Insights from a previous study on a meaningful reception learning environment

Wang et al. (2013) developed CLLSS, an ontology-based language learning support system with a meaningful receptive learning environment. In CLLSS, whenever a new KP is studied by the learner, the relations between the new KP and its related KPs will be presented in a topic map to assist the learner to acquire full understanding. The interface of CLLSS is similar to that of VSSE, as shown in Figure 2, but without the e-book location information. The difference between CLLSS and VSSE is that in CLLSS the learners can further access learning materials (including the explanation files and various practices files) via the topic map.

It was found that the subjects who studied in CLLSS achieved significantly better learning outcomes than those who did only self-study with textbooks after studying the same target Japanese grammar content (Wang et al., 2014). This suggests that with this visualization support, new knowledge can readily be comprehended and retained. However, participants reported that they felt pressure and were disturbed when more than four related KPs were shown at one time. In other words, with regard to cognitive load, the e-learning environment should avoid giving too much information at one time.

In terms of learning attitude and motivation, the analysis results suggest improvement in both learner attitude towards Japanese grammar learning and motivation to learn Japanese language after studying with CLLSS (Wang & Mendori, 2015). Furthermore, learners with high attitude and motivation levels reported (a) greater development of the habit of “learning by comparing related knowledge” and (b) more satisfaction with the CLLSS environment learning mode. Moreover, compared to learners with low level pre-activity attitude towards Japanese grammar, learners with high level pre-activity attitude reported significantly less mental effort in study with CLLSS and performed better on the grammar post-test. These results confirm that learning attitude and motivation are factors that must be considered in the promotion of meaningful learning. However, 32% of participants reported that since CLLSS already provides numerous bits of related knowledge, they didn't have the inclination to proactively search for more knowledge. Furthermore, several participants reported that their curiosity and willingness to seek more related knowledge decreased. This result suggests that since CLLSS directly displays the information about related concepts and relations, the participants made comparisons between concepts in a passive receptive manner. This passive learning might impact negatively on learner willingness to explore. This indicates that CLLSS needs to be modified so as to encourage learners to actively engage in the construction of topic maps.

3.3. Cache-cache comparison mode in VSSE

CLLSS provides expert-generated topic maps; another option is to encourage the learner to create meaningful concepts as nodes and connect them spatially with meaningful links. However, the creation of a topic map from scratch requires that the learner first identify the hierarchical concepts from e-book materials, and then discover and specify the relations between the leaf concepts. This is onerous work for novices: even just creating nodes that represent the concepts embedded in an e-book is daunting. Clearly more attention should be paid to the discovery and articulation of the relationships between concepts. Therefore, VSSE provides a cache-cache mode that can display the concepts (including knowledge points and their upper concepts) which meet certain conditions and encourage the learner to notice, abstract and then specify the relationships between each KP and its upper concepts and also between KPs before comparing with expert-generated relations. In this mode, all the information concerning the relations are hidden at the first stage of learning, and the learners are encouraged to actively discover those hidden parts in the second stage. Finally, the hidden information of the relations, extracted from ontology, will be displayed for the learners to compare with the relations that they designed themselves. This process, involving discovery learning, is termed “cache-cache comparison” here. In other words, VSSE provides a supportive environment for the completion of partially learner-generated maps. This environment provides an integration of discovery learning (for the three main functions of reception comparison mode mentioned in 3.1 above) to encourage active engagement in meaningful learning.

Figure 3 shows an instance of cache-cache comparison mode: the content of interest to the learner is pages 9–11 of e-book A-08. First, as shown in Figure 3(a), cache-cache comparison mode displays all the KPs that appear in the page range of interest in red; the related KPs that do not appear in the pages of interest in blue; and the upper concepts of all those KPs in pink. The learner is required to classify the KPs by connecting them to their pink upper concepts, as shown in Figure 3(b). Next, the learner is encouraged to find out the relations between KPs by connecting red nodes, or connecting red nodes to blue nodes. The descriptions of the relation arcs made by the learner can be modified and saved anytime. VSSE will automatically adjust the position of the nodes in the map so that the multiply-connected nodes will be placed more centrally to reduce the number of link crossings. Conversely, the deletion of a link will also cause nodes to change position. Finally, after the topic map is completed, the “Compare with experts” button should be clicked. As shown in Figure 3(c), all the relations extracted from the ontology will be displayed as red lines. The learner can easily compare those red lines with the black lines that she/he has made.

Figure 3. An instance of “cache-cache comparison” mode.

The cache-cache comparison mode is intended to support the learner in actively locating new knowledge in her/his knowledge framework and checking the logical consistency of her/his ideas. When the learner connects the KPs to their upper concepts, he/she must determine which KPs belong to the same general concept; when the learner connects two KPs, he/she needs to judge if there are similarities, contrasts or dependencies between them. This elaborative processing is intended to promote meaningful learning because the learner has to fully understand the meaning of each concept and discover similarities, contrasts or dependencies between concepts by actively reviewing prior knowledge. It is expected that this active engagement will help avoid erosion of curiosity and willingness to explore, as was the case for the CLLSS meaningful reception learning environment.

This mode is not only a learning tool; it also serves as an evaluation tool which encourages learners to construct their knowledge using meaningful learning strategies. It can be used to identify the relevant knowledge a learner possesses before and after a learning activity. This mode also effectively identifies learner misunderstandings of relations and aids the learner to overcome those misconceptions.

4. Research design

In our search for a learning mode that is optimal in terms of both learning effectiveness and perception of learning, two research questions emerged which directed the design of this study, as follows.

1. In VSSE, what are the differences in the learning performance (including learning achievement and perception of learning) of (a) participants who studied in the meaningful reception environment and (b) those who studied in the meaningful discovery environment?

To answer this question, reception comparison mode and cache-cache comparison mode for the support of meaningful learning, were provided via VSSE to learn the content of selected pages of an existing course. In reception comparison mode, learners were provided directly with full-information topic maps (example shown in Figure 1), while in cache-cache comparison mode all relations between KPs are hidden at the first stage of learning, and at the second stage the learners are encouraged to actively discover those relations and articulate them (example shown in Figure 3).

• (2) Does learning mode effectiveness depend on level of learner expertise?

According to expertise reversal effect theory (Sweller et al., 2003), an instructional technique that is beneficial for novices may be disadvantageous for more experienced learners. Therefore, this study examines the relationship between (a) learning mode effectiveness and (b) level of learner expertise. The experimental results are expected to suggest an optimum learning mode able to respond to demonstrated learner aptitude.

5. Experimental description

5.1. Participants and measurement techniques

Given the situation of having to use intact classrooms, this study used a quasi-experimental pretest-posttest group design to investigate the two research questions mentioned in the Section 4. One hundred and ninety-six first-year undergraduates from two classes at a University in Japan participated in this study. These students were all taught by the same instructor, who had taught computer science for around ten years. Before the experiment, all the students had studied image processing for eight weeks in learning support system environments (Moodle, Mahara and an e-book system).

Learning performance measurement techniques in this experiment included learning achievement tests (pre- and post-test), and a questionnaire for measuring learning related perceptions. Both test sheets had been developed by experienced teachers. The questionnaire consisted of 11 questions involving responses on a seven-point Likert scale (1-3: strongly to slightly disagree, 4: neutral, 5-7: slightly to strongly agree). Question content was related to learning perception, specifically technology acceptance (Chu, Hwang, Tsai, and Tseng, 2010; Davis, 1989), cognitive load (Sweller et al., 2003), and satisfaction with learning mode (Chu, Hwang, and Tsai, 2010). The Chinese version of this questionnaire was used in the previous study (Wang et al., 2014). In this study, the Japanese version of the questionnaire was used.

5.2. Experimental procedure

As can be seen in Figure 4, in the fifth week of the course all participants took the pre-test at the beginning of the class and then studied e-book c03, entitled, “Digital Image Processing.” The pre-test, aimed at evaluating participants’ prior knowledge of digital image processing, contained of three single-choice questions and one multiple-choice question with a perfect score of 10 (all quesitons weighted equally). In the ninth week, at the beginning of the class the purpose and objectives of the learning activity were explained to all participants. The first 15 pages of e-book c03 were chosen as target learning content for the students’ review work. Eleven KPs appear in those 15 pages: “digital image,” “noise reduction,” “robot vision,” “industrial image processing,” “medical image processing,” “image processing for security,” “image processing service,” “contrast conversion,” “pixel,” “raster scanning” and “image filtering.” These 11 KPs mostly occur under the following six upper concepts: “image,” “image editing,” “application of digital image processing,” “image correction and conversion,” “image digitization” and “digital image processing.”

Figure 4. Experimental procedure.

Subsequently, both classes (one as experiment group and another as control group) received training in the use of the Japanese version of VSSE (Wang et al., 2017), which can be opened in the browser of any PC, tablet or smart phone. During the 15-minute training, the study procedures were demonstrated to the experimental and the control group, using one sample map in cache-cache comparison mode and reception comparison mode, respectively; participants in both groups were then encouraged to repeat the demonstrated actions so as to familiarize themselves with system operation in the two modes. After the training, the control group studied the first 15 pages of e-book c03 with reception comparison mode support while the experimental group studied with cache-cache comparison mode support. For both groups, the target content learning activity was of 25 minutes’ duration.

After the learning activity, all participants took a post-test and responded to the questionnaire. The post-test, also with a perfect score of 10, contained six cloze tasks (all tasks weighted equally), of which three concerned single KPs and three addressed relations between KPs. Those six items were designed to assess (after the learning activity) participant performance in the solving of problem tasks involving target content.

6. Results

6.1. Analysis focusing on learning achievement

No significant correlation was found between the pre-test and post-test scores of the two groups (experimental: r = −0.141, control: r = 0.127). Therefore, ANCOVA could not be performed. Figure 5 shows the boxplots for the pre-test and the post-test scores of both groups. Table 2 shows the descriptive data and ANOVA results for the pre-test and post-test scores of experimental group and control group, respectively. The results indicate that there were no significant differences (F(1, 194) = 4.526, p > 0.05) between the pre-test scores of the control and experimental groups and no significant differences (F(1, 194) = 0.064, p > 0.05) between the post-test scores of the two groups.

Table 2. ANOVA results for two modes (df = 194).

		Mean	S.D.	Levene	ANOVA
					F	p
Post-test	Cache-cache (146)	7.05	1.60	0.801	0.064	0.739
	Reception (50)	6.96	1.60
Pre-test	Cache-cache (146)	3.09	2.14	0.136	4.526	0.061
	Reception (50)	3.78	2.47

Further analysis was conducted to determine whether or not learning mode effectiveness was dependent on level of learner expertise. Figure 6 shows the boxplot of the pre-test scores of all participants in both groups. We considered the score on the pre-test to represent the extent of the subjects’ exiting knowledge of the target content. To distinguish subjects with extensive knowledge from those with limited knowledge, the learners with a pre-test scores in the top quartile of overall participants (experimental group: N = 22 ; control group: N = 16) were considered as learners with higher prior knowledge and the learners with pre-test scores in the bottom quartile of overall participants were considered as learners with low prior knowledge (experimental group: N = 61; control group: N = 17). A number of previous studies (Koles et al., 2005; Koles et al., 2010; Whitfield, & Xie, 2002) have conducted top and bottom quartiles comparison to analyse the learning effectiveness of their systems on novice and more experienced learners; in other words, those studies examined the expertise reversal effect (Sweller et al., 2003) on their systems by categorizing the top quartile learners as more experienced learners and the bottom quartile learners as novice learners. As mentioned in the discussion of the second research question in Section 4, this study aims to examine the impact of expertise reversal effect on the two different learning modes provided by VSSE; therefore, only learners whose prior knowledge was identified as high or low level are investigated.

Figure 5. Boxplots for the pre-test and the post-test scores of both groups.

As shown in Figure 7, 2 × 2 ANOVA result indicates that there was significant interaction effect (F(1, 112) = 4.137, p < 0.05) between pre-test score levels and learning mode in the post-test. Although neither the main effect of learning mode on post-test score nor the effect of level of learner expertise on post-test score was significant, there was a crossover interaction. Also, there was a high probability (F(1, 112) = 2.961, p < 0.1) that experimental group learners with low prior knowledge had higher post-test scores (Mean = 7.29, S.D. = 1.61) than those with high prior knowledge(Mean = 6.59, S.D. = 1.66). This suggests that the learning achievement of novices who studied in cache-cache comparison mode may have been higher than that of experienced learners.

Figure 6. Distribution and boxplot for the pre-test scores of all the participants.

6.2. Analysis focusing on learning perception

System evaluation and feedback about the learning activity (172 responses collected from participants in both the control and experimental groups) are summarized in Table 3. In the analyses, data of students who did not take the post-test (n = 4) was excluded. Factor analysis of participant responses to the 11 questions on the learning perception questionnaire reveals four distinct scales: mental effort (2 items, α = 0.854), mental load (2 items, α = 0.521), technology acceptance (2 items, α = 0.583) and learning mode satisfaction (5 items, α = 0.858). These four scales are the same as those used in the previous study (Wang et al., 2014), using the same questionnaire.

Table 3. Analytics results for learning perception items in questionnaire.

Item		Mental effort		Mental load		Technology acceptance		Satisfaction
		Purpose	Learning activity	Distraction	Pressure	Ease of use	Usefulness
Cache-cache (123)	Mean	3.98	4.28	3.00	3.41	4.08	4.50	4.51
	S.D.	1.63	1.52	1.33	1.59	1.28	1.26	1.04
Reception (45)	Mean	3.42	3.78	2.60	2.58	4.47	4.76	4.96
	S.D.	1.25	1.18	1.23	1.23	1.36	1.25	1.18
MANOVA (Wilks’ Lambda)	F	2.420		5.254		1.569
	p	0.092		0.006**		0.211
One-way ANOVA	F	4.388	4.073	3.210	9.972	2.289	1.335	5.680
	p	0.038	0.045	0.075	0.002**	0.091	0.250	0.018*

Note: df (1,165) in MANOVA, df (1,166) in ANOVA, ** p < 0.01, *p < 0.05.

In terms of “mental effort,” the average rating for “effort required for understanding the purpose of the learning activity” was less than 4 (the neutral point) for both groups, indicating that most participants in both groups felt that they could easily understand the purpose of the activity. The average ratings of “effort required for learning the target GPs” were 4.28 and 3.78 for the experimental group and the control group, respectively; this suggests that the difficulty of the learning activity was moderate (neither too easy nor too difficult) for the participants in both groups. The MANOVA result indicates no significant difference (F(1, 165) = 2.420, p > 0.05) between the two groups’ responses to mental effort items. These results suggest that there was no difference between these two groups in terms of mental effort required.

In terms of “mental load,” the average rating for degree of distraction and degree of pressure while using VSSE was less than 4 for both the experimental and the control group; this implies that the participants felt little pressure while concentrating on learning with VSSE. However, the MANOVA result for “cognitive load” (F(1, 165) = 5.254, p < 0.001) indicates a significant difference between the two groups. The results of individual univariate analyses support the claim that there was a significant difference between the two groups in the rating of “pressure” (F(1, 166) = 9.972, p < 0.025); this suggests that while using VSSE, the participants learning in cache-cache comparison mode felt more pressure than those learning in reception comparison mode.

In terms of “technology acceptance” measures, the average rating on the “perceived ease of use” item was 4.08 for the experimental group and 4.47 for the control group; most participants reported that VSSE was easy to operate and become familiar with. The average rating of “perceived usefulness” was 4.50 for the experimental group and 4.76 for the control group, which implies that in both groups, most participants thought that VSSE was useful for improving their learning performance. The MANOVA result indicates no significant difference (F(1, 165) = 1.569, p > 0.05) between the responses to “technology acceptance” by the two groups.

As can be seen in Table 3, the average ratings (using the mean rankings for the five related items) for “satisfaction with learning mode” were 4.51 and 4.96 for the experimental group and the control group, respectively; this implies that most participants in both groups were satisfied with the learning mode provided. Moreover, the result of ANOVA (considering the mean rankings for these five items as one dependent variable) indicates a significant difference (F(1, 166) = 5.680, p < 0.05) between the responses to these items by the two groups; this suggests that while using VSSE, the participants who learned in reception comparison mode were more satisfied with that learning mode than those who learned in cache-cache comparison mode.

Further analyses were also conducted to determine whether or not learning perception for different learning modes is correlated with level of learner expertise. As mentioned in Section 6.1, only the learners whose pre-test scores were identified as high or low level were investigated. Three 2 × 2 MANOVAs were conducted on the “mental effort”, “mental load” and “technology acceptance” scales, respectively; as for the “learning mode satisfaction” scale, 2 × 2 ANOVA was conducted with the mean rankings for these five items considered as one dependent variable. Significant interaction effect (F(1, 96) = 4.552, p < 0.05) was found only between the levels of pre-test scores and the learning modes on the responses to “technology acceptance” measures (i.e. “perceived ease of use” and “perceived usefulness”). Further ANOVA results indicate significant interaction effect between the pre-test score level and learning mode in the rating of “perceived ease of use” (as shown in Figure 8, F(1, 96) = 8.812, p < 0.025), but no significant interaction effect in the rating of “perceived usefulness” (F(1, 96) = 2.301, p > 0.05). Individual univariate analyses reveal a significant difference in the rating of “perceived ease of use” between experimental group and control group learners with high prior knowledge (F(1, 96) = 12.645, p < 0.025); in other words, while using VSSE, the experienced learners who learned in reception comparison mode reported greater ease of operation (Mean = 5.07, Mean > 4, i.e. 4 is the neutral point) in that learning mode than did those who learned in cache-cache comparison mode (Mean = 3.56, Mean < 4). Moreover, the results of individual univariate analyses reveal that experimental group learners with low prior knowledge gave significantly higher ratings of “ease of use” (F(1, 96) = 8.275, p < 0.025) than those with high prior knowledge; this suggests that in cache-cache comparison mode in VSSE, learners with low prior knowledge felt that operation in that learning mode was easy (Mean = 4.51, Mean > 4), whereas learners with higher prior knowledge (Mean = 3.56, Mean < 4) felt that operation in that learning mode was difficult (Figure 8).

Figure 7. Interaction effect between learning modes and level of pre-test in post-test (+ p<0.10).

Figure 8. Interaction effect between learning modes and level of pre-test in “easiness” of technique acceptance (*p < 0.01).

7. Discussion and conclusion

To support the effective construction of knowledge frameworks by the e-book learners in this study, a visualization learning support system based on ontology technique was developed. To encourage active engagement, a discovery learning environment called “cache-cache comparison” was designed to encourage learners to detect hidden relationships between relevant KPs through reflection on the attributes of knowledge acquired visually. The seeking of hidden relations is intended to encourage learners to situate new knowledge in their knowledge structures and restructure their existing knowledge; meanwhile the iterative procedure of confirmation and modification within their own topic maps ensures that they check the logical consistency of their ideas and resolve any misunderstandings.

In an experiment conducted in two existing computer science classes at a University in Japan, students in one class (the control group) studied in the meaningful reception learning environment provided by reception comparison mode in VSSE, while students in the other class (the experimental group) studied in the meaningful discovery learning environment provided by cache-cache comparison mode in VSSE. The following discussion of the responses to the research questions presented in Section 4 is based on the analysis of the learner data.

1. Learning performance differences between the experimental group and the control group are summarized below. Analysis of the learning achievement data suggests that despite the slightly higher pre-test scores of the control group, there were no significant differences in learning achievement between the two groups. At the same time, analysis of reported perception of learning suggests that learners who studied in cache-cache comparison mode felt significantly more pressure and less satisfaction than those who studied in reception comparison mode. It should be noted that for both groups the average rating of perceived pressure was less than 4 (neutral) and the average rating of “satisfaction with learning mode” was greater than 4; this suggests that cognitive load was not excessive for either group and that there was high learning mode satisfaction in both groups. These results are as expected: learners who studied in cache-cache comparison mode were required to complete the topic map proactively, which would have led to greater pressure.

2. On the one hand, further analysis of learning achievement by learners with different levels of prior knowledge shows that novices who studied in cache-cache comparison mode had greater learning achievement than experienced learners. However, for study in reception comparison mode, there was no correlation between level of learner expertise and learning achievement. On the other hand, except for the interaction effect on “technology acceptance,” no significant interaction effect was found between pre-test score level and learning mode for other learning perceptions. Among learners with high prior knowledge using VSSE, those who studied in reception comparison mode evaluated learning mode as much easier to operate than those who studied in cache-cache comparison mode. As well, learners with lower prior knowledge felt that cache-cache comparison mode was easier to operate than did learners with higher prior knowledge.

The findings regarding the impact of level of learner expertise on learning effectiveness in each learning mode are in harmony with the “expertise reversal effect” (Sweller et al., 2003), which holds that the success of instructional techniques designed to facilitate schema construction depends heavily on initial level of learner expertise. That is, inexperienced learners evaluated learning mode A higher than learning mode B, whereas experienced learners gave higher evaluations to learning mode B. Learners with low prior knowledge tended to lack knowledge about the relations between KPs; therefore, in cache-cache mode there was less burden on the learner’s working memory than in reception mode, since in cache-cache mode all the relations between KPs are hidden at the first stage and schema construction is only encouraged at the second stage, i.e. the environment offers less information at the outset. In contrast, learners with more prior knowledge already have knowledge of a number of relations between KPs, so when confronted with a specific topic map in reception mode, they are able to recognize it as a familiar schema and treat it as a single unit. However, when they are required to create the relations between KPs in cache-cache mode, some difficulty in system operation is unavoidable, which places additional load on working memory resources.

To sum up, the finding that in cache-cache comparison mode learners with low prior knowledge showed greater increases in performance than learners with high prior knowledge suggests that cache-cache comparison mode is more appropriate than reception comparison mode for learners with low prior knowledge. On the other hand, for learners with high prior knowledge, learning mode made no significant difference to learning achievement. However, learners felt significantly less pressure and more satisfaction in reception comparison mode than in cache-cache comparison mode. In light of the above, for learners with high prior knowledge, reception comparison mode is indicated.

In future work, further experimental studies will be conducted to explore the long-term effects of learning support in the two learning modes of VSSE. In the experiments in this study, participants were required to use VSSE to learn the content of selected pages of an existing course, and only the short-term learning performance of the learners was recorded and analysed. In the next stage of this project, the same system will be used to track long term learning performance, i.e. the learning process throughout a course, so as to determine whether or not differences in learning perception resulting from different learning modes will result in differences in long-term learning achievement.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by JSPS KAKENHI Grant-in-Aid for Scientific Research (S) [grant number 16H06304]; JSPS KAKENHI Grant-in-Aid for Young Scientists (B) [grant number 17K17936].

Notes on contributors

Jingyun Wang

Jingyun Wang is an Assistant Professor at Research Institute for Information Technology, Kyushu University, Japan. She received her PhD in Science from Kochi University of Technology in 2014. Her current research focuses on visualization learning support systems, personalized language support, ontology technique, and educational big data analysis. Her work involves the integration of the traditional education methodologies with ontology technique.

Atsushi Shimada

Atsushi Shimada received the DE degrees from Kyushu University in 2007. He is an associate professor of Department of Advanced Information Technology, Kyushu University. He also works as a JST-PRESTO researcher since 2015. His current research interests focused on image processing, pattern recognition, and learning analytics.

Misato Oi

Misato Oi is an associate professor at Innovation Center for Educational Resource, Kyushu University, Japan. She received her Doctor of Philosophy from Nagoya University in 2010. Her research includes Computer Supported Ubiquitous and Mobile Learning, bilingualism, and non-verbal communication.

Hiroaki Ogata

Hiroaki Ogata is a full professor at Faculty of Academic Center for Computing, Kyoto University, Japan. He was a visiting researcher of Center of Lifelong Learning and Design, the University of Colorado at Boulder, USA from 2001 through 2003. His research includes CSUML, CSCL, CSCW, CALL, and Learning Analytics.

Yoshiyuki Tabata

Yoshiyuki Tabata is a Full Professor in the Research Institute for Information Technology, Kyushu University. He received his BA and MA degrees from Tokyo University of Foreign Studies, Japan, in 1983 and 1986, respectively. He is a Member of JGG, JDV, GDDJ, JAECS and JEI. His current interests are in language-learning environments supported by ICT.