Developing and validating a model for assessing paid mobile learning app success

ABSTRACT

With the proliferation of paid mobile learning applications (m-learning apps), understanding how to assess their success has become an important issue for academics and practitioners. Based on the information systems (IS) success models and the value-based adoption model, this study developed and validated a multidimensional model for assessing paid m-learning app success. The proposed model describes the interrelationships among seven paid m-learning app success variables: system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, intention to reuse, and learning effectiveness. Data collected from 160 paid m-learning app users were tested against the research model using structural equation modeling (SEM). The empirical findings provide evidence that learning effectiveness is affected by user satisfaction and intention to reuse, which, in turn, are determined by system quality, information quality, perceived enjoyment, and perceived fee. The findings of this study provide several important theoretical and practical implications for the development, implementation, and promotion of paid m-learning apps.

KEYWORDS:

• Paid mobile learning apps
• information system success model
• value-based adoption model
• intention to reuse
• learning effectiveness

1. Introduction

Mobile computing technology provides learners with unprecedented opportunities for communicating, interacting, sharing, meaning-making, and content and context generation (Cook & Pachler, 2012). Mobile learning, or m-learning, will play a vital role in the rapidly growing electronic education market (Dimakopoulos & Magoulas, 2009; Wang, Wu, & Wang, 2009). RnRMarketResearch.com (April 2015) predicted that the worldwide market for m-learning applications (apps) will grow from US$7.98 billion in 2015 to US$37.60 billion by 2020 and mushroom to US$325 billion by 2025, with a compound annual growth rate (CAGR) of over 7%. North America, Europe, and the Asia-Pacific region are forecast to have the highest growth rates in this period. Moreover, approximately 47% of organizations globally are already using mobile devices for their online training needs.

M-learning is individualized and facilitated by mobile computing devices (Gikas & Grant, 2013; Herrington & Herrington, 2007; Valk, Rashid, & Elder, 2010). Amobile learner (m-learner) can use wirelessly connected mobile computing devices to acquire knowledge at any time and place. The rapid development and market penetration of mobile computing has made m-learning ubiquitous (Motiwalla, 2007). Common forms of m-learning include e-books, video-based courseware, mobile content authoring, portable learning management systems (LMS), and various m-learning apps specially designed to run on mobile devices (e.g. dictionaries, language learning, image editing, and math games). These apps are often preloaded on handheld devices and are also available to download from device/OS-specific app marketplaces (e.g. Google Play, Apple App Store, and Windows Store). These apps can make learning easier because they allow m-learners to keep up with their courses whenever and wherever is convenient (Quinn, 2011). The use of m-learning apps has been found to improve learning experiences, especially when its usage is coupled with additional learner-centered instruction (Zhu & Kaplan, 2002), or when learning takes place asynchronously at the m-learner’s own pace or on an as-needed basis (Palloff & Pratt, 2001).

The rapid growth potential of m-learning apps means this industry is highly competitive. M-learners already have a enormous choice of providers and learning options to choose from, many of which are free. One outcome of this development is that m-learning apps that require payment to download or involve in-app purchases are much less likely succeed than free ones (Statista, 2015). Therefore, understanding the key factors that affect m-learners’ decisions to use/purchase paid m-learning apps is crucial for m-learning marketers and app developers. Effective and successful m-learning apps need to take advantage of the strengths and overcome the weaknesses of mobile technology and at the same time use the most suitable and effective pedagogies. However, little is known about the success and effectiveness of paid m-learning apps. Therefore, this study aimed to examine paid m-learning app success and from an m-learner perspective, not just from a mobile technology user perspective.

The evaluation of information system (IS) success or effectiveness has been a widely discussed topic in the IS field. And many IS models have been proposed (Chiu, Chao, Kao, Pu, & Huang, 2016; Wang, 2008; Wang & Liao, 2008; Zheng & Liang, 2017). The most prominent IS success model that explains the benefits of information technology(IT) usage by individuals and organizations is the DeLone and McLean (2003) IS success model. However, DeLone & McLean’s IS success model has limited applicability to the paid m-learning app context. Most m-learners use/purchase paid m-learning apps for personal purposes, so the cost of voluntary use/purchase is usually borne by individuals. Therefore, one of the major barriers purchasing a paid m-learning app is monetary cost. For potential adopters of these apps, they are likely to evaluate the m-learning app’s fee/cost/price and its perceived qualities/benefits.

Improving the trade-off between the benefits and sacrifices of a mobile product/service, such as an m-learning app, can help create sustainable competitive advantage through value creation (Eggert & Ulaga, 2002; Kim, Chan, & Gupta, 2007). In the mobile services context, a user’s perceived value has been empirically linked to increased usage/purchase intentions (Chung & Koo, 2015; Kim et al., 2007; Tseng & Lo, 2011; Wang, Lin, Wang, Shih, & Wang, 2018). However, while the IS success model has been widely used, few studies have used it to examine the success or effectiveness of paid m-learning apps from the perspective of user-perceived value. As such, there remains a need to examine the success measures or surrogates of paid m-learning apps from m-learners’ perceived value perspectives.

The current research was conducted to fill this knowledge gap. The authors developed and validated a comprehensive model for assessing paid m-learning app success based on the IS success models (e.g. DeLone & McLean, 2003; Wang, 2008) and the value-based adoption model (Kim et al., 2007). The findings of this study should be useful to researchers who are interested in developing and validating theories related to paid m-learning app success. For practitioners, such as app developers and marketers, the findings can be used to better understand m-learners’ decisions to reuse, which can help improve learning performance and app success.

The rest of this article is organized as follows. In the next section, the relevant literature is reviewed and the research model and hypotheses are developed. Following this, the construct measures and data collection methods are described. After presenting the analytical results, the authors conclude the paper with a discussion of the findings and related theoretical and practical implications.

2. Theoretical foundations

2.1. Mobile learning

Learning through the use of mobile technologies is commonly termed m-learning or mlearning (O’Malley et al., 2005). There are three essential elements of m-learning: mobile learning devices, the communication infrastructure, and the learning activity model (Chang, Sheu, & Chan, 2003). Mobile learning devices are small, light, portable (usually handheld or wearable), equipped with wireless communication capabilities, and come with computational power and context-aware tools; current examples include smartphones, laptops, tablet computers, digital audio players, and e-book readers. The communication infrastructure connects mobile computing devices to relevant learning materials and/or other learners, using access points, base stations, and other relevant technologies. The learning activity model can be in traditional classrooms, outside the classroom, or in informal learning contexts, and it can be used by either a single learner or a group of learners.

There are three ways in which learning can be considered “mobile.” First is space – learning can happen anywhere, including at the workplace, at home, or at places of leisure. Second is time – learning can happen at different times of the day, during work days, or on weekends. Third is different areas of life – learning can relate to work demands, self-improvement, or leisure (Vavoula & Sharples, 2002). Simply speaking, m-learning provides learners with unique opportunities only available through mobile computing devices (Keegan, 2002; O’Malley et al., 2005). It can be used to support traditional, electronic, and distance learning (Korucu & Alkan, 2011), and it can further improve learning performance (Tan, Liu, & Chang, 2007).

Crowdsourcing has further influenced m-learning (Corneli & Mikroyannidis, 2012). Crowdsourcing refers to outsourcing a task to a large, undefined “crowd,” rather than to a designated contractor (e.g. organization, informal or formal team, or individual) through an open call (Howe, 2006, 2008; Jeppesen & Lakhani, 2010). Crowdsourcing is done through online networked communities that facilitate collaboration, discussion, and the exchange of ideas, problem solving, and sharing of entertainment (Brabham, 2008). Primary crowdsourcing strategies include crowd creation, crowdvoting, crowdfunding, and crowd wisdom/collective intelligence (Brabham, 2008; Howe, 2008). Crowdsourcing has had a great impact on innovation in the educational technology application industry, which has provided independent learners with many benefits, including quality information, solution assistance, and guided discussion (Al-Jumeily, Hussain, Alghamdi, Dobbins, & Lunn, 2015). Popular crowdsourcing apps and communities that are used as learning support tools include Stack Overflow (crowd wisdom/crowdvoting), Wikipedia (crowd wisdom), Coursera (crowd creation), and so on (Al-Jumeily et al., 2015; Buecheler et al., 2010). As a result, m-learners have a wide variety of free and paid-for content and educational tools from which to choose from on their learning journey. However, the reasons why m-learners are willing pay for some m-learning apps needs when there are free ones available has yet to be investigated. Thus, there are academic and practical benefits for developing and validating a model that assesses paid m-learning app success.

2.2. IS success models

DeLone and McLean (1992) comprehensively reviewed articles published from 1981 to 1987 to synthesize a six-factor taxonomy of IS success. They identified six variable categories in the inter-relational model: system quality, information quality, use, user satisfaction, individual impact, and organizational impact. DeLone and McLean (1992) developed this original taxonomy using established communication theories adapted to IS contexts (Mason, 1978; Petter & McLean, 2009; Shannon & Weaver, 1949), and they advocated the additional development and validation of their taxonomy. The multidimensional and interdependent nature of IS success requires careful attention to the definition and measurement of each variable in the inter-relational model. Selection of IS success measures should be contingent on the context of the study to ensure a complete understanding of IS success (DeLone & McLean, 1992). In the years that followed, several researchers altered or extended the model of IS success in specific applications, such as e-commerce, knowledge management systems (KMS), enterprise resource planning (ERP), and e-government systems (e.g. Bernroider, 2008; DeLone & McLean, 2004; Jennex & Olfman, 2006; Jennex, Olfman, Panthawi, & Park, 1998; Kulkarni, Ravindran, & Freeze, 2007; Petter & McLean, 2009; Rai, Lang, & Welker, 2002; Saarinen, 1996; Seddon & Kiew, 1996; Wang, 2008; Wang & Liao, 2008).

Later, Seddon (1997) argued that the DeLone and McLean’s (1992) original IS success model was too encompassing and introduced the potential for confusion because it mixed process and variance (causal) explanations for IS success. Adapting the original model, Seddon (1997) replaced use with perceived usefulness and developed an alternative model that focused on the variance (causal) aspects of the interrelationships among the taxonomic categories. Seddon’s IS success model, as a variance (causal)model, considered three classes of variables: measures of information and system quality, general measures of net benefits of IS use (i.e. perceived usefulness and user satisfaction), and behavior with respect to IS use (i.e. net benefits to individuals, organizations, and society). In sum, the DeLone and McLean’s (1992) model assumes volitional usage; however, Seddon’s model incorporates both volitional and nonvolitional IS usage contexts. Since both DeLone and McLean’s (1992) and Seddon’s (1997) models have several commonalities and important distinctions, Rai et al. (2002) specified and tested the two success models in a quasi-voluntary IS use context and found evidence that both models have explanatory power; This finding suggests that both models have merit for assessing and explaining IS success.

Reflecting on this outcome, DeLone and McLean (2003) revised their original IS success model to address its restrictions. They proposed an updated model that added service quality as an additional aspect of IS success and grouped all the impact measures (i.e. individual impact and organizational impact) into a single net benefits variable. In the updated IS success model, quality was considered to be a three-dimensional construct (i.e. information quality, system quality, and service quality), each measured and controlled separately. DeLone and McLean (2003) also recommended assigning different weights to the three quality dimensions depending on the context. The quality dimensions were predicted to singularly or jointly affect intention to use, use, and user satisfaction, resulting in certain net benefits. DeLone and McLean (2003) suggested that, given the variability of IS contexts, it may sometimes be suitable to measure intention to use (an attitude) rather than use (a behavior). They went on to state that if intention to use was an IS success measure, then increased user satisfaction would lead to higher intention to use, which would subsequently affect usage behavior. This reasoning resulted in the addition of intention to use in the updated IS success model. DeLone and McLean (2003) also addressed the criticism that IS can affect levels other than individuals and organizations. Accordingly, their updated IS success model accounted for net benefits occurring at any level of analysis (e.g. workgroups, industries, customers, and societies); the choice of which level to use was up to the researchers using the model. In the m-learning context, m-learners use m-learning apps to acquire knowledge, seek information, discuss with others, or just have fun. This suggests that m-learning apps are a kind of IS and thus the updated DeLone and McLean IS success model should be applicable to this context.

Further developments followed when Wang (2008) proposed a multidimensional model for assessing e-commerce systems success by adding perceived value, replacing system use/intention to use with intention to reuse, and re-specifying a new construct of other net benefits in the DeLone and McLean (2003) model. Wang’s (2008) model describes the interrelationships among seven IS success measures: information quality, system quality, service quality, perceived value, user satisfaction, intention to reuse, and other net benefits (see Figure 1). Furthermore, the nomological structure of Wang’s (2008) model is consistent with that of IS adoption models and consumer behavior models. Given that paid m-learning apps are kind of e-commerce system for m-learning app developers and users/purchasers, Wang’s (2008) e-commerce system success model also seems ideally suited to explore and explain the phenomenon of paid m-learning apps success. One of the most important IS success measures proposed by Wang (2008) is perceived value; however, he did not elaborate all of the possible constituents of perceived value, especially the sacrifice side of perceived value.

Figure 1. Wang’s (2008) e-commerce systems success model.

2.3. Value-based adoption model

The value-based adoption model (VAM), developed by Kim et al. (2007), enables an overall estimation of the perceived value of a choice (e.g. whether or not to adopt a paid app) from among an individual’s various decision-making strategies. When consumers encounter an object, they usually have a choice of behaviors to follow (e.g. adopt or purchase a paid m-learning app), which is based on both visible and invisible costs and benefits. From the behavioral decision theory perspective (Johnson & Payne, 1985), the cost–benefit paradigm is treated as a cognitive trade-off between give and get components of a choice. Similarly, in the VAM, a consumer’s overall assessment of the value of a choice (e.g. the utility of using a paid mobile app) is based on perceptions of what is given and what is received (Zeithaml, 1988). Consumers estimate the overall value of the choice by considering all relevant perceived benefits and perceived sacrifices. Based on the VAM, mobile product/service (e.g. mobile Internet) adoption is determined by perceptions of the value of using the mobile product/service, and these in turn are determined by the mobile users’ perceptions of usefulness, enjoyment, technicality, and perceived fee (Kim et al., 2007). Accordingly, this study assumed that m-learners’ value perceptions can be adapted to a paid app success measurement in the m-learning context, especially measures of perceived enjoyment (perceived benefit) and perceived fee (perceived sacrifices).

3. Research model and hypotheses

The evolution of learning systems, learners, and learner requirements means that evaluating the effectiveness or success of paid m-learning apps is an important consideration for researchers and practitioners (Keen, 1980; Tate, Sedera, McLean, & Burton-Jones, 2011). In general, using a learning system is a long-term investment, the performance of which is subjected to a range of contextual factors. As a long-term investment, mobile technology is expected to yield a continuing flow of benefits into the future. The potential value of mobile technology has motivated many mobile app providers to implement m-learning or develop paid m-learning apps. While the earlier IS success models (e.g. DeLone & McLean, 1992, 2003) have been applied in m-learning contexts (e.g. Kim & Ong, 2005; Yi, Liao, Huang, & Hwang, 2010), prior studies have not investigated the relationships between the success measures/surrogates in the context of paid m-learning apps; therefore, it is imperative to develop and validate a comprehensive model to measure paid m-learning app success.

3.1. The stakeholders, level of analysis, and scope of net benefitsin the paid m-learning app success model

Researchers have developed several methods for assessing IS success using intangible measures. The original and updated DeLone and McLean IS success models and related IS impact models (which drew extensively on the DeLone and McLean IS success models) have been the most influential (Tate et al., 2011). These models use stakeholder assessments of the target system. As DeLone and McLean (2003) indicated, “the challenge for the researcher is to define clearly and carefully the stakeholders and context in which net benefit are to be measured” (p. 23). In general, different stakeholders with different target systems have different perspectives on net benefits (DeLone & McLean, 2003; Seddon, Staples, Patnayakuni, & Bowtell, 1999). Based on previous IS success models (i.e. DeLone & McLean, 1992, 2003; Wang, 2008), the current study treated both paid m-learning app providers and m-learners as stakeholders. However, the success/effectiveness/net benefits variables were measured merely from the m-learners’ perspective, and employed existing IS success measures, including system quality, information quality, user satisfaction, and intention to reuse (a substitute for intention to use/use). These measures were used to develop and validate a success model specific to the paid m-learning app context. Because m-learners seldom contact service personnel for help in the usage of m-learning apps and previous research also suggested that service quality is relatively less important in assessing knowledge-oriented system success (Wu & Wang, 2006), the measure of service quality was not incorporated into the paid m-learning apps success model.

Many researchers have employed objective financial indicators, such as return on investment and return on assets (e.g. Brynjolfsson & Hitt, 1996), to assess the impact of an information system. However, some researchers (e.g. Davenport, 2000; Kaplan & Norton, 2000) have pointed out that contemporary information systems also provide substantial non-financial benefits. As noted earlier, both Seddon (1997) and DeLone and McLean (2003) compromised on the use of net benefits as an IS success measure. For example, Seddon (1997) suggested that user satisfaction and perceived usefulness (a substitute for use) can also be general perception measures of net benefits. Similarly, in the updated DeLone and McLean IS success model, both user satisfaction and intention to use/use were measures of net benefits. Even so, scholars have suggested that additional research is needed to find a more comprehensive or reliable measure of net benefits than perceived usefulness, user satisfaction, and intention to use/use (Seddon, 1997; Wang, 2008).

Value perception is a more reliable measure of m-learners’ overall assessment of a mobile product/service (e.g. a paid m-learning app) than service quality (Bolton & Drew, 1991). According to the VAM (Kim et al., 2007), perceived value is frequently conceptualized as a mobile user’s overall assessment of the ratio of perceived benefits to perceived sacrifices (Monroe, 1990; Zeithaml, 1988). That is, perceived value involves a tradeoff between give and get components. Perceived usefulness, however, only taps the get component. Similar to user satisfaction, perceived value taps a wider range of the give and get components than perceived usefulness (Wang, 2008). In Wang’s (2008) success model, perceived usefulness was replaced with perceived value. It is a richer measure of net benefits in e-commerce contexts. Based on Kim et al.’s (2007) VAM, the current research treated perceived enjoyment as a get component and perceived fee as a give component in assessing paid m-learning app success. Both perceived enjoyment (i.e. the benefits an m-learner derives from an m-learning app’s offering) and perceived fee (i.e. the monetary costs m-learners must bear when acquiring the offering) are functions of perceived value (Kim et al., 2007).

3.3. Intention to reuse

Compared to IS use, intention to use may be a more appropriate measure in some contexts (DeLone & McLean, 2003). Previous researchers have contended that initial system usage and intentions to use in the future differ (e.g. Karahanna, Straub, & Chervany, 1999). User intention to reuse an IS in the future is a more appropriate measure of system success/net benefits than user initial or current system usage (Wang, 2008). Wang’s (2008) e-commerce system success model therefore adopted intention to reuse as a system success surrogate variable to simplify the closed-loop relationships between user satisfaction, intention to use, and use, as represented in the updated DeLone and McLean IS success model. Following Wang’s suggestion, the current research also employed intention to reuse as a measure of paid m-learning app success. In the context of m-learning apps, intention to reuse is the positive attitude of an m-learner towards a paid m-learning app, which results in repeat use behavior.

3.4. Learning effectiveness

Technology-assisted learning has had profound and lasting impacts on education, and researchers and educators have investigated related topics (Hui, Hu, Clark, Tam, & Milton, 2008). In the context of paid m-learning apps, understanding m-learners’ learning performance remains a critical issue for making successful paid m-learning apps. M-learners’ learning performance seems to be the ultimate success measure of using paid m-learning apps from the m-learners’ perspectives. Therefore, learning effectiveness was added to the proposed paid m-learning app success model. Previous research has shown that using test scores to measuring learning performance may not be appropriate for assessing learning quality or learning experience and their affects on learner retention (Neumann & Finaly-Neumann, 1989; Norman & Spohrer, 1996). Accordingly, the current study examined m-learners’ perceived learning effectiveness, or the extent to which m-learners believe they have acquired specific knowledge/skills from using a specific m-learning app (Hui et al., 2008). Learning effectiveness can also represent one of the other net benefits in the Wang’s (2008) systems success model.

3.5. A research model of paid M-learning app success

Based on the preceding discussion, the current research developed a comprehensive multidimensional model of paid m-learning app success (see Figure 2), which proposes that system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, intention to reuse, and learning effectiveness are success measures relevant to paid m-learning apps.

Figure 2. The research model of m-learning app success.

Note: t-values for standardized path coefficients are described in parentheses.

The hypothesized relationships between system quality, information quality, user satisfaction, and intention to reuse are grounded on the theoretical and empirical underpinnings of the updated DeLone and McLean IS success model and Wang’s (2008) model. Following the suggestions of previous researchers to create model parsimony (e.g. DeLone & McLean, 2003; Wang, 2008), the current study excluded perceived value from the paid m-learning app success model because its constituent perceived benefits components (i.e. information quality, system quality and perceived enjoyment) and perceived sacrifice components (i.e. perceived fee) were all added to the proposed success model. Researchers have also suggested that system quality and information quality are antecedents of user satisfaction and/or intention to use/use/intention to reuse (e.g. Berger, Geimer, & Hess, 2017; Bolton & Drew, 1991; Cronin, Brady, & Hult, 2000; DeLone & McLean, 2003; Durvasula, Lysonski, Mehta, & Tang, 2004; Forsgren, Durcikova, Clay, & Wang, 2016; Hellier, Geursen, Carr, & Rickard, 2003; Laumer, Maier, & Weitzel, 2017; Oghuma, Libaque-Saenz, Wong, & Chang, 2016; Patterson & Spreng, 1997; Wang, 2008). Combing these concepts, the following hypotheses are proposed:

H1. System quality positively affects user satisfaction in the paid m-learning app context.

H2. Information quality positively affects user satisfaction in the paid m-learning app context.

H3. System quality positively affects intention to reuse in the paid m-learning app context.

H4. Information quality positively affects intention to reuse in the paid m-learning apps context.

In prior studies, hedonic motivation and price value were antecedents of behavioral intention to use (Van der Heijden, 2004; Venkatesh, Thong, & Xu, 2012). From a hedonic motivation perspective (conceptualized as perceived enjoyment), behavior is evoked from feelings of pleasure, joy, and fun. In the mobile service/commerce context, perceived enjoyment positively influences attitude (i.e. user satisfaction) and behavioral intentions – intention to reuse (Cheong & Park, 2005; Chung & Koo, 2015; Hong, Cao, & Wang, 2017; Hsiao, Chang, & Tang, 2016; Kim et al., 2007; Kuo, Wu, & Deng, 2009; Lu & Yu-Jen Su, 2009; Mallat, Rossi, Tuunainen, & Oorni, 2009; Tseng & Lo, 2011; Turel, Serenko, & Bontis, 2010; Wang & Li, 2012). Price value is defined as the cognitive tradeoff between the perceived benefits of the applications and the monetary cost/price (i.e. perceived fee) of using them (Dodds, Monroe, & Grewal, 1991; Venkatesh et al., 2012). Higher monetary cost/price perceptions are related to lower value perceptions (Chang & Wildt, 1994). Hence, perceived fee has a negative impact on attitude and behavioral intentions in the judgment stage, according to the value perspective (Dodds et al., 1991; Gupta & Kim, 2010; Kim & Gupta, 2009; Kim et al., 2007; Wang & Wang, 2010; Wang, Yeh, & Liao, 2013). Therefore, this study tested the following hypotheses:

H5. Perceived enjoyment positively affects user satisfaction in the paid m-learning app context.

H6. Perceived fee negatively affects user satisfaction in the paid m-learning app context.

H7. Perceived Enjoyment positively affects intention to reuse in the paid m-learning app context.

H8. Perceived fee negatively affects intention to reuse in the paid m-learning app context.

As DeLone and McLean (2003) noted, user satisfaction, intention to use, and IS use are closely interrelated. More specifically, positive experience with use leads to greater user satisfaction, which leads to increased intention to use and use (DeLone & McLean, 2003). To simplify the closed-loop relationships between use, satisfaction, and intention to use, Wang (2008) suggested that increased user satisfaction leads to increased intention to reuse in the post-use situation. Previous studies have also suggested that user satisfaction is a dependable predictor of intention to reuse (Berger et al., 2017; Cronin et al., 2000; Eggert & Ulaga, 2002; Lam, Shankar, Erramilli, & Murthy, 2004; Lin & Wang, 2006; Malhotra, Sahadev, & Purani, 2017; Oghuma et al., 2016; Patterson & Spreng, 1997; Wang, 2008). Therefore, the following hypothesis was also tested:

H9. User satisfaction positively affects intention to reuse in the paid m-learning app context.

From the user’s perspective, the ultimate goal of m-learning is knowledge acquisition. Keller (1983) suggested that learning satisfaction relates directly to perceptions and feelings about learning effectiveness or outcomes. Thus, degrees of learner satisfaction have been widely used to evaluate learning effectiveness in the technology-assisted learning research area (Hwang, Wu, Zhuang, & Hwang, 2013; Smith, Ling, & Hill, 2006; Zhang, Zhou, Briggs, & Nunamaker, 2006). Previous empirical research has also found a positive correlation between learner satisfaction and learning effectiveness in the context of blog-based learning systems (Wang, Li, Li, & Wang, 2014). Additionally, a significant relationship between learners’ behavioral intention and learning effectiveness exists (Liaw, 2008). A higher degree of learner engagement can positively improve learning effectiveness (Zhang et al., 2006). Wang et al. (2014) also found that system use has a positive influence on learning performance. Thus, the current research underscored the importance of intention to reuse, which was also expected to enhance the effectiveness of m-learning. The following hypotheses were also tested:

H10. User satisfaction positively affects learning effectiveness in the paid m-learning app context.

H11. Intention to reuse positively affects learning effectiveness in the paid m-learning app context.

4. Research methods

4.1. Measures of the constructs

To ensure content validity, measurement items must characterize the concepts about which valid generalizations are to be made (Bohmstedt, 1970). Therefore, most measurement items in this study were adapted from previous research and then modified to suit the paid m-learning app context.

System quality is usually a multidimensional variable. However, the intent of this study was not to explore the complex multidimensional nature of the system quality measure; therefore, ease of use/user friendliness was used to measure system quality. Three items were selected from prior system quality/ease of use/technicality concepts (Kim et al., 2007), and they were adapted to measure system quality in the context of paid m-learning apps. Following the same method, information quality was measured with three items adapted from previous research (e.g. Wang, 2008). Three items for the perceived enjoyment construct and three items for the perceived fee construct were adapted from Kim et al. (2007). The indicators of both user satisfaction and intention to reuse were adapted from previous applications of e-commerce system success (i.e. Wang, 2008). Finally, three items were adapted from previous e-learning effectiveness measures (i.e. Liaw, 2008) to measure m-learning effectiveness. Responses for all items were reported on 5-point Likert scales (1 = strongly disagree; 5 = strongly agree). After the pre-testing of the research instrument, these items were modified to fit the paid m-learning app context. The final survey items are listed in Appendix A.

4.2. Data collection

To test the hypotheses, data were collected with an online survey. The questionnaire was uploaded to a survey portal (http://survey.youthwant.com.tw/) in Taiwan and made available for Internet users. Respondents were first asked whether they had ever used paid m-learning apps; if they replied in the affirmative, they were asked to participate in the survey. The questionnaire requested that respondents recall the last time they had used a paid m-learning app and then write down the name of this app. They were instructed to answer all questions according to their experience using this app. For each question, respondents were asked to circle the response that best described their level of agreement. The survey yielded 160 usable responses (86.02% usable response rate), and comprised 85 males and 75 females. About 87.5% were under 35 years of age, while nearly 96.25% had at least a college degree, indicating that the respondents were mainly young and educated. The largest number of respondents had experience with the language learning apps (88.125%). Table 1 shows the summarized demographics of the respondents.

Table 1. Respondent characteristics (n = 160).

Characteristic	Items	Frequency	Percentage
Gender	Male	85	53.125
	Female	75	46.875
Age	20 orbelow	21	13.125
	21–35	119	74.375
	36–55	18	11.250
	56 or above	2	1.250
Educational background	High school and under	3	1.875
	Junior college	3	1.875
	Bachelor’s degree	112	70.000
	Master’s degree and over	42	26.250
Occupation	Student	65	40.625
	Services	35	21.875
	Education and research	8	5.000
	Government agencies	13	8.125
	Others	39	24.375
Weekly usage time	Less than 1 h	72	45.000
	1–3 h	76	47.500
	3–5 h	8	5.000
	Over 5 h	4	2.500
Purchasing costs (NT)	100 or below	77	48.125
	101–300	67	41.875
	301–500	6	3.750
	501 or above	10	6.250
Learning topic types	Language learning	141	88.125
	Learning about computers	7	4.375
	Math learning	7	4.375
	Others	5	3.125

5. Results

5.1. Measurement model

Using Amos 19.0, a confirmatory factor analysis (CFA) was carried out to test the measurement model. The overall model fit was assessed using six common goodness-of-fit indices: the ratio of χ² to degrees of freedom (df), adjusted goodness of fit index (AGFI), non-normalized fit index (NNFI), comparative fit index (CFI), incremental fit index (IFI) and root mean square error of approximation (RMSEA).The results of these indices, along with their recommended values for the common model fit, are shown in Table 2. All the model-fit indices exceeded their respective common acceptance levels suggested by previous research, thus demonstrating that the measurement model exhibited a good fit with the data collected (χ² = 205.359 with df = 168, AGFI = 0.855, NNFI = 0.971, CFI = 0.977, IFI = 0.977, RMSEA = 0.037). Thus, further analyses could proceed to evaluate the psychometric properties of the measurement model in terms of reliability, convergent validity, and discriminant validity.

Reliability and convergent validity were evaluated using composite reliability (CR) and average variance extracted (AVE). As indicated in Table 3, the CR values of all constructs exceed 0.80, which was significantly above the 0.60 recommended level (Henseler, Ringle, & Sinkovics, 2009), justifying the reliability of the measurements for model testing. Additionally, a convergent validity test was performed using factor loadings from CFA (see Table 4). The values of all factor loadings of the items exceed 0.70, which were greater than the recommended 0.50 level (Hair, Anderson, Tatham, & Black, 1992). Accordingly, an examination of the CR and AVE suggested the adequacy of the reliability and convergent validity of the measurement model.

Table 3. Reliability, average variance extracted, and discriminant validity.

Construct	CR	SQ	IQ	PE	PF	US	IR	LE
SQ	0.858	0.818
IQ	0.875	0.586	0.837
PE	0.830	0.539	0.570	0.787
PF	0.871	−0.562	−0.564	−0.573	0.832
US	0.868	0.629	0.652	0.584	−0.644	0.829
IR	0.909	0.624	0.647	0.661	−0.697	0.677	0.877
LE	0.870	0.536	0.540	0.546	−0.515	0.666	0.664	0.832

Notes: 1. CR: composite reliability. 2. SQ: system quality; IQ: information quality; PE: perceived enjoyment; PF: perceived fee; US: user satisfaction; IR: intention to reuse; LE: learning effectiveness. 2. Diagonal elements are the square roots of the average variance extracted (AVE) values; off-diagonal elements are correlations among constructs.

Table 4. Factor loadings, t-values and error terms.

Item	Factor loading	t value	Error terms
SQ1	0.756	7.859	0.243
SQ2	0.863	5.747	0.115
SQ3	0.831	6.920	0.138
IQ1	0.814	7.417	0.117
IQ2	0.867	6.382	0.109
IQ3	0.829	6.903	0.107
PE1	0.773	7.675	0.253
PE2	0.813	6.727	0.270
PE3	0.776	7.597	0.322
PF1	0.801	7.776	0.236
PF2	0.852	6.730	0.173
PF3	0.843	6.546	0.195
US1	0.829	7.146	0.163
US2	0.821	7.518	0.157
US3	0.836	7.395	0.196
IR1	0.872	7.243	0.170
IR2	0.885	6.967	0.168
IR3	0.874	6.858	0.154
LE1	0.832	6.923	0.221
LE2	0.795	7.566	0.256
LE3	0.866	6.114	0.135

Notes: SQ: system quality; IQ: information quality; PE: perceived enjoyment; PF: perceived fee; US: user satisfaction; IR: intention to reuse; LE: learning effectiveness.

Moreover, comparisons of the shared variances between factors with the AVE values of the individual factors (Fornell & Larcker, 1981) were done to examine discriminant validity; the results are in Table 3. As shown, the shared variances between factors were lower than the AVE values of the individual factors, which confirmed discriminant validity. To conclude, the measurement model demonstrated adequate reliability as well as convergent and discriminant validities.

5.2. Structural model

A similar set of model-fit indices was used to examine the structural model (see Table 2). The results show that all six fit indices for the structural model (χ² = 210.821 with df = 172, AGFI = 0.854, NNFI = 0.970, CFI = 0.976, IFI = 0.976, RMSEA = 0.038) clearly exceeded the recommended values suggested for a good model fit, implying the adequacy of the structural model for further statistical analysis, including causal link tests.

Table 2. Fit indices for measurement and structural models.

Fit indices	Recommended value	Measurement model	Structural model
χ^2/df	≤3.00	1.222	1.226
AGFI	≥0.80	0.855	0.854
NNFI	≥0.90	0.971	0.970
CFI	≥0.90	0.977	0.976
IFI	≥0.90	0.977	0.976
RMSEA	≤0.08	0.037	0.038

Notes: AGFI: adjusted goodness-of-fit index; NNFI: non-normalized fit index; CFI: comparative fit index; IFI: incremental fit index; RMSEA: root mean square error of approximation.

Properties of the causal paths, including standardized path coefficients, t-values, and variance explained for each equation in the hypothesized model, are presented in Figure 3. As expected, the results support H1, H2, H3, H4, H5 and H7 (γ = 0.230, p < 0.001; γ = 0.283, p < 0.001; γ = 0.141, p < 0.05; γ = 0.172, p < 0.01; γ = 0.147, p < 0.05; γ = 0.232, p < 0.001, respectively). System quality, information quality, and perceived enjoyment all had significant positive effects on both user satisfaction and intention to reuse. Results also support H6 and H8 (γ = -0.267, p < 0.001; γ = -0.269, p < 0.001, respectively). Perceived fee exhibited significant negative effects in influencing both user satisfaction and intention to reuse. Also, H9 and H10 are supported (β = 0.171, p < 0.05; β = 0.399, p < 0.001, respectively). The effects of user satisfaction on both intention to reuse and learning effectiveness were significant. Lastly, intention to reuse appeared to play a significant role in determining learning effectiveness, which supports H11 (β = 0.391, p < 0.001).

Figure 3. Hypothesis test results.

Overall, 52.2% of the variance in learning effectiveness was accounted for by the research model, with both user satisfaction and intention to reuse exerting strong direct effects on learning effectiveness. In addition, 65.2% of the variance in intention to reuse was explained by system quality, information quality, perceived enjoyment, perceived fee, and user satisfaction, while 58.3% of the variance in user satisfaction was explained by system quality, information quality, perceived enjoyment, and perceived fee. The direct and total effects of intention to reuse on learning effectiveness were 0.391. It is noteworthy that the direct and total effects of user satisfaction on learning effectiveness were 0.399 and 0.466, respectively. User satisfaction exhibited stronger direct and total effects on learning effectiveness than intention to reuse. Among the four perceived value constituents, perceived fee had the strongest total effect on learning effectiveness. The direct, indirect, and total effects of system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, and intention to reuse on learning effectiveness are summarized in Table 5.

Table 5. The direct, indirect, and total effect of variables depicted.

	Direct effect			Indirect effect			Total effect
	US	IR	LE	US	IR	LE	US	IR	LE
SQ	0.230	0.141			0.039	0.162	0.230	0.180	0.162
IQ	0.283	0.172			0.048	0.199	0.283	0.221	0.199
PE	0.147	0.232			0.025	0.159	0.147	0.257	0.159
PF	−0.267	−0.269			−0.046	−0.229	−0.267	−0.315	−0.229
US		0.171	0.399			0.067		0.171	0.466
IR			0.391						0.391

Notes: SQ: system quality; IQ: information quality; PE: perceived enjoyment; PF: perceived fee; US: user satisfaction; IR: intention to reuse; LE: learning effectiveness.

6. Discussion

Despite the innovative developments in the paid m-learning app market, few studies have investigated the relevant success measures of these apps, including m-learners’ reuse behaviors and learning performance. This study helps fill this knowledge gap by developing and validating a theoretical model of paid m-learning app success that was based on the existing IS success models and VAM. This model captured the multidimensional and interdependent nature of factors determining paid m-learning app success. The empirical results indicate that system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, intention to reuse, and learning effectiveness are valid measures of paid m-learning app success. All of the hypothesized relationships between the seven success variables are supported by the analytical findings.

From these results, several important implications can be drawn. According to the proposed model, learning effectiveness is a more effective measure of paid m-learning app success than the other six success variables. Learning effectiveness should develop if the formation of system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, and intention to reuse are suitably managed. Accordingly, practitioners need to focus on the development of cognitive, affective, and behavioral processes. In order to increase m-learners’ learning behaviors and performance, paid m-learning app developers should develop apps that have high system quality and information quality and provide an enjoyable experience at a reasonable price. This, in turn, influences m-learner satisfaction evaluation and reuse behavior and learning performance. User satisfaction partially mediates the effects of system quality, information quality, perceived enjoyment, and perceived fee on intention to reuse and learning effectiveness. User satisfaction continues to be an important determinant of intention to reuse and learning effectiveness. The findings suggest that user satisfaction has the strongest direct and total effect on learning effectiveness (see Table 5), indicating the importance of learner satisfaction in promoting m-learners’ continuous learning behaviors and performance. This research also confirms Wang’s (2008) systems success model, which suggests user satisfaction affects other net benefits (e.g. learning effectiveness) directly or indirectly through the mediation of intention to reuse. This result also implies that m-learners’ satisfaction and system reuse intentions are necessary for promoting learning effectiveness.

Previous marketing literature has proposed that both perceived quality/benefit and perceived sacrifice are antecedents of consumer value perception (Cronin et al., 2000; Dodds et al., 1991; Teas & Agarwal, 2000; Zeithaml, 1988). This study adopted Wang’s (2008) concept of perceived value constituents and incorporated both perceived quality/benefit measures (i.e. system quality, information quality and perceived enjoyment) and perceived sacrifice measures (i.e. perceived fee). Similar to previous IS success models (e.g. DeLone & McLean, 2003; Wang, 2008; Wang et al., 2014, 2017; Wu & Wang, 2006), both system quality and information quality were also confirmed as valid IS success measures in the context of paid m-learning apps. On the other hand, and departing from prior IS success models, perceived enjoyment was also successfully added into the paid m-learning app success model and was found to be a valid IS success measure. Prior research has suggested that the inclusion of perceived price (i.e. perceived fee) in the IS success model is also a crucial issue in need of theoretical reasoning and empirical evidence (Wang, 2008). The empirical results show that the newly added perceived fee influences learning effectiveness through the mediation of user satisfaction and intention to reuse. Consequently, the findings support the nomological validity of the paid m-learning app success model and are consistent with the linkages of quality/sacrifice → satisfaction → reuse/repurchase/loyalty → other net benefits, as suggested in previous IS and marketing studies (e.g. Cronin et al., 2000; Durvasula et al., 2004; Hellier et al., 2003; Parasuraman & Grewal, 2000; Patterson & Spreng, 1997; Wang, 2008; Zeithaml, 1988).

Among the four success variables concerning perceived quality/benefit and perceived sacrifice, the total effect of information quality on user satisfaction was greater than system quality, perceived enjoyment, and perceived fee. This finding is consistent with Chae, Kim, Kim, and Ryu’s (2002) argument that information quality is an important determinant of user satisfaction towards mobile services. This means that m-learners show more concern about an m-learning app’s content than other considerations (e.g. ease of use or enjoyment). Learning content is a crucial factor of m-learning because it is the basis of value (Alla & Faryadi, 2013). The learning content on a mobile device should not distract m-learning apps users; it should increase interest in m-learning day by day. Therefore, paid m-learning app developers should pay more attention to this issue in order to satisfy m-learners’ actual needs.

Furthermore, the total effect of perceived fee on intention to reuse was larger than that of system quality, information quality, and perceived enjoyment. In other words, in the context of paid m-learning apps, the findings suggest that monetary costs have a dominant influence on m-learners’ reuse intentions compared to other qualities/benefits. This result also reflects the nature of m-learning crowdsourcing where mobile apps may not be free. Therefore, paid m-learning app developers should enhance their apps’ user-perceived value from a cost–benefit perspective. More specifically, to increase m-learners’ value perception, paid m-learning app developers need to improve app qualities/benefits and ascertain an acceptable price range for m-learners.

The empirical results highlight the importance of a multidimensional approach in assessing paid m-learning app success. Researchers can gain from this model by using it as the foundation to develop comprehensive paid m-learning app success measures, explore interrelationships between the proposed success variables, and compare different paid m-learning app success models. The empirical findings also encourage developers to include measures of system quality, information quality, perceived enjoyment, perceived fee, user satisfaction, intention to reuse, and learning effectiveness in evaluations of their product.

7. Limitations and future research

A rigorous procedure was implemented to develop and validate the proposed model; nevertheless, this empirical study has several limitations that can be addressed in future research. First, this study examined a limited number of paid m-learning app categories(mainly language learning) and a specific m-learner group (Taiwanese learners). Thus, future research can examine other m-learning app categories or m-learner groups to establish the robustness of the current results. Second, the use of self-reported data to investigate the research elements introduces the risk of common method bias. Future research should include both objective and subjective measurements and consider their correlations, or more aptly the lack thereof. Finally, the authors used a cross-sectional research approach; the possible feedback links from learning effectiveness to user satisfaction and intention to reuse were excluded from this study. Therefore, a longitudinal research design could deepen understanding of the causal relationships and correlations between the variables of paid m-learning app success.

8. Conclusions

With the proliferation of paid m-learning apps, developing a better understanding of how to measure these apps’ success is an important issue for both researchers and practitioners. Based on previous IS success models and VAM, this study developed and validated a comprehensive, multidimensional model of paid m-learning apps success. Seven success measures were used: information quality, system quality, perceived enjoyment, perceived fee, user satisfaction, intention to reuse, and learning effectiveness. The empirically validated model suggests that information quality, system quality, perceived enjoyment, perceived fee influence m-learners’ learning effectiveness through the mediation of user satisfaction, and intention to reuse paid m-learning apps.

The contributions of this study to m-learning success research are threefold. First, this study successfully developed and validated a paid m-learning app success model that combined perceived enjoyment, perceived fee, and learning effectiveness with traditional IS success models (e.g. DeLone & McLean, 2003; Wang, 2008). Second, the inclusion of the perceived value constituents to the success model is a more accurate representation of the dynamics surrounding quality measures, cost perception, satisfaction assessment, repeat use intentions, and learning performance. Finally, this study provides a theoretical framework for researchers to further develop and test theories of m-learning app success, and for practitioners it helps identify what factors make m-learning apps more successful, which can be useful when crowdsourcing. The authors encourage future researchers to continue challenging and testing this paid m-learning app success model.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Yu-Yin Wang is a post-doctoral fellow in the Department of Information Management at the National Changhua University of Education, Taiwan. She received her Ph.D. in Information Management from National Sun Yat-sen University, Taiwan. Her current research interests include mobile learning, technology upgrade model, and educational technology success. She has published papers in Information Technology & People, Internet Research, Behaviour & Information Technology, and International Journal of Information Management.

Yi-Shun Wang is a Distinguished Professor in the Department of Information Management at the National Changhua University of Education, Taiwan. He received his Ph.D. in MIS from National Chengchi University, Taiwan. His current research interests include information and educational technology adoption strategies, IS success models, online user behavior, knowledge management, Internet entrepreneurship education, and e-learning. He has published papers in journals such as Academy of Management Learning and Education, Computers & Education, British Journal of Educational Technology, Information Systems Journal, Information & Management, International Journal of Information Management, Government Information Quarterly, Internet Research, Computers in Human Behavior, International Journal of Human-Computer Interaction, Information Technology and People, Information Technology and Management, Journal of Educational Computing Research, among others. He is currently serving as a Convener of the research discipline of Applied Science Education in the Ministry of Science and Technology of Taiwan.

Hsin-Hui Lin is a Professor and Head in the Department of Distribution Management at National Taichung University of Science and Technology, Taiwan. She received her Ph.D. in Business Administration from National Taiwan University of Science and Technology. Her current research interests include electronic commerce, service marketing, and customer relationship management. Her work has been published in academic journals such as Academy of Management Learning & Education, British Journal of Educational Technology, Information & Management, Information Systems Journal, International Journal of Information Management, Internet Research, Managing Service Quality, Service Industries Journal, Computers in Human Behavior, Journal of Global Information Management, Information Systems and e-Business Management, and Journal of Educational Computing Research.

Tung-Han Tsai is a graduate student in the Department of Information Management at National Changhua University of Education, Taiwan. He received his bachelor's degree in Information Management from National Changhua University of Education, Taiwan. His current research interests include mobile learning, online entrepreneurship, and educational technology acceptance. He has ever served as a principal investigator of the university student project in the Ministry of Science and Technology of Taiwan.

ORCID

Yi-Shun Wang http://orcid.org/0000-0002-0161-5520

Additional information

Funding

This research was substantially supported by the Ministry of Science and Technology Taiwan under grant number MOST 104-2815-C-018-010-U.

Appendix A. Measurement items used in this study

Table

Variable	Item	Description	Reference
System Quality	SQ1	The m-learning app is easy to use.	Kim et al. (2007)
	SQ2	The m-learning app system is reliable.
	SQ3	I find it easy to get the m-learning app to do what I want it to do.
Information Quality	IQ1	The m-learning app provides precise information that I need.	Wang (2008)
	IQ2	The information content meets my needs.
	IQ3	I feel the output is reliable.
Perceived Enjoyment	PE1	I have fun interacting with the m-learning app.	Kim et al. (2007)
	PE2	Using the m-learning provides me with a lot of enjoyment.
	PE3	I enjoy using the m-learning app.
Perceived Fee	PF1	The fee that I have to pay for the use of the m-learning app is too high.	Kim et al. (2007)
	PF2	The fee that I have to pay for the use of the m-learning app is unreasonable.
	PF3	I am unpleased with the fee that I have to pay for the use of the m-learning app.
User Satisfaction	US1	I am satisfied with the m-learning app.	Wang (2008)
	US2	The m-learning app is of high quality.
	US3	The m-learning app has met my expectations.
Intention to Reuse	IR1	Assuming that I have access to the m-learning app, I intend to reuse it.	Wang (2008)
	IR2	I will reuse the m-learning app in the future.
	IR3	I will frequently use the m-learning app in the future.
Learning Effectiveness	LE1	The m-learning app can assist learning performance.	Liaw (2008)
	LE2	The m-learning app can assist learning effectively.
	LE3	The m-learning app can assist learning motivation.