External and Internal Consistency of User Evaluations

Abstract

User evaluations may be useful to improve the quality of public services if they are consistent with service characteristics (i.e., external consistency) and user satisfaction (i.e., internal consistency). This article analyses the internal and external consistency of user evaluations of water services in Nicaragua. Internal consistency is assessed through item-total correlations, Cronbach's alpha coefficient, and factor analysis. External consistency is analyzed through ordered logit and regression models. Results show a considerable degree of internal consistency. However, user evaluations are only partially consistent with service performance as personal characteristics also influence individual perceptions of the overall service, water pressure, system reliability, and water quality.

Keywords:

• user evaluation
• consumer satisfaction
• evaluation validity
• water
• Nicaragua

Notes

¹Citizen feedback has been used to evaluate public services in many developed countries including Australia, New Zealand, the United Kingdom, and the United States (see Griffiths, 2003; Holmes et al., 2006; Van Ryzin, 2008; Van Ryzin et al., 2004). The popularity of citizen feedback is also increasing in Asian and African developing countries such as Bangladesh, India, Indonesia, Pakistan, and South Africa (see Alam et al., 2008; Andaleeb et al., 2007; Deichmann and Lall, 2007; Lewis and Pattinasarany, 2009; Myburgh et al., 2005), and is more broadly applied to programmes targeting democratic decentralisation and voice in development (Brocklesby & Hobley, 2003; Mehrotra, 2006).

²See Munteanu et al. (2010) for a recent application of Cronbach's coefficient alpha to analyse the internal consistency of students' evaluations of several aspects of higher education in Romania. Also see Cortina (1996) for a theoretical discussion of Cronbach's coefficient alpha as a measure of internal consistency.

³Ordered logit models shown in Table 4 were selected based on a number of goodness-of-fit measures including the Wald χ², Akaike's information criterion (AIC), and Bayesian information criterion (BIC). McKelvey and Zavoina's (1975) pseudo R² was also used as it often scores best under comparability criterion (see Veall & Zimmermann, 1996). As Hosmer and Lemeshow (2000) argue, pseudo R² values are useful to select among competing models but may fail to assess goodness of fit because they compare predicted values from the fitted model to those from the intercept only model. Alternatively, Table 4 shows count R² values which are based strictly on a comparison of observed to predicted values from fitted models.