Consistency in performance rankings: the Peru water sector

Abstract

Regulators can utilize a number of alternative methodologies for comparing firm efficiency, but these approaches need to be robust to be accepted by stakeholders. This study evaluates the consistency of water-utility performance rankings for Peruvian water utilities. The results indicate that data envelopment analysis (DEA) and stochastic frontier analysis (SFA) yield similar rankings in this case. In addition, the techniques have comparable success for identifying the best and worst performing utilities. However, these rankings based on sophisticated statistical techniques are not highly correlated with those developed by the Peruvian water regulator (SUNASS). This result does not invalidate the performance rankings obtained by the regulator, since those rankings are based on more dimensions of utility performance. However, they illustrate the importance of developing sound techniques for identifying weak utilities. Improvements in sector performance require that benchmarking be given greater attention than in the past.

Notes

¹ Recent studies have focused on the relative performance of public and private water utilities in the United States (Bhattacharyya et al., 1994; Wallsten and Kosec, 2005), the United Kingdom (Hunt and Lynk, 1995; Cubbin and Tzanidakis, 1998; Ashton, 2000; Saal and Parker, 2001), Asia (Estache and Rossi, 2002), Latin America (Clarke et al ., 2004; Tupper and Resende, 2004) and Africa (Estache and Kouassi 2002; Kirkpatrick et al ., 2004). For a more detailed and comprehensive literature review, readers are referred to the recent work by Estache et al ., 2005, who survey recent productivity and efficiency literatures in infrastructure industries  – energy, ports, railways, roads, telecommunications and water/sewerage) in developing countries.

² Researchers have used a combination of DEA and SFA (or other methodologies) to assess technical efficiency in several sectors. For example, Mizala et al ., 2002 analyze the technical efficiency of schools in Chile, finding that the two methods give similar rankings for schools. Latruffe et al ., 2004 report the determinants of technical efficiency for crop and livestock farms in Poland (DEA-second stage analysis is used to check the robustness of the SFA results). Also, Fiordelisi and Molyneux, 2004 examined efficiency and productivity in the Italian factoring industry using DEA and the Malmquist Index.

³ As Smith (1990) observes, comparative data provide a mechanism whereby consumers can appraise the quality of local services. The benchmarking results can also help regulators decide the efficient cost and determine the appropriate price caps (Carrington et al ., 2002). In the case of Peru, all the utilities are publicly owned, so using higher X-factors (as a stick) for poorly performing firms may be inappropriate: citizens benefit from lower prices, but they are likely to experience lower service quality as the poorly managed utilities have reduced cash flows. Informed public opinion can exert pressure on managers via local elections. Perhaps devising managerial compensation schemes based on improvements in rankings offers greater promise (as a carrot).

⁴ Books by Cooper et al ., 2004 and Charnes et al ., 1994 survey DEA applications. The book by Kumbhakar and Lovell, 2003 provides a comprehensive overview of various SFA methodologies.

⁵ With a sufficiently large number of periods, a SFA model using panel data can help in mitigating the strong distributional assumptions that are necessary to disentangle the effects of inefficiency and random noise (Coelli et al ., 2005) and reduce the bias due to unobserved heterogeneity (Farsi et al ., 2005). Given the limited number of periods, the panel structure is not explored in this study. A comprehensive review and application of different panel SFA models can be found in Farsi et al ., 2005.

⁶ The detail descriptions of the models are presented in the Technical Appendix available from the authors.

⁷ We estimated a production frontier model: . The result shows that inefficiency accounts for 97% of the deviation. All the signs of the coefficients are consistent with the expectation. The OPEX and number of connections have positive and significant impact on the water delivered.

⁸ We did run the translog model in Equation 9. The monotonicity requirement cannot be satisfied. Two out of the six first order terms have wrong signs for coefficients. One of the two is statistically significant at p  = 0.05. None of the rest first order terms are statistically significant. We also found a very serious muticollinearity problem in the translog function. Many correlations between the regressors exceed 0.85 or even 0.95. In the present study, these problems may prevent us from obtaining meaningful results from the translog specification.

⁹ Caudill et al . (1995) suggests that size-related heteroskedasticity of inefficiency variable could lead to biased estimates if the inefficiency term embodies factors ‘under firm control’. We did try the model proposed by Caudill et al . (1995) where variance of inefficiency term is modelled as a function of firm size (log revenue of the firm in a specific year). The estimation results are consistent with the results presented in Table 3 in terms of sign and statistical significance. The mean efficiency scores of the two models are roughly the same (0.89 vs. 0.9) and individual scores are significantly correlated with each other (the correlation is 0.8). Furthermore, the inclusion of the heteroskedasticity adjusted SFA model does not change our interpretation and conclusion qualitatively in the following comparisons.

¹⁰ For the input-oriented super efficient model, the efficiency scores range from 0 to ∞. The higher the efficiency scores, the more efficient the firms would be. However, the efficiency scores can only give a ranking order of the firms. They are not quantitatively comparable as the CCR and BCC models, because the efficient DMUs are not compared to the same ‘standards’. (The frontier constructed from the remaining DMUs changes for each efficient DMU under evaluation.) (Cooper et al ., 2004)

¹¹ This methodology is somewhat inaccurate in the sense of considering the multiple inputs and outputs simultaneously and drawing the whole picture. However, it provides an intuitive explanation regarding the source of the difference between the SUNASS approach and frontier techniques.

¹² Since there are nonnegligible discrepancies across different measures of efficiency, a comprehensive measure might be useful to regulators in the benchmarking studies. Coelli and Perelman (1999) have suggested combining the results from alternative modelling exercises by using the geometric means of the performance scores for each data point in order to mitigate potential bias of specific methods. This idea is borrowed from the time-series forecasting literature where many authors assert that the mean of the predictions from various models will often outperform any one particular predictive model.

¹³ The policy implications of previous studies that have used only one methodology to evaluate relative efficiency would be strengthened by utilizing the types of sensitivity tests described here. Recent studies of other sectors include European banking (Casu and Molyneux, 2003), museums (Bishop and Brand, 2003), Korean container terminals (Cullinane and Song, 2003), Swiss hospitals (Steinmann and Zweifl, 2003) and the Greek agricultural sector (Rezitis et al ., 2002).

¹⁴ Lynch et al ., 1994 use hierarchical conjoint analysis to derive weights for dimensions of telephone service quality. The methodology is applicable to water as well—for example, by giving different weights to system expansion and to improvements in service continuity.