The Periodic Review of Performance Indicators: An Empirical Investigation of the Dynamism of Performance Measurement Systems

Abstract

The aim of this study is to examine one attribute of performance measurement systems (PMS) that is not widely addressed in the management accounting literature, namely the dynamism of PMS. This attribute refers to the periodic review of performance indicators by organizations. Based on contingency theory and using survey data from a sample of manufacturing firms, this study examines whether the association between the dynamism of PMS and organizational performance is contingent on the level of external and internal changes. The results suggest three main conclusions. First, even though the current business environment is characterized by fast changes, manufacturing organizations do not appear to revise their PMS to a great extent. Second, while periodic revisions of performance indicators are beneficial, these revisions may not necessarily be appropriate at all times and in all circumstances as the need for dynamic PMS varies depending on the degree of external and internal change. Third, an absence of dynamic PMS may be more harmful in a context of higher levels of change than to have dynamic PMS even if they are not required.

Acknowledgments

The author would like to thank Maurice Gosselin, Sylvie Héroux, Claude Laurin, Alfred Seaman, and Nicole Thibodeau for their insightful comments and suggestions, as well as the two anonymous reviewers.

Notes

Although few accounting studies have explicitly examined the review of performance indicators, many have implicitly addressed this attribute when referring to the feedback process of control systems. Indeed, the feedback process involves changes in objectives and actions, as well as changes in the control systems (e.g. Ittner and Larcker, 2001; Otley, 1999). While not focusing on the review process to ensure that PMS are regularly updated, this framework suggests that the control systems may also be subject to re-evaluations and modifications.

Capabilities are the organizational and strategic routines by which firms synthesize and acquire knowledge resources, and achieve new resource configurations (Eisenhardt and Martin, 2000; Kogut and Zander, 1992).

In this study, ‘firm’ is a fully autonomous entity or a subunit of a larger firm. In all cases, firms appeared as separate entities in the database.

Following other upper echelon studies (e.g. Carpenter and Fredrickson, 2001), a top management team is defined as the top two tiers of an organization's management team, which include CEO/general manager, chief operating officer (COO), chief financial officer (CFO), and the next highest management tier of a firm (senior vice-presidents).

The response rate was calculated as the percentage of the number of usable returned questionnaires to the number of questionnaires sent, after adjusting for the firms that had closed, ended manufacturing activities or moved, or for which the contact person had left the organization.

Early respondents correspond to the first 10% of all respondents ranked following the reception date of the completed questionnaire. Late respondents correspond to the last 10% of all respondents.

The results of the second-order CFA suggest that innovativeness and learning orientation are two first-order constructs that relate to a second-order construct labeled strategic capabilities (see Appendix 2). Thus, the mean score of innovativeness and learning orientation are combined into one variable (strategic capabilities).

The indices used to assess the model are among the most frequently reported, namely NNFI (non-normed fit index), CFI (comparative fit index), and RMSEA (root mean square error of approximation). The threshold values recommended are (i) NNFI > 0.90 (Tabachnick and Fidell, 2001), (ii) CFI > 0.95 (Hu and Bentler, 1995), and (iii) RMSEA < 0.l0 (Browne and Cudeck, 1993).

To assess the inter-rater agreement, an average deviation (AD) index is calculated (Burke et al., 1999). For all the constructs in the validation sample, the AD is estimated at 0.35 and ranges from 0.27 to 0.49 for each construct. Compared with the criterion for acceptable inter-rater agreement and practical significance estimated at 1.2 (Burke and Dunlap, 2002), these results are quite satisfactory. The criterion is approximated as c/6, where c is the number of response options for a Likert-type item. A series of t-tests was carried out to determine whether the mean ratings provided by the first respondent for each construct were significantly different from the mean ratings from the second respondent. The results do not reflect significant differences for any construct (p > 0.05). Moreover, t-tests and chi-square analyses were carried out to compare the mean ratings and sample characteristics (size, industry, location) of the firms with two respondents with those of the firms that have one respondent. No significant differences were found for all constructs and sample characteristics (p > 0.05) except for size. Indeed, the firms that have two respondents appear to be smaller.

As mentioned by Ittner and Larcker (2001), this model is based on numerous assumptions: (i) the model is assumed to be the same for each firm, (ii) the model exhibits the correct functional form, (iii) the model has predictor variables that are measured without errors, and (iv) the model includes all relevant predictor variables. Furthermore, the choice of logistic regression is threefold (Tabachnick and Fidell, 2001). Unlike multiple-regression, logistic regression has no assumptions about the distributions of the predictor variable and it cannot produce negative predicted probabilities. In addition, logistic regression is especially useful when the distribution of responses of the dependent variable is expected to be nonlinear with one or more independent variable.