Using state space models as a statistical impact measurement of survey redesigns: a case study of the labour force survey of the Australian Bureau of Statistics

ABSTRACT

The goals of any major business transformation programme in an official statistical agency often include improving data collection efficiency, data processing methodologies and data quality. However, the achievement of such improvements may have transitional statistical impacts that could be misinterpreted as real-world changes if they are not measured and handled appropriately.

This paper describes a development work that sought to explore the design and analysis of a times-series experiment that measured the statistical impacts that sometimes occur during survey redesigns. The Labour Force Survey (LFS) of the Australian Bureau of Statistics (ABS) was used as a case study. In the present study:

1. A large-scale field experiment was designed and conducted that allowed the outgoing and the incoming surveys to run in parallel for some periods to measure the impacts of any changes to the survey process; and

2. The precision of the impact measurement was continuously improved while the new survey design was being implemented.

The state space modelling (SSM) technique was adopted as the main approach, as it provides an efficient impact measurement. This approach enabled sampling error structure to be incorporated in the time-series intervention analysis. The approach was also able to be extended to take advantage of the availability of other related data sources (e.g., the data obtained from the parallel data collection process) to improve the efficiency and accuracy of the impact measurement. As stated above, the LFS was used as a case study; however, the models and methods developed in this study could be extended to other surveys.

KEYWORDS:

• Intervention analysis
• survey sampling
• structural time series modelling
• labour force
• survey
• Kalman filter

Acknowledgements

The authors would especially like to thank the unknown referees and the Associate Editor for careful reading of the manuscript and providing constructive comments as well as Oksana Honchar and Cedric Wong for their valuable contributions to this study, and Sybille McKeown, Professor James Brown, Kristen Stone, Annette Kelly, Rosalynn Mathews, Bruce Fraser, Jacqui Jones and Bjorn Jarvis for their constructive and valuable suggestions and comments. It should be noted that the views expressed in the paper do not necessarily represent those of the ABS and Statistics Netherlands. Any errors in this paper those of the authors.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 There is anecdotal evidence that Wave 7 contained is less biases and more stable among the eight waves.

2 There was no official accuracy criterion at the time at which the paper was written. A hypothetical accuracy criterion was used purely to assist discussion in this paper.

3 In the context of this paper, population refers to employed or unemployed persons.

4 Further elaboration of this simple model may be needed if evidence emerges that this assumption needs revision (see our discussion of future avenues of research in Section 5).

5 It should be noted that the sampling error for the treatment sample may be larger due to the smaller sample size of the treatment sample.

6 Without losing generality, the state equation is written as an AR(2) process but Waves 1 and 2 follow a white noise process and an AR1 process, respectively.

7 The estimated SE of ${\alpha }_i$ for each replicate appeared to be consistent regardless of the waves and the size of ${\alpha }_i$. avg.se was calculated as the average of all replicates across the eight waves; that is, $avg.se=\text{(}1/8\times 100\text{)}{\sum }_{i=1}^8{\sum }_{j=1}^{100}{\hat{\sigma }}_{i,j}$ where ${\hat{\sigma }}_{i,j}$ is the estimated SE of ${\alpha }_i$ (Wave i) for replicate j.

8 A scientific adjustment is described as a hybrid option in Section 4.2.

9 Most of the precision discussions in this section focused on the rotation group level, as the SE of a statistical impact to the overall composite estimates can be approximated as the SE at the rotation level times a multiplier.

Additional information

Notes on contributors

Xichuan (Mark) Zhang

Dr. Xichuan (Mark) Zhang is a statistical methodology specialist in the areas of time series analysis, econometric and survey methodology at Methodology Division of Australian Bureau of Statistics.

Jan A. van den Brakel

Dr. Jan A. van den Brakel is an extraordinary professor of survey methodology at Maastricht University and works as senior statistician at the Methodology Department of Statistics Netherlands.

Siu-Ming Tam

Dr. Siu-Ming Tam is the former chief methodologist and general manager of Methodology Division of Australian Bureau of Statistics, and honorary professorial fellow of University of Wollongong.