Using state space models as a statistical impact measurement of survey redesigns: a case study of the labour force survey of the Australian Bureau of Statistics

Figures & data

Figure 1. A scheme for 15 months parallel run.

Figure 2. Average SE against treatment sample size, intra-cluster correlation and parallel duration.

Table 1. Average SE Regress on X.

	Employed	Unemployed
Coefficient of X	1.035e-02 (5.482e-05)	0.0706435 (0.0002898)
Null deviance	1.6335e-03	7.6127e-02
Residual deviance	2.1526e-06	6.0167e-05
R^2	0.9987	0.9992

Table 2. Sample size and the length of the parallel run required for unemployment Options A and B.

	A100	A50	B75
Control sample size %	100	100	75
Treatment sample size %	100	50	75
SE on published estimates	Current	Current	1.15 times larger
Duration	32 months	>44	36 months
Risk	Low	Low	Moderate

Table 3. Total level shift detected by SSM across 100 replicates (Unemployed).

Periods after the first new design rotation group is introduced (month)	Overall impact %	MDI ratio
Simulated	2.60
3	1.86	4.1
5	2.23	4.0
8	2.34	3.9

Table 4. Different parallel-run scenarios used in the simulation^a.

	Sample size			Unemployed			Employed
				Standard error			Standard error
Scenario	Control sample	Treatment sample	Parallel-run period month	% points	total	MDI ratio	% points	total	MDI ratio
1	100%	20%	18	7.9	61,620	4.05	1.08	135,000	5.25
2	100%	50%	12	5.6	43,680	2.87	0.81	101,250	3.94
2	100%	20%	24	5.6	43,680	2.87	0.81	101,250	3.94
3	100%	50%	18	3.9	30,420	2.00	0.54	67,500	2.63
4	100%	100%	12	3.7	29,016	1.90	0.54	67,500	2.63
5	100%	25%	24	4.7	36,348	2.41	0.68	85,000	3.31
6	100%	100%	18	2.5	19,500	1.28	0.44	55,000	2.14

^aIt should be noted that in relation to employed persons, Scenario 4 equals Scenario 3.

Figure 3. Minimum detectable impact ratio at the 5% significance level and 50% power obtained with the times-series model for different periods after the changeover among the unemployed labour force (left panel) and the employed labour force (right panel).

Table 5. The relative SEs for the impact measurement estimates and revisions for the unemployed labour force in percentage points after 12 months under different parallel-run options.

	RG1	RG2	RG3	RG4	RG5	RG6	RG7	RG8	Average
SE of impact
No PR	3.18	3.35	3.45	3.52	3.58	3.63	3.66	3.69	3.5
Scenario 1	2.36	2.47	2.51	2.54	2.57	2.63	2.67	2.74	2.6
Scenario 2	2.05	2.13	2.15	2.17	2.19	2.23	2.29	2.37	2.2
Scenario 3	1.74	1.79	1.79	1.79	1.81	1.84	1.91	2.00	1.8
Scenario 4	1.70	1.75	1.74	1.75	1.77	1.80	1.86	1.95	1.8
Scenario 5	1.88	1.95	1.96	1.97	1.99	2.03	2.09	2.18	2.00
Scenario 6	1.42	1.44	1.43	1.43	1.44	1.47	1.52	1.61	1.47
Revision
Scenario 1	5.62	6.13	5.88	6.53	4.71	5.71	5.54	5.93	5.8
Scenario 2	3.97	4.22	3.93	4.57	3.29	3.89	3.82	4.02	4.0
Scenario 3	2.73	2.83	2.59	3.11	2.23	2.56	2.58	2.65	2.7
Scenario 4	2.60	2.69	2.45	2.95	2.12	2.43	2.45	2.51	2.52
Scenario 5	3.29	3.46	3.17	3.77	2.71	3.15	3.13	3.25	3.24
Scenario 6	1.69	1.72	1.55	1.88	1.38	1.54	1.58	1.56	1.61

Table 6. The relative SEs for the impact measurement estimates and revisions for the employed labour force in percentage points after 12 months under different parallel-run options.

	RG1	RG2	RG3	RG4	RG5	RG6	RG7	RG8	Average
SE of impact
No PR	0.57	0.58	0.58	0.61	0.64	0.63	0.63	0.65	0.61
Scenario 1	0.37	0.36	0.35	0.36	0.38	0.37	0.37	0.40	0.37
Scenario 2	0.31	0.30	0.29	0.30	0.31	0.31	0.31	0.33	0.31
Scenario 3	0.25	0.23	0.22	0.22	0.23	0.23	0.24	0.26	0.27
Scenario 4	0.25	0.23	0.22	0.22	0.23	0.23	0.24	0.26	0.27
Scenario 5	0.27	0.28	0.28	0.29	0.29	0.30	0.31	0.33	0.29
Revision
Scenario 1	0.77	0.87	0.81	0.90	0.67	0.78	0.78	0.83	0.80
Scenario 2	0.58	0.63	0.59	0.67	0.50	0.57	0.57	0.61	0.59
Scenario 3	0.38	0.41	0.37	0.44	0.32	0.36	0.36	0.37	0.38
Scenario 4	0.38	0.41	0.37	0.44	0.32	0.36	0.36	0.37	0.38
Scenario 5	0.48	0.52	0.48	0.56	0.41	0.47	0.47	0.49	0.49

Figure 4. Comparisons of the initial SE from parallel run, final SE and revision after the 12-month changeover.

Table 7. Revision size regressing on ini SE.

	Unemployed	Employed
Intercept	–0.367054 (0.021980)	–0.0395645 (0.0008405)
ini SE	0.779785 (0.004307)	0.7774410 (0.0010967)
Null deviance	10.9734000	1.2628e-01
Residual deviance	0.0016739	1.0051e-06
R^2	0.99984	0.999992

Table 8. SE of final estimates regressing on ini SE.

	Unemployed	Employed
Intercept	0.99578 (0.05123)	0.169087 (0.009134)
ini SE	0.20939 (0.01004)	0.180600 (0.011918)
Null deviance	0.8002000	0.0069333
Residual deviance	0.0090932	0.0001187
R^2	0.988636	0.98288

Table

$e_{i,t}^g={\delta }_{i,t}^g$

No AR process

Table

$e_{i,t}^g={\phi }_1{e}_{i-1,t-1}^g+{\delta }_{i,t}^g$

AR(1) process

Table

$e_{i,t}^g={\phi }_1{e}_{i-1,t-1}^g+{\phi }_2{e}_{i-2,t-2}^g+{\delta }_{i,t}^g$

AR(2) process

Table A1. Parameters for Simulation data generation

	Employed	Unemployed
Sample size per month	30,000
Sampling error AR1 for wave 2	0.835	0.589
Sampling error AR2 for wave 3–8	0.585, 0.3	0.466, 0.208
	RGB Control
RSE at RGB	0.94%	6.60%
b1	0.007	0.058267930
b2	0.001	0.019303798
b3	−0.0044	0.006714512
b4	−0.0044	0.000405143
b5	0.0005	0.017966054
b6	0.0001	0.019514134
b7	0	0
b8	0.0002	0.046400917
	RGB Treatment
b1 + α1	0.6	0.7
b2 + α2	0.001	0.2
b3 + α3	−0.001	0.2
b4 + α4	−0.001	−0.02
b5 + α5	0.001	−0.02
b6 + α6	0	−0.02
b7 + α7	0	0
b8 + α8	−0.6	−0.68

Table

Set RSE RG control sample

Iterate replicates 1–100

Iterate parallel run duration: 11, 13, 15 19

Iterate Kappa: 0.3. 0.5. 0.8, 1

Derived Treatment sample RSE from Kappa.

Iterate intra cluster correlation: 0, 0.3, 0.5, 0.8

Generate both control and treatment sample

End

End

End

End

Note: The simulation programme required the input parameters, ${\phi }_1$ for wave 2, ${\phi }_1$ and ${\phi }_2$ for wave 3 to wave 8, standard error of control and treatment group (reflects $\kappa$), $\rho$, and RGB control and treatment parameters.

In the data simulation, the AR(1) parameter ${\phi }_1$ for wave 2 was different to the AR(1) parameter ${\phi }_1$

A different seed was used to generate the white noise component in each replication and therefore, the simulated observations differed only in the sampling error component.