A head-to-head comparison of the out-of-control performance of control charts adjusted for parameter estimation

Abstract

When in-control process parameters are estimated, this can have a substantial effect on the control chart performance. In particular, it may lead to high false alarm rates for a large number of practitioners. In recent literature, control limit adjustments have been proposed for Shewhart, CUSUM and EWMA control charts in order to provide a specified in-control performance with a specified high probability. In this paper, we compare the out-of-control performance of these adjusted Shewhart, CUSUM and EWMA control charts for sustained shifts in the process mean. We find that the CUSUM control chart has faster detection of sustained shifts compared to both the EWMA and Shewhart control charts. This finding generalizes to almost all shift sizes and estimation errors considered in this paper. The performance of the EWMA is not far worse than that of the CUSUM, but the Shewhart control chart is much slower in detecting sustained shifts in the mean compared to these other two charts.

Keywords:

• control charts
• CUSUM control chart
• EWMA control charts
• process monitoring
• statistical process control

Introduction

Control charts are an important tool to aid in the detection of process changes. Three commonly used types of control charts are the Shewhart, cumulative sum (CUSUM), and exponentially weighted moving average (EWMA) charts (see e.g. Montgomery 2013). Each of these charts have their own characteristics which make them more or less applicable to detect certain types of shifts. For example, the Shewhart control chart is better suited to detect large shifts, while the CUSUM and EWMA yield better detection capabilities against small sustained shifts. These aspects increase the demand for comparative studies, where the control chart capabilities are evaluated based on different scenarios.

To this end, Hawkins and Wu (2014) performed a comparative study of the CUSUM and EWMA control charts. They found that the CUSUM outperforms the EWMA in the case that the actual shift to be expected is (approximately) known in size. However, their comparison is based on the assumption of known in-control process parameters. In practice, these parameters have to be estimated using a Phase I reference sample. Since different practitioners obtain different Phase I samples their parameter estimates will differ, leading to different estimated control limits. The performance of the control chart is then conditional on these obtained estimates. This effect of Phase I estimation has received much attention in recent literature. We refer to Jensen et al. (2006) and Psarakis, Vyniou, and Castagliola (2014) for literature reviews on this topic.

Zwetsloot and Woodall (2017) perform a comparative study on the conditional performance of the Shewhart, CUSUM, and EWMA control charts, where they compare the effect of estimation error across these charts. They conclude that the Shewhart chart is most affected by estimation error, and that the EWMA and CUSUM charts behave quite differently when evaluated on conditional performance. They advise to consider the conditional performance for an appropriate comparison. This is the main topic of this paper. Additionally, it is possible to adjust the control chart design to match a certain performance criterion when parameters are estimated. The performance is generally measured in terms of the false alarm rate (FAR) or the average run length (ARL). This approach has been proposed by various researchers, among others Nedumaran and Pignatiello (2001), Albers and Kallenberg (2004a, 2004b, 2005), Tsai, Wu, and Lin (2004), Tsai et al. (2005). Albers and Kallenberg (2004a, 2004b) introduced the exceedance probability criterion, which aims to provide a specified minimum in-control performance with a specified high probability. This criterion has gained increasing attention in the more recent literature on statistical process monitoring (SPM), and was recommended by Jones and Steiner (2012), Gandy and Kvaløy (2013), and Saleh, Mahmoud, Keefe, et al. (2015), Saleh, Mahmoud, Jones-Farmer, et al. (2015), amongst others. Moreover, bootstrap, analytical, and numerical (approximation) methods have been applied to determine the required control limits for Shewhart, CUSUM and EWMA control charts. For recent papers on such methods, we refer to Gandy and Kvaløy (2013), Faraz, Woodall, and Heuchenne (2015), Saleh, Mahmoud, Jones-Farmer, et al. (2015), Goedhart, Schoonhoven, and Does (2017), Goedhart, da Silva, et al. (2017), Goedhart, Schoonhoven, and Does (2018), Diko, Chakraborti, and Does (2019a, 2019b) and Jardim, Chakraborti, and Epprecht (2019).

With these new conditional control chart designs for location, the question arises how it influences the performance of the Shewhart, CUSUM and EWMA control charts in various scenarios. In this paper we therefore perform a comparative study between these control charts with estimated parameters. Because the exceedance probability criterion provides a specified in-control performance, we focus mainly on the out-of-control detection capability of the control charts.

This paper is organized as follows. In the next section we define the conditional Shewhart, CUSUM and EWMA control chart designs, the estimators used in Phase I estimation, and our evaluation criteria. In Section Evaluation we explain our simulation procedures and discuss the corresponding results. In the final section we provide some concluding remarks.

Shewhart, CUSUM and EWMA control charts

In this paper we consider the Shewhart, CUSUM and EWMA control charts. For each of these charts, the first step (Phase I) is to estimate the in-control behavior of the underlying process, before one can start the monitoring phase (Phase II). Suppose that in Phase I, an in-control sample consisting of m subgroups of size n is available to estimate the in-control behavior of the process. Let $X_{\mathit{ij}}$ (for i = 1,…,m and j = 1,…,n) be the j-th observation in the i-th subgroup in Phase I, where $X_{\mathit{ij}}$ follows a normal distribution with mean ${\mu }_0$ and standard deviation ${\sigma }_0$ (i.e., $N({\mu }_0,\mathrm{}{\sigma }_0)$). In Phase II ($i=m+1,\mathrm{}m+2,\dots ,$), we assume that there may be a shift in the mean of $\delta$ standard deviations, such that the observations are drawn from a $N\left({\mu }_0+\delta {\sigma }_0,\mathrm{}{\sigma }_0\right)$ distribution. We denote the Phase II mean as $\mu ={\mu }_0+\delta {\sigma }_0.$ Note that $\delta =0\mathrm{}$ corresponds to the in-control situation. This setup is similar to that of the one used in the comparison papers of Hawkins and Wu (2014) and Zwetsloot and Woodall (2017).

Since the values of ${\mu }_0$ and ${\sigma }_0$ are generally unknown, they have to be estimated. As an estimator for ${\mu }_0$ we use the overall sample mean: [1] $${\widehat{\mu }}_0=\frac{1}{\mathit{mn}}\sum _{i=1}^m\sum _{j=1}^n{X}_{\mathit{ij}}$$[1]

As an estimator for ${\sigma }_0$ we use the pooled sample standard deviation, defined as [2] $${\widehat{\sigma }}_0=\sqrt{\frac{1}{m}\sum _{i=1}^m{S}_i^2}$$[2] where $S_i^2$ is the i-th subgroup sample variance, i.e. $\frac{1}{n-1}{\sum }_{i=1}^n{\left(X_{\mathit{ij}}-{\overline{X}}_i\right)}^2.$ Note that the pooled standard deviation is only applicable when $n>1.$ When $n=1,$ one has to resort to other estimators of ${\sigma }_0$ such as the average moving range or the sample standard deviation. However, we do not consider this situation explicitly in this paper.

In Phase II we monitor the process characteristic [3] $$W_i=\frac{{\overline{X}}_i-{\widehat{\mu }}_0}{{\widehat{\sigma }}_0/\sqrt{n}}$$[3] and we want to detect changes in the location parameter ${\mu }_0.$ Similarly to Jones, Champ, and Rigdon (2001), we expand this equation for comparison purposes into [4] $$W_i=\frac{{\overline{X}}_i-{\widehat{\mu }}_0}{{\widehat{\sigma }}_0/\sqrt{n}}=\left[T_i+\delta \sqrt{n}-\frac{Z}{\sqrt{m}}\right]Q^{-1}$$[4] where $T_i=\frac{\left({\overline{X}}_i-\mu \right)}{{\sigma }_0/\sqrt{n}},$ $\delta =\frac{\mu -{\mu }_0}{{\sigma }_0},$ $Z=\frac{\sqrt{mn}\left({\widehat{\mu }}_0-{\mu }_0\right)}{{\sigma }_0},$ and $Q=\frac{{\widehat{\sigma }}_0}{{\sigma }_0}$. Note that $T_i$ represents the standardized difference between the mean of Phase II subgroup $i$ and the Phase II population mean, $\delta$ represents the size of the Phase II mean shift, $Z$ represents the estimation error of the mean in Phase I, and $Q$ represents the Phase I estimation error of the standard deviation. Note also that, for the chosen estimators as in [1][1] $${\widehat{\mu }}_0=\frac{1}{\mathit{mn}}\sum _{i=1}^m\sum _{j=1}^n{X}_{\mathit{ij}}$$[1] and [2][2] $${\widehat{\sigma }}_0=\sqrt{\frac{1}{m}\sum _{i=1}^m{S}_i^2}$$[2] , $T_i$ and $Z$ are standard normal variables, and that $m(n-1)Q^2$ follows a chi-square distribution with $m(n-1)$ degrees of freedom.

Shewhart control chart

The Shewhart control chart consists of a lower control limit (LCL) and an upper control limit (UCL), and a charting statistic (cf. Montgomery 2013). The charting statistic in Phase II at time period $i=m+1,\mathrm{}m+2,\dots ,$ is equal to the $i$-th subgroup average ${\overline{X}}_i.$ The chart provides a signal when either ${\overline{X}}_i>\widehat{\mathit{UCL}}$ or ${\overline{X}}_i<\widehat{\mathit{LCL}},$ where $\widehat{\mathit{LCL}}$ and $\widehat{\mathit{UCL}}$ are the estimated control limits defined as [5] $$\begin{array}{c}\widehat{\mathit{LCL}}={\widehat{\mu }}_0-C\frac{{\widehat{\sigma }}_0}{\sqrt{n}}\\ \widehat{\mathit{UCL}}={\widehat{\mu }}_0+C\frac{{\widehat{\sigma }}_0}{\sqrt{n}}\end{array}$$[5]

Note that this is equivalent to a signal when $W_i>C$ or $W_i<-C.$ Here $C$ is some constant determined such that the control chart yields a specified in-control performance. For example, when parameters are known and would not require estimation, a value of $C=3$ would provide a false alarm rate of 0.0027, or equivalently an in-control average run length ($\mathit{AR}{L}_0$) of 370.4. When parameters are estimated the conditional alarm rate (CAR), conditional on the estimates Q and Z, and given the size of shift $\delta ,$ is equal to

[6] $$\begin{array}{c}C\mathit{AR}=P\left(W_i>C|Q,Z,\delta \right)+P\left(W_i<-C|Q,Z,\delta \right)\\ =P\left(\left[T_i+\delta \sqrt{n}-\frac{Z}{\sqrt{m}}\right]Q^{-1}>C|Q,Z,\delta \right)+P\left(\left[T_i+\delta \sqrt{n}-\frac{Z}{\sqrt{m}}\right]Q^{-1}<-C|Q,Z,\delta \right)\\ =P\left(T_i>\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}|Q,Z,\delta \right)+P\left(T_i<-\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}|Q,Z,\delta \right)\\ =1-\Phi \left(\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}\right)+\Phi \left(-\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}\right)\end{array}$$[6] where $\mathrm{\Phi }$ represents the standard normal cdf. The conditional run length distribution of this control chart is then equal to a geometric distribution with parameter CAR, so that the conditional average run length (CARL) is equal to $1/\mathit{CAR}.$ Note that, because $Z$ and $Q$ are in practice unknown, the CAR and CARL are not known to the practitioners, but can only be used for a general performance evaluation as in this paper.

CUSUM control chart

Page (1954) introduced the cumulative sum (CUSUM) control chart for monitoring the process mean. In this paper, we consider the two-sided CUSUM chart of Ewan and Kemp (1960). This CUSUM control chart plots two charting statistics at the same time: $C_i^{+}=\max (0,C_{i-1}^{+}+W_i-k)$ and $C_i^{-}=\min (0,C_{i-1}^{-}+W_i+\mathit{k}).$ As starting values we choose $C_0^{+}=C_0^{-}=0.$ The value $k$ is usually chosen to be half the shift that is considered important enough to be detected, measured as number of standard deviations from the mean. The CUSUM control chart provides a signal when either $C_i^{+}>h$ or $C_i^{-}<-h,$ where the constant $h$ is determined such that the control chart yields a specified in-control performance.

In this paper we consider CUSUM charts designed to detect shifts in the mean equal to 0.5 (small), 1 (medium), and 1.5 (large) standard deviations of the monitoring statistic, which means the corresponding values of $k$ are equal to 0.25, 0.5 and 0.75 respectively. When parameters are known, the corresponding values of $h$ for an in-control $\mathit{AR}{L}_0=370$ are obtained using the function xcusum.crit(k,$\mathit{AR}{L}_0$, sided=“two”) from the R-package spc.

EWMA control chart

The exponentially weigthed moving average (EWMA) chart was introduced by Roberts (1959). For the EWMA control chart, the charting statistic at time $i$ is defined as $Y_i=\lambda W_i+\left(1-\lambda \right)Y_{i-1},$ where $\lambda$ is a smoothing constant that depends on the size of the shift that is desired to be detected quickly. As a starting value we set $Y_0=0.$ The EWMA control chart provides a signal when $Y_i>L\sqrt{\lambda /(2-\lambda )}$ or $Y_i<-\mathrm{}L\sqrt{\lambda /(2-\lambda )}$ where the constant $L$ is determined such that the control chart yields a specified in-control performance. For simplicity, we use the asymptotic (steady state) control limits.

There are many combinations of $L$ and $\lambda$ that achieve the same in-control performance. Among these, the chart with the lowest out-of-control ARL at a specified shift of interest, is considered to be the best. To find such a chart, we generated a sequence of $\lambda$ values from 0.01 to 1 using steps of 0.01. For each $\lambda$ value, the corresponding L value was found such that ${\mathit{ARL}}_0=370$ when parameters are known. This was done by using the function xewma.crit($\lambda ,$$\mathit{AR}{L}_0,$ sided=“two”) from the R-package spc. For each obtained combination of $L$ and $\lambda ,$ and for a given shift $\delta ,$ the out-of-control ARL was calculated using the function xewma.arl($\lambda ,$$L,$$\delta ,$sided=“two”) from the same R-package. This approach is similar to that of Hawkins and Wu (2014). We consider EWMA charts designed to detect shifts in the mean equal to 0.5 (small), 1 (medium), and 1.5 (large) standard deviations. The corresponding best charts for these shifts were found to be ($\lambda ,\mathrm{}L)$ equal to $\left(0.05,2.490\right),$ $\left(0.14,2.785\right)$ and $\left(0.25,2.898\right),$ respectively.

Adjusted control limits

When parameters are estimated, control chart properties such as the average run length become random variables. This leads to a large variation in control chart performance across practitioners. In order to prevent low in-control ARLs, many researchers recently suggested the use of the exceedance probability criterion (EPC), where a specified minimum in-control performance ${\mathit{ARL}}_0$ is obtained with a specified probability $1-p.$ See for example Albers and Kallenberg (2004a, 2005), Jones and Steiner (2012), Gandy and Kvaløy (2013), Saleh, Mahmoud, Keefe, et al. (2015), Saleh, Mahmoud, Jones-Farmer, et al. (2015), Goedhart, Schoonhoven, and Does (2017), Goedhart, da Silva, et al. (2017), Goedhart, Schoonhoven, and Does (2018), Diko, Chakraborti, and Does (2019a, 2019b), and Jardim, Chakraborti, and Epprecht (2019). More specifically, if we define $\mathit{CAR}{L}_{\mathit{IN}}$ as the conditional in-control ARL conditional on the Phase I parameter estimates, then the EPC can formally be written as [7] $$P\left({\mathit{CARL}}_{\mathit{IN}}\ge {\mathit{ARL}}_0\right)=1-p,$$[7]

Choosing a small value of $p$ such as 0.05 or 0.10 ensures that there is only a small probability of obtaining a ${\mathit{CARL}}_{\mathit{IN}}$ value that is below ${\mathit{ARL}}_0.$

For each of the discussed control charts, ${\mathit{CARL}}_{\mathit{IN}}$ depends on the Phase I parameters estimates as well as the chosen control limit constants $C,$ h, and L. For a given Phase I sample (size), these constants can be adjusted to match the EPC. We consider adjustments of $C$ as provided in Goedhart, Schoonhoven, and Does (2018), for h as provided in Diko, Chakraborti, and Does (2019b), and for L according to Diko, Chakraborti, and Does (2019a), respectively. In Table 1 we provide an overview of the required values for the Shewhart control chart, and for the EWMA and CUSUM control charts designed to detect shifts of $\delta$ equal to 0.5 (small), 1 (medium) and 1.5 (large). For all charts, we used a Phase I sample of $m=50$ subgroups of size $n=5,$ ${\mathit{ARL}}_0=370$ and $p=0.1$ in this paper. We have considered other values as well, but they did not provide any additional insights for the comparison.

Table 1. Control limit constants.

Chart	$\mathit{\delta }$ (shift)	k	h	Adjusted h
CUSUM	0.5 (small)	0.25	8.01	16.46
	1 (medium)	0.50	4.774	6.68
	1.5(large)	0.75	3.339	4.25
	$\mathit{\delta }$ (shift)	$\mathit{\lambda }$	L	Adjusted L
EWMA	0.5 (small)	0.05	2.490	3.60
	1 (medium)	0.14	2.785	3.42
	1.5(large)	0.25	2.898	3.35
Shewhart	$\mathit{\delta }$ (shift)	$\mathit{\lambda }$	C	Adjusted C
	3 (for all shifts)	1	3	3.24

Evaluation

In this section we evaluate the performance of the Shewhart, CUSUM and EWMA control charts with estimated parameters. We evaluate a wide range of combinations for estimation errors and shift sizes, and compare the performance of the charts based on these results.

The performance of a control chart is conditional on its parameter estimates, through Q and Z. Therefore, we consider various percentiles of Q and Z as well as various possible shift sizes $\delta$ to evaluate the control chart performances. For the Shewhart control chart, all properties of the conditional run length distribution can be determined from a geometric distribution with parameter $\mathit{CAR}$ according to [6][6] $$\begin{array}{c}C\mathit{AR}=P\left(W_i>C|Q,Z,\delta \right)+P\left(W_i<-C|Q,Z,\delta \right)\\ =P\left(\left[T_i+\delta \sqrt{n}-\frac{Z}{\sqrt{m}}\right]Q^{-1}>C|Q,Z,\delta \right)+P\left(\left[T_i+\delta \sqrt{n}-\frac{Z}{\sqrt{m}}\right]Q^{-1}<-C|Q,Z,\delta \right)\\ =P\left(T_i>\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}|Q,Z,\delta \right)+P\left(T_i<-\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}|Q,Z,\delta \right)\\ =1-\Phi \left(\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}\right)+\Phi \left(-\mathit{CQ}+Z/\sqrt{m}-\delta \sqrt{n}\right)\end{array}$$[6] . For the CUSUM control chart we have calculated the CARL values using a modified Siegmund formula approximation as described in Diko, Chakraborti, and Does (2019b), and for the EWMA we use a Markov-Chain approximation such as in Diko, Chakraborti, and Does (2019a). We have also evaluated other properties of the conditional run length distributions such as the standard deviation and several percentiles obtained by simulations. However, as these did not provide any additional insights, we do not discuss them further in this paper.

After obtaining all the results, we have determined, for each pair of control charts, the ratio between the average run length for any considered combination of Q, Z, and $\delta .$

For Q, we consider the 25^th, 50^th and 75^th percentile of its corresponding distribution. These correspond to an underestimation, good estimate (median), and overestimation of the process standard deviation respectively. Note that, because of the skewed distribution of Q, the median is a slight underestimation of the expectation of Q. However, for the values $m=50$ and $n=5$ considered here, the expectation of Q is equal to the 51^st percentile of its distribution, such that using the mean in the comparison instead of the median doesn’t make a real difference. For Z, we consider the 5^th, 25^th, 50^th, 75^th and 95^th percentile of its corresponding distribution. Here we included the 5^th and 95^th percentile in order to display the results for more substantial under- and overestimations respectively. For Q, the 5^th and 95^th percentiles did not provide any new insights for the comparison of the charts, so we have omitted them for a better overview.

For $\delta ,$ we consider the range of 0 to 1 in steps of 0.25. As described earlier, we consider CUSUM and EWMA charts for 0.5 (small), 1 (medium) and 1.5 (large) shifts in terms of standard deviation. However, one should note that these “optimal” designs are based on shifts in terms of the standard deviation of themonitoring statistic (${\sigma }_{{\overline{X}}_i}),$ and not of the standard deviation of the original,individual data (${\sigma }_0).$ The standard deviation of ${\overline{X}}_i\mathrm{}$ equals ${\sigma }_{{\overline{X}}_i}={\sigma }_0/\sqrt{n},$ such that a shift in the mean of $\delta {\sigma }_0$ is equivalent to $\sqrt{n}\delta {\sigma }_{{\overline{X}}_i}.$ This means that for $n=5,$ the value of $\delta$ corresponding to the design shift size of the EWMA and CUSUM is closest to 0.25, 0.5 and 0.75 for small, medium and large shifts respectively.

Results

General performance

The CARL values for the Shewhart, CUSUM and EWMA control charts are given in Tables 2, 3 and 4 respectively. For all three charts, we observe that the in-control ($\delta =0$) CARL values are in general larger when the mean is estimated more accurately (i.e., Z closer to the 50^th percentile). When the mean is accurately estimated, these CARL values are substantially larger than $\mathit{AR}{L}_0$ of 370 for the values of Q considered here. This is a natural consequence of the control limit adjustments in order to guarantee a minimum in-control ARL with a specified (large) probability. Note that large in-control CARL values are actually favorable, while for the out-of-control situation, smaller values mean quicker detection on average. Another general observation is that in the out-of-control situation ($\delta >0$), because we only consider positive shifts, the CARL values increase with the percentiles of Z. Obviously, if the mean is overestimated in Phase I, detection of increases in the process mean become more difficult to detect. If we would consider negative shifts, this would also hold the other way around.

Table 2. Shewhart CARL values.

Z	Q δ	25^th	50th	75^th
5^th	0	435	623	906
	0.25	102	137	187
	0.50	26	34	43
	0.75	9	11	13
	1	4	5	5
25^th	0	538	780	1149
	0.25	147	200	277
	0.50	36	46	60
	0.75	11	14	17
	1	5	5	6
50^th	0	564	821	1213
	0.25	191	263	368
	0.50	45	58	77
	0.75	14	17	21
	1	5	6	7
75th	0	538	780	1149
	0.25	247	344	487
	0.50	56	74	99
	0.75	16	20	26
	1	6	7	9
95th	0	435	623	906
	0.25	353	499	718
	0.50	80	106	143
	0.75	22	27	35
	1	8	9	11

Table 3. CUSUM CARL values.

		Small			Medium			Large
		k = 0.25, h = 16.46			k = 0.5, h = 6.68			k = 0.75, h = 4.25
Z	Q δ	25th	50^th	75^th	25th	50^th	75^th	25th	50^th	75^th
5^th	0	321	382	460	319	420	564	343	476	673
	0.25	29	31	32	19	21	23	22	25	29
	0.50	15	16	16	8	9	9	7	8	8
	0.75	10	10	11	5	5	6	4	4	5
	1	8	8	8	4	4	4	3	3	3
25^th	0	3226	4622	6785	1058	1529	2254	765	1128	1697
	0.25	38	40	43	29	32	35	36	42	51
	0.50	17	18	18	9	10	11	9	9	10
	0.75	11	11	12	6	6	6	5	5	5
	1	8	8	9	4	4	4	3	3	3
50^th	0	15977	26180	44042	1651	2494	3849	971	1465	2257
	0.25	49	52	55	41	46	52	54	66	81
	0.50	19	20	20	11	11	12	10	11	12
	0.75	12	12	13	6	6	7	5	5	6
	1	8	9	9	4	4	5	3	3	4
75th	0	3226	4622	6785	1058	1529	2254	765	1128	1697
	0.25	67	71	76	64	75	88	87	109	140
	0.50	21	22	23	12	13	14	12	13	15
	0.75	12	13	13	7	7	7	5	6	6
	1	9	9	10	4	5	5	3	4	4
95th	0	321	382	460	319	420	564	343	476	673
	0.25	135	150	168	151	189	239	190	253	344
	0.50	25	26	28	16	17	18	17	19	21
	0.75	14	14	15	7	8	8	6	7	7
	1	10	10	10	5	5	5	4	4	4

Table 4. EWMA CARL values.

		Small			Medium			Large
		λ = 0.05, L = 3.60			λ = 0.14, L = 3.42			λ = 0.25, L = 3.35
Z	Q δ	25th	50^th	75^th	25th	50^th	75^th	25th	50^th	75^th
5^th	0	330	407	510	334	437	583	341	460	632
	0.25	29	30	32	23	25	28	25	29	33
	0.50	15	15	16	10	10	11	9	9	10
	0.75	10	10	11	6	7	7	5	5	6
	1	8	8	8	5	5	5	4	4	4
25^th	0	1979	2749	3894	1084	1530	2198	797	1137	1651
	0.25	38	41	43	34	38	43	39	46	55
	0.50	17	17	18	11	12	13	10	11	12
	0.75	11	11	12	7	7	8	6	6	6
	1	8	8	9	5	5	5	4	4	4
50^th	0	5833	8800	13549	1715	2517	3763	1043	1523	2263
	0.25	50	53	58	48	55	64	57	70	85
	0.50	18	19	20	13	14	15	12	13	14
	0.75	12	12	12	7	8	8	6	7	7
	1	8	9	9	5	5	6	4	4	5
75th	0	1908	2671	3810	1062	1506	2172	788	1127	1639
	0.25	70	77	85	74	87	105	89	111	141
	0.50	21	21	22	15	16	17	15	16	18
	0.75	12	13	13	8	8	9	7	7	8
	1	9	9	10	6	6	6	4	5	5
95th	0	287	362	462	314	416	560	329	448	619
	0.25	148	171	201	166	207	264	189	247	328
	0.50	25	26	27	19	20	22	20	22	25
	0.75	14	14	15	9	9	10	8	8	9
	1	10	10	10	6	6	6	5	5	5

When considering the results for the Shewhart chart only (Table 2), we observe that the estimate of the mean has substantially less impact on the CARL than the estimate of standard deviation. This can be seen by first considering only the values for which Q is at the 50^th percentile (good estimate), and evaluating the impact of changes in Z. Comparing the in-control situation for the different percentiles, we observe some, but not too much change in the CARL values. However, if we consider a good estimate of the mean (50^th percentile of Z), we observe that the impact of estimation of standard deviation contributes to larger differences in the CARL values. This result was also obtained in Zwetsloot and Woodall (2017). Note that an overestimation of the mean will increase both the UCL and the LCL, which will increase the likeliness of a signal below the LCL, but decrease that of a signal above the UCL. On the other hand, overestimation of the standard deviation increases the UCL, and decreases the LCL, decreasing the alarm rate on both sides of the control chart.

The results of the CUSUM and EWMA are quite similar. Compared to the Shewhart chart, these charts are much more affected by estimation error in the mean. This is especially so for the charts designed to detect small shifts. Without adjustments, small estimation errors can lead to low in-control CARL values (see for example Saleh, Mahmoud, Jones-Farmer, et al. 2015). In order to correct for this, substantial control limit adjustments are required. When designed to detect small shifts, this required correction is additionally large, as small estimation errors would otherwise quickly be detected as process changes. Note that this also has consequences for the out-of-control performance: the charts designed for medium shifts are often better at detecting small shifts than charts designed for small shifts, as they yield lower out-of-control CARL values for many particular combinations of Z and Q. For CUSUM and EWMA charts designed for larger shifts, the impact of estimation error of the mean is more similar to that of the Shewhart charts. Note also that the Shewhart chart is actually a special case of the EWMA when $\lambda =1.$ The impact of estimation error of the standard deviation on the CUSUM and EWMA control charts is quite similar to that of the Shewhart control chart. Overestimation of the standard deviation leads to larger CARL values, both in-control and out-of-control.

Pairwise comparison

For an easier comparison, we have determined the CARL ratios for the considered values of Q, Z and $\delta$ for each of the charts. In Tables 5, 6 and 7 we list the results for the pairwise results between the Shewhart, CUSUM and EWMA when the latter two are designed for small, medium, or large shift detection respectively. Note that a value larger than 1 means that the chart considered in the numerator has a larger CARL value than the one in the denominator for that particular combination of Q, Z and $\delta .$ A ratio of 1 means that the CARL values are equal.

Table 5. Pairwise CARL ratios (CUSUM and EWMA designed for small shift).

		CUSUM/EWMA			CUSUM/SHEWHART			EWMA/SHEWHART
Z	Q δ	25th	50^th	75^th	25th	50^th	75^th	25th	50^th	75^th
5^th	0	0.97	0.94	0.90	0.74	0.61	0.51	0.76	0.65	0.56
	0.25	1.00	1.03	1.00	0.28	0.23	0.17	0.28	0.22	0.17
	0.50	1.00	1.07	1.00	0.58	0.47	0.37	0.58	0.44	0.37
	0.75	1.00	1.00	1.00	1.11	0.91	0.85	1.11	0.91	0.85
	1	1.00	1.00	1.00	2.00	1.60	1.60	2.00	1.60	1.60
25^th	0	1.63	1.68	1.74	6.00	5.93	5.91	3.68	3.52	3.39
	0.25	1.00	0.98	1.00	0.26	0.20	0.16	0.26	0.21	0.16
	0.50	1.00	1.06	1.00	0.47	0.39	0.30	0.47	0.37	0.30
	0.75	1.00	1.00	1.00	1.00	0.79	0.71	1.00	0.79	0.71
	1	1.00	1.00	1.00	1.60	1.60	1.50	1.60	1.60	1.50
50^th	0	2.74	2.98	3.25	28.33	31.89	36.31	10.34	10.72	11.17
	0.25	0.98	0.98	0.95	0.26	0.20	0.15	0.26	0.20	0.16
	0.50	1.06	1.05	1.00	0.42	0.34	0.26	0.40	0.33	0.26
	0.75	1.00	1.00	1.08	0.86	0.71	0.62	0.86	0.71	0.57
	1	1.00	1.00	1.00	1.60	1.50	1.29	1.60	1.50	1.29
75^th	0	1.69	1.73	1.78	6.00	5.93	5.91	3.55	3.42	3.32
	0.25	0.96	0.92	0.89	0.27	0.21	0.16	0.28	0.22	0.17
	0.50	1.00	1.05	1.05	0.38	0.30	0.23	0.38	0.28	0.22
	0.75	1.00	1.00	1.00	0.75	0.65	0.50	0.75	0.65	0.50
	1	1.00	1.00	1.00	1.50	1.29	1.11	1.50	1.29	1.11
95th	0	1.12	1.06	1.00	0.74	0.61	0.51	0.66	0.58	0.51
	0.25	0.91	0.88	0.84	0.38	0.30	0.23	0.42	0.34	0.28
	0.50	1.00	1.00	1.04	0.31	0.25	0.20	0.31	0.25	0.19
	0.75	1.00	1.00	1.00	0.64	0.52	0.43	0.64	0.52	0.43
	1	1.00	1.00	1.00	1.25	1.11	0.91	1.25	1.11	0.91

Table 6. Pairwise CARL ratios (CUSUM and EWMA designed for medium shift).

		CUSUM/EWMA			CUSUM/SHEWHART			EWMA/SHEWHART
Z	Q δ	25th	50^th	75^th	25th	50^th	75^th	25th	50^th	75^th
5^th	0	0.96	0.96	0.97	0.73	0.67	0.62	0.77	0.70	0.64
	0.25	0.83	0.84	0.82	0.19	0.15	0.12	0.23	0.18	0.15
	0.50	0.80	0.90	0.82	0.31	0.26	0.21	0.38	0.29	0.26
	0.75	0.83	0.71	0.86	0.56	0.45	0.46	0.67	0.64	0.54
	1	0.80	0.80	0.80	1.00	0.80	0.80	1.25	1.00	1.00
25^th	0	0.98	1.00	1.03	1.97	1.96	1.96	2.01	1.96	1.91
	0.25	0.85	0.84	0.81	0.20	0.16	0.13	0.23	0.19	0.16
	0.50	0.82	0.83	0.85	0.25	0.22	0.18	0.31	0.26	0.22
	0.75	0.86	0.86	0.75	0.55	0.43	0.35	0.64	0.50	0.47
	1	0.80	0.80	0.80	0.80	0.80	0.67	1.00	1.00	0.83
50^th	0	0.96	0.99	1.02	2.93	3.04	3.17	3.04	3.07	3.10
	0.25	0.85	0.84	0.81	0.21	0.17	0.14	0.25	0.21	0.17
	0.50	0.85	0.79	0.80	0.24	0.19	0.16	0.29	0.24	0.19
	0.75	0.86	0.75	0.88	0.43	0.35	0.33	0.50	0.47	0.38
	1	0.80	0.80	0.83	0.80	0.67	0.71	1.00	0.83	0.86
	0	1.00	1.02	1.04	1.97	1.96	1.96	1.97	1.93	1.89
75^th	0.25	0.86	0.86	0.84	0.26	0.22	0.18	0.30	0.25	0.22
	0.50	0.80	0.81	0.82	0.21	0.18	0.14	0.27	0.22	0.17
	0.75	0.88	0.88	0.78	0.44	0.35	0.27	0.50	0.40	0.35
	1	0.67	0.83	0.83	0.67	0.71	0.56	1.00	0.86	0.67
95th	0	1.02	1.01	1.01	0.73	0.67	0.62	0.72	0.67	0.62
	0.25	0.91	0.91	0.91	0.43	0.38	0.33	0.47	0.41	0.37
	0.50	0.84	0.85	0.82	0.20	0.16	0.13	0.24	0.19	0.15
	0.75	0.78	0.89	0.80	0.32	0.30	0.23	0.41	0.33	0.29
	1	0.83	0.83	0.83	0.63	0.56	0.45	0.75	0.67	0.55

Table 7. Pairwise CARL ratios (CUSUM and EWMA designed for large shift).

		CUSUM/EWMA			CUSUM/SHEWHART			EWMA/SHEWHART
Z	Q δ	25th	50^th	75^th	25th	50^th	75^th	25th	50^th	75^th
5^th	0	1.01	1.03	1.06	0.79	0.76	0.74	0.78	0.74	0.70
	0.25	0.88	0.86	0.88	0.22	0.18	0.16	0.25	0.21	0.18
	0.50	0.78	0.89	0.80	0.27	0.24	0.19	0.35	0.26	0.23
	0.75	0.80	0.80	0.83	0.44	0.36	0.38	0.56	0.45	0.46
	1	0.75	0.75	0.75	0.75	0.60	0.60	1.00	0.80	0.80
25^th	0	0.96	0.99	1.03	1.42	1.45	1.48	1.48	1.46	1.44
	0.25	0.92	0.91	0.93	0.24	0.21	0.18	0.27	0.23	0.20
	0.50	0.90	0.82	0.83	0.25	0.20	0.17	0.28	0.24	0.20
	0.75	0.83	0.83	0.83	0.45	0.36	0.29	0.55	0.43	0.35
	1	0.75	0.75	0.75	0.60	0.60	0.50	0.80	0.80	0.67
50^th	0	0.93	0.96	1.00	1.72	1.78	1.86	1.85	1.86	1.87
	0.25	0.95	0.94	0.95	0.28	0.25	0.22	0.30	0.27	0.23
	0.50	0.83	0.85	0.86	0.22	0.19	0.16	0.27	0.22	0.18
	0.75	0.83	0.71	0.86	0.36	0.29	0.29	0.43	0.41	0.33
	1	0.75	0.75	0.80	0.60	0.50	0.57	0.80	0.67	0.71
75^th	0	0.97	1.00	1.04	1.42	1.45	1.48	1.46	1.44	1.43
	0.25	0.98	0.98	0.99	0.35	0.32	0.29	0.36	0.32	0.29
	0.50	0.80	0.81	0.83	0.21	0.18	0.15	0.27	0.22	0.18
	0.75	0.71	0.86	0.75	0.31	0.30	0.23	0.44	0.35	0.31
	1	0.75	0.80	0.80	0.50	0.57	0.44	0.67	0.71	0.56
95th	0	1.04	1.06	1.09	0.79	0.76	0.74	0.76	0.72	0.68
	0.25	1.01	1.02	1.05	0.54	0.51	0.48	0.54	0.49	0.46
	0.50	0.85	0.86	0.84	0.21	0.18	0.15	0.25	0.21	0.17
	0.75	0.75	0.88	0.78	0.27	0.26	0.20	0.36	0.30	0.26
	1	0.80	0.80	0.80	0.50	0.44	0.36	0.63	0.56	0.45

Note also that for $\delta =0,$ a ratio above 1 is advantageous for the chart in the numerator, while for $\delta >0$ it is better for the chart in the numerator to have a ratio below 1. Since the control limit adjustments considered in this paper provide a minimum in-control performance, we consider the out-of-control performance to be most important in the comparison.

From Table 5 we observe that the out-of-control performance of the EWMA and CUSUM charts designed to detect small shifts is very similar, with many ratios close or equal to 1. The CUSUM performs slightly better (ratios of 0.84 to 0.91) for detecting shifts of $\delta =0.25$ when the mean is substantially overestimated (95^th percentile of Z). The in-control values differ the most when the mean is estimated more accurately (closer to 50^th percentile of Z).

The Shewhart control chart is only able to outperform the CUSUM and EWMA schemes for the detection of large shifts ($\delta =1$). For other sizes of shifts, the out-of-control CARL values are larger than these of the CUSUM and EWMA scheme for almost any combination of Q and Z.

The comparison when the CUSUM and EWMA charts are designed for medium or large shifts are quite similar, as can be observed from Tables 6 and 7. As can be seen, the CUSUM chart outperforms the EWMA and Shewhart in terms of out-of-control performance on almost any combination and shift size. For the in-control situation the Shewhart chart has larger in-control CARL values for more severe over- and underestimation of the mean (5^th and 95^th percentile of Z). However, as stated earlier, the control limit adjustments considered here already provide a good in-control performance, such that the faster out-of-control detection is probably more valuable. From Tables 6 and 7, it can also be seen that the Shewhart control chart is far behind in terms of out-of-control detection speed compared to both CUSUM and EWMA schemes.

Discussion

We find that CUSUM charts yield faster detection of sustained shifts than Shewhart and EWMA charts when their designs are adjusted conform the exceedance probability criterion. This result holds regardless of whether the CUSUM and EWMA are designed for small, medium or large shifts. Zwetsloot and Woodall (2017) found that, without adjustment, the EWMA yields lower in-control CARL values than the CUSUM. Because of these lower CARL values, a larger adjustment is required for the EWMA chart to provide the desired in-control performance. This leads to larger out-of-control CARL values for the adjusted charts. Since the in-control performance is already sufficient due to the adjustments according to the exceedance probability criterion, this out-of-control performance is most important for the comparison.

The differences between the CUSUM and EWMA control charts are not very large. On the other hand, the Shewhart control chart is not able to compete with these two schemes in general, except for especially large shifts when the CUSUM and EWMA are designed for small shifts. This has to do with the fact that we are considering sustained shifts in this paper. The EWMA and CUSUM are especially suitable to detect these kind of shifts, while the Shewhart chart is memoryless (i.e., does not use information from previous Phase II observations). Of course, countless many other types of issues are thinkable in practice, but in this paper we focus on the detection of sustained shifts.

Concluding remarks

In this paper we compare the performance of Shewhart, CUSUM and EWMA control charts when they are adjusted for parameter estimation. The adjustments are made to provide a specified minimum in-control performance with a specified probability. This is in line with adjustments advocated recently by many authors, such as Jones and Steiner (2012), Gandy and Kvaløy (2013), and Saleh, Mahmoud, Keefe, et al. (2015), Saleh, Mahmoud, Jones-Farmer, et al. (2015), in order to reduce the risk of large false alarm rates when parameters are estimated.

We find that CUSUM control charts provide the fastest out-of-control detection of sustained shifts for almost all shift sizes and estimation errors. Since the control limit adjustments are designed to control the in-control performance, the out-of-control detection speed is considered the most important aspect in the comparison. The EWMA control chart is only slightly behind the CUSUM, while the Shewhart control chart is only able to compete with the other two schemes for very large shifts.

Additional information

Notes on contributors

Mandla D. Diko

Mandla D. Diko obtained his MSc in Statistics from the University of Pretoria. He is currently working as a PhD student in the Department of Operations Management of the University of Amsterdam, the Netherlands. His research interests are in Statistical Process Quality Control.

Rob Goedhart

Rob Goedhart is an Assistant Professor in the Department of Operations Management and senior consultant at the Institute for Business and Industrial Statistics of the University of Amsterdam, the Netherlands. His current research topic is control charting techniques with estimated parameters.

Ronald J. M. M. Does

Ronald J. M. M. Does is a Professor of Industrial Statistics at the University of Amsterdam, Director of the Institute for Business and Industrial Statistics and Head of the Department of Operations Management at the University of Amsterdam. He is an associate editor of the Journal of Quality Technology and Quality Engineering. He is the recipient of the Hunter Award (2008), Shewhart Medal (2019), and Box Medal (2019). He is a fellow of the American Statistical Association, a fellow of the American Society for Quality, an Academician of the International Academy for Quality, and an elected member of the International Statistical Institute. His current research activities include the design of control charts for nonstandard situations, healthcare engineering, and operations management methods.