Variational Autoencoder for Classification and Regression for Out-of-Distribution Detection in Learning-Enabled Cyber-Physical Systems

Figures & data

Figure 1. Variational autoencoder for classification and regression model. Note that “Predictor” can be a classification or regression network.

Table 1. False alarms for detecting OOD data in N-BaIot dataset.

Types	NCM	$N,\omega ,\tau$	False positives	False negatives
Novelty detection	$A_{\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\sigma ,\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\sigma ,\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
Intrusion detection	$A_{\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\sigma ,\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$

Table 2. Classification accuracy and AUROC for detecting OOD data in gearbox dataset.

NCM	Single example	Multiple examples	Accuracy(ID/OOD)
$A_{\mathrm{R}\mathrm{C}}$	$0.573$	$0.581$	$99.1\mathrm{\%}$/$60.5\mathrm{\%}$
$A_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$	$0.690$	$0.698$

Table 3. AUROC for detecting OOD data in MNIST dataset.

Types	NCM	Single example	Multiple examples
Covariate shift	$A_{\mathrm{R}\mathrm{C}}$	$1.000$	$1.000$
	$A_{\sigma ,\mathrm{R}\mathrm{C}}$	$1.000$	$1.000$
	$A_{\mathrm{V}\mathrm{A}\mathrm{E}}$	$1.000$	$1.000$
Label shift	$A_{\mathrm{R}\mathrm{C}}$	$0.852$	$0.878$
	$A_{\sigma ,\mathrm{R}\mathrm{C}}$	$0.879$	$0.879$
	$A_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$	$0.874$	$0.870$
	$A_{\sigma ,\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$	$0.803$	$0.805$
	$A_{\mathrm{V}\mathrm{A}\mathrm{E}}$	$0.704$	$0.723$

Table 4. VAE for classification architecture in MNIST.

Module	Layer	Number of neurons (filters)	Activation	Kernel size	stride
Encoder	Convolutional	32	LeakyReLU	$4\times 4$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	64	LeakyReLU	$4\times 4$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	128	LeakyReLU	$4\times 4$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	256	LeakyReLU	$4\times 4$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Fully Connected	512	LeakyReLU	N/A	N/A
	BatchNorm	N/A	N/A	N/A	N/A
	Fully Connected $\times$ 2	20	Sigmoid	N/A	N/A
Latent generator	Fully Connected	20	N/A	N/A	N/A

Figure 2. Histogram of ground-truth distance in training dataset.

Figure 3. The 2D embedded features visualized by t-SNE.

Table 5. VAE for regression architecture in AEBS.

Module	Layer	Number of neurons (filters)	Activation	Kernel size	stride
Encoder	Convolutional	32	ELU	$5\times 5$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	64	ELU	$5\times 5$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	128	ELU	$5\times 5$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Convolutional	256	ELU	$5\times 5$	$2$
	BatchNorm	N/A	N/A	N/A	N/A
	Fully Connected	1568	ELU	N/A	N/A
	BatchNorm	N/A	N/A	N/A	N/A
	Fully Connected $\times$ 2	1024	Sigmoid	N/A	N/A
Latent generator	Fully Connected	256	ELU	N/A	N/A
	BatchNorm	N/A	N/A	N/A	N/A
	Fully Connected	1024	N/A	N/A	N/A

Figure 4. An episode with in-distribution data in AEBS (detector parameter: $N=10$, $\omega =4$, $\tau =40$).

Figure 5. An episode with OOD data caused by covariate shift in AEBS (detector parameter: $N=10$, $\omega =4$, $\tau =40$).

Table 6. False alarms for detecting OOD data in AEBS.

Types	NCM	$N,\omega ,\tau$	False positives	False negatives
Covariate shift	$A_{\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\mathrm{R}\mathrm{C}-\mathrm{L}\mathrm{R}\mathrm{P}}$	$5,2,50$	$4/50$	$0/50$
		$10,4,40$	$3/50$	$0/50$
Label shift	$A_{\mathrm{R}\mathrm{C}}$	$5,2,50$	N/A	$13/50$
		$10,4,40$		$11/50$
	$A_{\mathrm{R}\mathrm{C}-\mathrm{L}\mathrm{R}\mathrm{P}}$	$5,2,50$	N/A	$3/50$
		$10,4,40$		$2/50$
Label concept shift	$A_{\mathrm{R}\mathrm{C}}$	$5,2,50$	$0/50$	$0/50$
		$10,4,40$	$0/50$	$0/50$
	$A_{\mathrm{R}\mathrm{C}-\mathrm{L}\mathrm{R}\mathrm{P}}$	$5,2,50$	$4/50$	$0/50$
		$10,4,40$	$3/50$	$0/50$

Figure 6. Histogram of ground-truth distance of training dataset that excludes data ranging from $15$ to $45$.

Figure 7. An episode with OOD data caused by label shift in AEBS (detector parameter: $N=10$, $\omega =4$, $\tau =40$).

Figure 8. An episode with OOD data caused by label concept shift in AEBS (detector parameter: $N=10$, $\omega =4$, $\tau =40$).

Figure 9. Execution times of proposed method.