IMPROVING MULTI-CAMERA ACTIVITY RECOGNITION BY EMPLOYING NEURAL NETWORK BASED READJUSTMENT

Figures & data

FIGURE 1 Sequences from our industrial environment dataset. Object tracking as well as activity recognition is extremely challenging due to occlusions, low resolution, and high intraclass and low interclass variance. The first two rows depict two different activities that are executed during the production cycle: their resemblance is so high, that they would be difficult to distinguish even for the human eye; the third row shows some example frames of occlusions, outliers, sparks, abnormalities, etc. (Figure is provided in color online.)

FIGURE 2 HMM-based fusion approaches for streams. Symbols s and o stand for states and observations, respectively. The first index indicates the stream and the second the time. (Figure is provided in color online.)

FIGURE 3 Schematic overview: The neural network-based rectification mechanism is examined under two different approaches (corresponding to the green and red paths, respectively). The green approach rectifies the fused result produced by the fused HMM, whereas the red one performs streamwise rectification and in the sequel the rectified streams are fused (RDFHMM). (Figure is provided in color online.)

FIGURE 4 Depiction of workcell together with the position of the cameras and racks #1–5. (Figure is provided in color online.)

TABLE 1 Results Obtained from Dataset-1 and Dataset-2 Using (1) Individual HMMs to Model Information from Stream 1 (HMM1); (2) Individual HMMs to Model Information from Stream 2 (HMM2); (3) State-Synchronous HMMs (SYNC); (4) Parallel HMMs (PARAL); and (5) Multistream-Fused HMMs (MULTI) with (a) Gaussian and (b) Student's -distribution as Observation Likelihood

		Dataset-1			Dataset-2
		Recall	Precision	F-measure	Recall	Precision	F-measure
HMM1	Gauss	82.4%	78.1%	80.2%	51.8%	49.8%	50.8%
	Student-t	85.0%	80.2%	82.5%	63.2%	58.6%	60.8%
HMM2	Gauss	86.4%	82.7%	84.5%	53.7%	49.9%	51.7%
	Student-t	88.6%	84.1%	86.3%	56.3%	53.4%	54.8%
SYNC	Gauss	70.3%	62.1%	65.9%	53.9%	49.5%	51.6%
	Student-t	73.3%	64.1%	68.4%	57.1%	53.9%	55.5%
PARAL	Gauss	83.3%	78.9%	81.0%	54.9%	49.9%	52.3%
	Student-t	84.5%	80.8%	82.6%	59.3%	54.3%	56.7%
MULTI	Gauss	89.1%	86.5%	87.8%	59.1%	52.5%	55.6%
	Student-t	92.1%	89.8%	90.9%	67.6%	59.6%	63.3%

FIGURE 5 Confusion matrices from dataset-1 for (a) individual HMM for camera 1, (b) individual HMM for camera 2 and (c) multistream-fused HMM, using Student's t -distribution. (Figure is provided in color online.)

TABLE 2 Results Obtained from Dataset-1 and Dataset-2 After Applying the Rectification Mechanism (RM) Using (1) Individual HMMs to Model Information from Stream 1 (HMM1); (2) Individual HMMs to Model Information from Stream 2 (HMM2); (3) State-Synchronous HMMs (SYNC); (4) Parallel HMMs (PARAL); (5)Multistream-Fused HMMs (MULTI); and (6) Rectification-Driven Fused HMM (RDFHMM) with (a) Gaussian and (b) Student's -distribution as Observation Likelihood

		Dataset-1			Dataset-2
		Recall	Precision	F-measure	Recall	Precision	F-measure
HMM1 + RM	Gauss	89.6%	83.9%	86.7%	72.0%	62.4%	66.9%
	Student-t	90.2%	86.8%	88.5%	77.4%	66.7%	71.7%
HMM2 + RM	Gauss	91.0%	86.4%	88.6%	69.5%	61.3%	65.1%
	Student-t	93.5%	90.7%	92.1%	73.2%	62.3%	67.3%
SYNC + RM	Gauss	80.7%	77.1%	78.9%	70.4%	65.2%	67.7%
	Student-t	80.1%	74.8%	77.4%	72.5%	66.4%	69.3%
PARAL + RM	Gauss	90.5%	87.2%	88.8%	73.9%	67.7%	70.7%
	Student-t	90.3%	87.6%	88.9%	74.3%	66.3%	70.1%
MULTI + RM	Gauss	93.7%	91.2%	92.4%	73.1%	68.5%	70.7%
	Student-t	94.2%	91.8%	92.9%	78.9%	72.3%	75.4%
RDFHMM	Gauss	93.8%	91.3%	92.5%	75.7%	70.4%	72.9%
	Student-t	95.0%	93.2%	94.1%	79.8%	77.3%	78.5%

FIGURE 6 Classification error % with and without the rectification mechanism for all experimental setups: 1. HMM1-Gauss, 2. HMM1-Student-t, 3. HMM2-Gauss, 4. HMM2-Student-t, 5. SYNC-Student-t, 6. SYNC-Student-t, 7. PARAL-Gauss, 8. PARAL-Student-t, 9. MULTI-Gauss, 10. MULTI-Student-t, 11. RDFHMM-Gauss, 12. RDFHMM-Student-t, (11 & 12 have no corresponding nonrectified setup). (Figure is provided in color online.)

FIGURE 7 Improvement ratio % in terms of error for all experimental setups: 1. HMM1-Gauss, 2. HMM1-Student-t, 3. HMM2-Gauss, 4. HMM2-Student-t, 5. SYNC-Gauss, 6. SYNC-Student-t, 7. PARAL-Gauss, 8. PARAL-Student-t, 9. MULTI-Gauss, 10. MULTI-Student-t, (above mentioned 11 & 12 have no corresponding non-rectified set-up therefore no improvement rate can be calculated). (Figure is provided in color online.)