Effective human–object interaction recognition for edge devices in intelligent space

Figures & data

Figure 1. Overview of the proposed human–object interaction.

Figure 2. Human–object interaction class examples recognized by the proposed method.

Figure 3. Action class examples recognized by the proposed method.

Figure 4. Model accuracies of the proposed method: (a) bottle, (b) mobile phone, (c) book, (d) keyboard, and (e) actions (no interaction).

Table 1. Robustness experiment.

Recognition method	Accuracy (%)
Baseline	98.7
(a) Different subject	98.3
(b) Different camera angle	98.9
(c) Different camera frame size	98.5
(d) Different environment	98.4

Table 2. Robustness with various subjects.

Subjects	Accuracy (%)
Baseline (male, Japanese)	98.7
(a) Person A (male, Japanese)	98.2
(b) Person B (male, Japanese, wearing glasses)	98.7
(c) Person C (female, Japanese)	98.3
(d) Person D (female, Vietnamese)	97.8

Table 3. Robustness with various camera angles.

Angle	Accuracy (%)
Baseline $\left(0^{\circ }\right)$	98.7
(a) Angle A $\left(-{15}^{\circ }\right)$	98.3
(b) Angle B $\left(-{30}^{\circ }\right)$	99.2
(c) Angle C $\left(-{45}^{\circ }\right)$	98.9
(d) Angle D $\left({15}^{\circ }\right)$	99.1

Table 4. Robustness with various camera frame sizes.

Camera Frame Size	Accuracy $\left(\mathrm{\%}\right)$
Baseline ($640\times 480$ pixel)	98.7
(a) Size A ($640\times 640$ pixel)	98.3
(b) Size B ($1080\times 720$ pixel)	98.8
(c) Size C ($1280\times 960$ pixel)	99.0
(d) Size D ($1920\times 1080$ pixel)	98.1

Table 5. Robustness in various environments.

Environment	Accuracy (%)
Baseline (clean, narrow space)	98.7
(a) Environment A (messy, narrow space)	98.2
(b) Environment B (clean, wide space)	98.6
(c) Environment C (messy, wide space)	98.3

Table 6. Ablation study on the explanatory variables in the proposed method.

Recognition method	Accuracy (%)
Baseline (pose estimation only)	85.8
(a) Proposed w/o object features	87.4
(b) Proposed w/o $d_{\text{so }}$	93.8
(c) Proposed w/o $d_s$	94.4
(d) Proposed	95.7

Table 7. Comparison of accuracy on consideration of multi-camera complementarity.

Recognition method	Accuracy (%)
(a) One camera	87.4
(b) Two cameras	95.8

Table 8. Effect of accuracy on changing the size of training dataset.

Size of dataset (frame)	Accuracy (%)
(a) 100	98.7
(b) 1000	98.9
(c) 5000	98.3

Table 9. Effect of accuracy on changing the number of recognized interaction models.

Number of interaction models	Accuracy (%)
(a) One (bottle)	99.3
(b) Two (bottle, mobile phone)	98.7
(c) Three (bottle, mobile phone, book)	99.1
(d) Four (bottle, mobile phone, book, keyboard)	99.3

Table 10. Recognition speed experiment on various processor.

Processor	Frames-per-second
(a) CPU-only (Intel^® Core™ i7-5557U)	11.6
(b) CPU-only (Intel^® Core™ i7-1165G7)	20.4
(c) CPU (Intel^® Core™ i7-10700) GPU (GeForce RTX 3080 Ti)	166.7

Table A1. Comparison of accuracy on consideration of depth coordinates.

Recognition method	Accuracy (%)
(a) 2D recognition	87.4
(b) 3D recognition	92.4

Figure A1. Multi-person interaction recognition.