Smart Control of Home Appliances Using Hand Gesture Recognition in an IoT-Enabled System

Figures & data

Figure 1. Positions of 21 hand key points.

Table 1. Four elements of Petri net.

Elements	Notations
Place
Transition	or
Arc
Token

Figure 2. System operation flowchart.

Figure 3. Different angles of gesture 1.

Table 2. Parameters of the fingers.

Parameters	Index finger	Middle finger	Ring finger	Little finger	Thumb
Xa	Key point 0	Key point 0	Key point 0	Key point 0	Key point 0
Ya	Key point 6	Key point 10	Key point 14	Key point 18	Key point 2
Xb	Key point 7	Key point 11	Key point 15	Key point 19	Key point 3
Yb	Key point 8	Key point 12	Key point 16	Key point 20	Key point 4
x1	X coordinate of key point 0	X coordinate of key point 0	X coordinate of key point 0	X coordinate of key point 0	X coordinate of key point 0
x2	X coordinate of key point 6	X coordinate of key point 10	X coordinate of key point 14	X coordinate of key point 18	X coordinate of key point 2
y1	Y coordinate of key point 0	Y coordinate of key point 0	Y coordinate of key point 0	Y coordinate of key point 0	Y coordinate of key point 0
y2	Y coordinate of key point 6	Y coordinate of key point 10	Y coordinate of key point 14	Y coordinate of key point 18	Y coordinate of key point 2
x3	X coordinate of key point 7	X coordinate of key point 11	X coordinate of key point 15	X coordinate of key point 19	X coordinate of key point 3
x4	X coordinate of key point 8	X coordinate of key point 12	X coordinate of key point 16	X coordinate of key point 20	X coordinate of key point 4
y3	Y coordinate of key point 7	Y coordinate of key point 11	Y coordinate of key point 15	Y coordinate of key point 19	Y coordinate of key point 3
y4	Y coordinate of key point 8	Y coordinate of key point 12	Y coordinate of key point 16	Y coordinate of key point 20	Y coordinate of key point 4
L1	Vectors for key points 0 and 6	Vectors for key points 0 and 10	Vectors for key points 0 and 14	Vectors for key points 0 and 18	Vectors for key points 0 and 2
L2	Vectors for key points 7 and 8	Vectors for key points11 and 12	Vectors for key points 15 and 16	Vectors for key points 19 and 20	Vectors for key points 3 and 4
A	The angle between L1 and L2 vectors	The angle between L1 and L2 vectors	The angle between L1 and L2 vectors	The angle between L1 and L2 vectors	The angle between L1 and L2 vectors

Table 3. Ten gestures.

Numbers	Gestures	Left-hand finger states (thumb, index finger, middle finger, ring finger, little finger)
0		(0, 0, 0, 0, 0)
1		(0, 1, 0, 0, 0)
2		(0, 1, 1, 0, 0)
3		(0, 1, 1, 1, 0)
4		(0, 1, 1, 1, 1)
5		(1, 1, 1, 1, 1)
6		(1, 0, 0, 0, 1)
7		(1, 1, 0, 0, 0)
8		(1, 1, 1, 0, 0)
9		(1, 1, 1, 1, 0)

Figure 4. Petri net model of system design process.

Table 4. Interpretation of places.

Place	Interpretation	Place	Interpretation
p1	System preparation starts	p12	Take the intersection line of four key points to calculate the angle
p2	Prepare to capture the image and adjust image size	p13	Calculated finger angle > 40 degrees
P3	Execute palm detector	p14	Calculated finger angle < 40 degrees
p4	Confirm the detection of hand	p15	Confirm finger angle
p5	Failure in confirming palm’s position in the first attempt	p16	Compare gesture definition sources
p6	Confirm hand model	p17	Matching with the corresponding command
p7	Confirm palm capture	p18	Matching with Wi-Fi successfully
p8	Confirm palm’s presence in image	p19	Compare gesture to command data
p9	Confirm hand position	p20	Confirm the status of home appliances
p10	The value of hand model on a confidence level	p21	The end of system design process
p11	Confirm image confidence level >0.5

Table 5. Interpretation of transitions.

Transition	Interpretation	Transition	Interpretation
t1	Start	t15	Output 21 hand key point images
t2	Smartphone lens captures image and adjusts image size	t16	Finger bent > 40 degrees (Yes)
t3	First time using palm detector (Yes)	t17	Finger bent > 40 degrees (No)
t4	First time using palm detector (No)	t18	Finger bent
t5	Palm detected	t19	Finger straightening
t6	Hand is found (No)	t20	Display gesture recognition result
t7	Hand is found (Yes)	t21	Finger has the corresponding finger (Yes)
t8	Palm presents in the image (No)	t22	Finger has the corresponding finger (No)
t9	Palm present in the image (Yes)	t23	Wi-Fi connection (Yes)
t10	Crop hand area extending from palm	t24	Wi-Fi connection (No)
t11	Crop the image according to the previous hand area	t25	Send the command to ESP8266
t12	Build hand model	t26	Control home appliances
t13	Hand in image confidence level > 0.5 (Yes)	t27	The end
t14	Hand in image confidence level > 0.5 (No)

Figure 5. Semantical analysis.

Figure 6. Gesture recognition - Light off.

Figure 7. Gesture recognition - Light on.

Figure 8. Gesture recognition –red RGB light bar.

Figure 9. Gesture recognition –blue RGB light bar.

Figure 10. Red RGB light bar (normal).

Figure 11. Red RGB light bar (slightly dim).

Figure 12. Red RGB light bar (slightly bright).

Table 6. Recognition results - average precision of 10 gestures.

Gesture	TP	FP	Precision	Average precision
0	50	0	1	0.988
1	50	0	1
2	50	0	1
3	50	0	1
4	50	0	1
5	50	0	1
6	47	3	0.94
7	50	0	1
8	50	0	1
9	47	3	0.94

Table 7. Training results – average recall of 10 gestures.

Gesture	TP	FN	Recall	Average Recall
0	50	3	0.943	0.9767
1	50	0	1
2	50	0	1
3	50	0	1
4	50	0	1
5	50	0	1
6	47	0	1
7	50	0	1
8	50	3	0.943
9	47	0	1

Table 8. Comparison of the precision with different methods for gestures in ASL.

Methods Gestures	Proposed (%)	Shen, X. et al. (%)	Chen, Y.-C. (%)
0	100	95.27	100
1	100	91.75	98
2	100	89.25	95
3	90	90	98
4	100	100	95
5	100	100	100
6	90	77	91
7	90	84.25	91
8	90	76.25	83
9	100	74	96
Average	96	86.94	94.4

Table 9. Comparison of this study with other methods.

Methods Metrics	Proposed	Yu, C.-R.	Shih, W.-H.	Padhy, S.	Chen, X., et al.
Hardware	Smartphone, ESP8266 microcontroller chip	USB 3.0 bus; At least 4GB RAM webcam or smartphone	Wearable smart gloves, Arduino YUN	Surface electromyography (sEMG), Multilinear singular value decomposition (MLSVD)	Surface electromyography (sEMG)-based, CNN+LSTM (long short-term memory)
Software	MediaPipe Hand Android Studio Thonny 3.7.5	Windows 10 operating system, Python 3.6	Arduino operating system	Tensor-based approach, Dictionary Learning (DL)	Transfer learning (TL) strategy, CNN-based source network
Number of gestures	10	7	4	10	20
Control of the actions	1. All home appliances switch on/off 2. Control of light color	1. PPT files zoom in and zoom out 2. Slides playing, next page, and close	1. Power switch of light bulb, color and brightness setting 2. Power setting of air conditioner, temperature, and wind speed	Upper limb motion classification Biomedical engineering systems	Myoelectric control systems Biomedical engineering systems
Distance	2.5m	1.9m	2.0m	0.0m	0.0m
The success rate when the light illumination is 2.95 lux	32%	22%	25%	28%	29%
Response time(sec.)	0.62	0.87	0.91	0.90	0.89
Precision %	98.80	90.65	91.71	92.12	93.32
Recall %	97.67	89.11	90.16	91.23	92.77
System validation tool	WoPeD	N/A	N/A	N/A	N/A

N/A: Unavailable