Exploring the Potentials of Crowdsourcing for Gesture Data Collection

Figures & data

Table 1. The 12 upper-body gestures

One-hand gestures						Two-hand gestures
Come^1		Face		Hi		Baby		Big		Bus
Light		Sea		Sky		Snow		Take		Together

Subjects learned gestures from the official tutorial video performed by 6 signers.

Table 2. 2D gesture features redefined from 2D features (Kang et al., 2021)

Feature	Description	Abbr.
Gesture area	Area of the rectangle drawn by min, max x, y coordinates of both wrist joints	GA
Quantity of movement	Total length of gesture trajectory for every joint	QoM
Performance time	Gesture perform ance time measured in the number of frames	PT
Average gesture speed	Average of the movement speed of all joints	AS
Body posture variation	Sum of variance from the mean of each joint in all frames	BPV
Body posture density	Variation of body posture over the gesture area (= BPV ÷ GA)	BPD

Table 3. Details of the datasets collected in the study

Type	Environment	Additional instruction	Ways of performing gestures	Participants	Abbreviation	# of Samples
Training	Laboratory (LAB)	Styling Words (SW)	Watching a tutorial video, then immediately performing the gestures	P1–P20	TR-LAB-SW	4800
		Common (CO, without SW)		P1–P20	TR-LAB-CO	4800
	Crowdsourcing (CR)	SW		P21–P40	TR-CR-SW	4800
		CO		P21–P40	TR-CR-CO	4800
Testing	LAB	CO	Memorizing then recalling and performing the gestures later	P41–P50	TE-LAB	1200
	CR			P51–P60	TE-CR	1200

Figure 1. Examples of the crowdsourcing collection web page. (a) illustrates the process of collecting gesture videos through online. (b) shows the part of the guidelines page for collecting quality gesture videos. (c) shows a list of gestures with illustrations for participants to select gestures.

Figure 2. Examples of collected data in the LAB (left) and CR (right) datasets

Figure 3. Test accuracy of the four models according to the number of participants. The test results were evaluated with TE-CR data.

Figure 4. Boxplots of (a) average and (b) standard deviation for the six gesture features (QoM, PT, AS, BPV, GA, and BPD). Boxplots from left to right for each graph: TE-CR, TE-LAB, TR-CR-CO, TR-CR-SW, TR-LAB-CO, TR-LAB-SW (*p < 0.05, **p < 0.01, ***p < 0.001)

Kang, W., Jung, I.-T., Lee, D., & Hong, J.-H. (2021). Styling words: A simple and natural way to increase variability in training data collection for gesture recognition [Paper presentation]. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan. https://doi.org/10.1145/3411764.3445457

 Google Scholar