Activity scenarios simulation by discovering knowledge through activities of daily living datasets

Figures & data

Figure 1. Example context related to human household activities, behaviour, or kitchen usage patterns.

Table 1. Consistent features observed among the real-world activity datasets.

Features	CLAD [18]	DAHLIA [19]	Charades [4]	Toyota SmartHome [5]	HOMAGE [7]	NTU RGB+D [6]
Rich context	No	No	Yes	Yes	Yes	Yes
Multiple rooms	No	No	Yes	Yes	Yes	Yes
Diverse participants	No	No	Yes	Yes	Yes	Yes
Spontaneous acting	No	No	Yes	Yes	Yes	Yes
Low framing	No	No	No	Yes	Yes	Yes
Cross-view	No	No	No	Yes	Yes	Yes

Table 2. Consistent features observed among the synthetic activity datasets.

Features	VirtualHome [21]	ElderSim [22]	SIMS4ACTION [23]	iGibson [24]	AI2-THOR [25]
Rich context	Yes	Yes	Yes	Yes	Yes
Interaction with objects	Yes	Yes	Yes	Yes	Yes
Multiple camera views	Yes	Yes	Yes	Yes	Yes
Action/activity support	Yes	Yes	Yes	Yes	Yes
Multi-agent support	Yes	Yes	Yes	Yes	Yes
Scene manipulation	Yes	No	Yes	Yes	Yes
Realistic motion	No	Yes	No	No	No
Application diversity	Yes	Yes	Yes	Yes	Yes

Table 3. Applied synthetic activity datasets in AI research tasks.

Datasets	Research area	Specific research tasks
iGibson [24]	Visual navigation	Point navigation, Object navigation
	Embodied question answering	Multi-target question answering (MQA), Interactive question answering (IQA)
AI2-THOR [25]	Object detection	Object detection in computer vision
	Visual navigation	Vision and language navigation (VLN)
	Object relationships	Study of object relationships
VirtualHome [21], ElderSim [22], SIM4ACTION [23]	Human activity recognition	Recognizing normal or abnormal human actions and activities

Figure 2. Overall workflow of activity scenario preparation and simulation task.

Figure 3. Part of the designed class diagram of ontology [8].

Table 4. Selected categories of daily living activities and their general description.

No.	Category	General description with activity examples
1	BedTimeSleep	Initiating a long sleep episode. Activities include going to sleep (e.g. Go to sleep) and winding down before bedtime (e.g. Read bedtime story)
2	HygieneStyling	Engaging in self-care activities. Activities encompass tasks such as ensuring personal hygiene (e.g. Brush teeth, Hand washing, etc.) at any time of the day promoting personal cleanliness and well-being
3	FoodPreparation	Involvement in meal preparation. Activities entail cooking tasks (e.g. Cook fried bread, Make cold cereal, etc.) for breakfast, ensuring that nutritious and tasty meals are ready to be consumed
4	EatingDrinkng	Consumption of food and beverages at any time of the day. Activities include tasks (e.g. Eat bread slice, Have morning beverage, etc.) maintaining a regular eating activity
5	HouseArrangement	Organizing and arranging the home environment. Activities consist of tasks (e.g. Put groceries in fridge, etc.), ensuring food storage, and setting up tables for meals (e.g. Set up table, etc.), creating an organized living space
6	HouseCleaning	Cleaning and maintaining cleanliness in living spaces. Activities encompass tasks (e.g. Clean desk, Clean sofa, etc.), ensuring a hygienic environment
7	Work	Engaging in work or study-related tasks. Activities involve completing assignments (e.g. Do homework on paper, etc.), and doing work (e.g. Do work on computer, etc.), contributing to personal and professional growth
8	PhysicalActivity	Participating in physical exercises. Activities consist of performing exercises (e.g. Squats, Straddle splits, etc.), promoting physical fitness and well-being
9	SocialInteraction	Interaction with pets for companionship/care. The activity involves petting or spending time with a cat (e.g. Pet cat), fostering a bond with an animal
10	Leisure	Enjoying leisure activities at any time. Activities encompass (e.g. Social media checks, Watch television), providing relaxation and entertainment
11	Other	Miscellaneous activities do not fit into the above categories. These can vary widely and are not explicitly categorized within the defined activities

Table 5. Selected abnormal activities occurred in home environments.

Activity name	General description
Fall backward while walking and turning	While turning with a cat in hand, the person falls backward due to a misstep, crossing their left leg in front of the right, leading to a loss of balance
Fall in bathroom	Upon finishing on the toilet, the person stands up without using arm support, relying on balance and strength, leading to a fall during the transition from sitting to standing
Fall while getting up or rising	A fall happens while trying to get up from the bed, attributed to a loss of support, possibly due to weak muscles, poor balance, or mobility issues, making the transition from sitting or lying to standing challenging
Fall while initiation of walking	A fall at the start of walking may signal balance problems linked to foot or leg issues (arthritis or deformities) and medication side effects, increasing the risk due to dizziness or impaired balance
Fall while preparing meal	A fall occurs while cooking and serving food when both hands are occupied with ingredients, affecting balance, and distractions like conversations or timers increase the risk
Fall while sitting down or lowering	A fall often occur when sitting down or lowering onto the bed, typically due to weakness, instability, or dizziness
Fall while standing and reaching	A person falls while standing and reaching for an object in a high place
Fall while standing and turning	The person falls due to improper weight shifting while standing and turning, a movement pattern, where the body rotates while the feet remain stationary
Fall while standing at somewhere height	A fall occurs if a person steps up to a height and stands to work at height
Fall while walking forward	The person falls while walking forward, hurrying to leave the house, experiencing a fall due to incorrect weight shifting during the movement

Figure 4. Static camera settings installed in each room’s corner.

Figure 5. Alignment of objects and actions with activity categories – Abnormal, EatingDrinking, and HouseCleaning.

Figure 6. Distribution of activities measured in the number of action steps in scenarios.

Figure 7. Some simulated daily activities in the virtual homes, exploring various scenarios and interactions.

Figure 8. Some simulated abnormal activities in the virtual homes, exploring unexpected risky situations.

Figure 9. Illustration of feature extraction: construction of virtual ground point (VGP) and posture deformities are estimated by calculating angles involving three points in the upper body of skeleton image.

Table 6. Performance evaluation on some simulated abnormal activity data.

		d			θ
Activity name	Scene	PRC	RCL	ACC	PRC	RCL	ACC
Fall in bathroom	1	100	65.62	96.82	100	64.06	96.68
Fall in bathroom	3	100	66.66	96.59	100	73.01	97.24
Fall in bathroom	5	100	66.66	97.61	100	73.01	98.07
Fall in bathroom	6	100	66.12	96.71	100	74.19	97.49
Fall while getting up or rising	1	100	66.66	96.79	100	71.42	97.25
Fall while getting up or rising	2	100	67.74	96.87	100	74.19	97.49
Fall while getting up or rising	3	100	65.62	96.78	100	73.43	97.51
Fall while getting up or rising	4	100	67.74	96.85	100	72.58	97.32
Fall while getting up or rising	5	100	63.63	96.29	100	53.03	95.20
Fall while preparing meal	1	77.08	100	97.33	85.36	94.59	98.06
Fall while preparing meal	2	52.38	100	95.20	62.85	100	96.88
Fall while preparing meal	3	100	80.76	97.75	100	69.23	96.41
Fall while preparing meal	4	100	73.21	96.75	100	83.92	98.05
Fall while preparing meal	5	52.38	100	95.43	46.80	100	94.29
Fall while preparing meal	6	83.33	100	98.16	92.10	100	99.21
Fall while standing and turning	1	51.22	100	96.77	61.76	100	97.90
Fall while standing and turning	3	71.42	100	98.4	68.18	100	98.13
Fall while standing and turning	3	100	75	98.21	100	83.92	98.85
Fall while standing and turning	4	63.41	100	98.13	60.46	100	97.88

Tayyub J, Hawasly M, Hogg DC, et al. CLAD: a complex and long activities dataset with rich crowdsourced annotations; ArXiv, abs/1709.03456; 2017.

 Google Scholar

Vaquette G, Orcesi A, Lucat L, et al. The daily home life activity dataset: a high semantic activity dataset for online recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017); 2017. p. 497–504.

 Google Scholar

Sigurdsson GA, Varol G, Wang X, et al. Hollywood in homes: crowdsourcing data collection for activity understanding. In: Leibe B, Matas J, Sebe N, et al., editors. Computer vision – ECCV; Lecture notes in computer science, Vol. 9905. Cham: Springer. doi:10.1007/978-3-319-46448-0_31

 Google Scholar

Das S, Dai R, Koperski M, et al. Toyota smarthome: real-world activities of daily living. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV); Seoul, Korea (South); 2019. p. 833–842. doi:10.1109/ICCV.2019.00092

 Google Scholar

Rai N, Chen H, Ji J, et al. Home action genome: cooperative compositional action understanding. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Nashville, TN; 2021. p. 11179–11188. doi:10.1109/CVPR46437.2021.01103

 Google Scholar

Shahroudy A, Liu J, Ng T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. p. 1010–1019.

 Google Scholar

Puig X, Ra KK, Boben M, et al. VirtualHome: simulating household activities via programs. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2018. p. 8494–8502.

 Google Scholar

Hwang H, Jang C, Park G, et al. ElderSim: a synthetic data generation platform for human action recognition in eldercare applications. IEEE Access. 2023;11:9279–9294. doi:10.1109/ACCESS.2021.3051842

 Web of Science ®Google Scholar

Roitberg A, Schneider D, Djamal A, et al. Let’s play for action: recognizing activities of daily living by learning from life simulation video games. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Prague, Czech Republic; 2021. p. 8563–8569. doi:10.1109/IROS51168.2021.9636381

 Google Scholar

Li C, Martín-Martín R, Lingelbach M, et al. iGibson 2.0: object-centric simulation for robot learning of everyday household tasks; ArXiv, abs/2108.03272; 2021.

 Google Scholar

Kolve E, Mottaghi R, Han W, et al. AI2-THOR: an interactive 3D Environment for visual AI; ArXiv, abs/1712.05474; 2017.

 Google Scholar

Egami S, Ugai T, Oono M, et al. Synthesizing event-centric knowledge graphs of daily activities using virtual space. IEEE Access. 2023;11:23857–23873. doi:10.1109/ACCESS.2023.3253807

 Web of Science ®Google Scholar