Providing Dual Awareness using Multimodal Cues for Collaborative Manipulation in Virtual Environments

Figures & data

Figure 1. Collaborative manipulation in virtual environments.

Figure 2. Block diagram of the dual awareness framework.

Figure 3. Interaction states of collaborative manipulation (a) hand’s states, (b) object’s states.

Figure 4. Collaborative manipulation model.

Figure 5. Substitution transitions in collaborative manipulation model (a) attach, (b) select, and (c) release.

Figure 6. Example of CPN (a) when User2 selects Cube2 with the left hand, (b) when User1 and User2 select Cube2 together.

Figure 7. Example of dominant and subsidiary feedback for two collaborators.

Figure 8. Experimental system setup. The display used was a 40″ three-dimensional monitor. We used the 3GearSystems camera as a depth camera to recognize the pose of the participant’s hands and to track the movement of the participant’s hands. To receive feedback on the collaborator’s interaction, participants wear a vibrating haptic device on their left index finger. Participants’ experimental systems in separate locations are networked with each other and collaborate by sharing the same virtual space (top).

Figure 9. Experimental tasks (a) Task 1: Move together (1. Two users grasp together, 2–3. Two users move together, 4. Two users stop together), (b) Task 2: Handover (1. The giver grasps, 2. The receiver grasps, 3. The giver releases, 4. The receiver moves).

Table 1. Statements in the questionnaire (five-point Likert items).

No.	Statements
1	I thought this feedback system was easy to use.
2	During the task, I found this feedback difficult.
3	I preferred to use this feedback.
4	I was disturbed by the provision of this feedback.
5	During the task, I was satisfied that this feedback.
6	During the task, I do not need this feedback.
7	During the task, I thought this feedback was clear.
8	During the task, it was hard for me to detect this feedback.
9	The task was completed quickly and accurately because of this feedback.
10	Feedback did not affect the performance of the task.
11	I felt like my hands were moving the virtual object.
12	It was hard to tell if my hands were moving the virtual object properly.
13	I felt that I was manipulating the object with the other user.
14	I was confused whether I was working with the other user.
15	When I used this feedback, I could perceive my action.
16	I could not understand what I was doing during the task.
17	When I used this feedback, I could perceive others’ actions.
18	I could not understand what the other user was doing during the task.
19	When I used this feedback, I could coordinate my actions.
20	I did not know what I should do to collaborate with others.

Figure 10. Moving together task: Ratio of collaborative manipulation to (a) cooperative distance, (b) cooperative time, (c) single distance, and (d) single time (x: mean, y: ratio from 0 to 1, *^, ***: pairwise significant difference).

Figure 11. Handover task: (a) completion time (x: mean, y: second), (b) handover time (x: mean, y: second), and (c) failure count (x: mean, y: count) (*^, ***: pairwise significant difference).

Figure 12. Average questionnaire scores (x: mean, y: normalized score from 0 to 10) (***: significant difference).

Figure 13. Number of votes of the most proper feedback (a) 10 items (multiple selections possible), (b) 2 tasks.