CLIP feature-based randomized control using images and text for multiple tasks and robots

Figures & data

Figure 1. Chair rearrangement task using CLIP feature-based randomized control. The text instruction is ‘place a green chair under the table.’

Figure 2. Overview of our control framework.

Figure 3. Simulation environment (the green dot indicates the target position of the handle). (a) drawer-close. (b) drawer-open. (c) door-close. (d) door-open. (e) window-close and (f) window-open.

Table 1. Hyperparameters used in the PPO algorithm.

Parameter	Value
Number of episodes	3.0$\times {10}^3$
Discount factor	0.99
Clipping parameter	0.2
Number of episodes for update	80
Learning rate for the critic network	3.0$\times {10}^{-4}$
Learning rate for the actor network	1.0$\times {10}^{-3}$

Table 2. Texts used in a multitask simulation.

Task	Variable	Text
drawer-close	$T_1$	close a drawer with a drawer handle
	$T_2$	open a drawer with a drawer handle
drawer-open	$T_1$	open a drawer with a drawer handle
	$T_2$	close a drawer with a drawer handle
door-close	$T_1$	close a door with a door handle
	$T_2$	open a door with a door handle
door-open	$T_1$	open a door with a door handle
	$T_2$	close a door with a door handle
window-close	$T_1$	close a window in the right direction
	$T_2$	open a window in the left direction
window-open	$T_1$	open a window in the left direction
	$T_2$	close a window in the right direction

Table 3. Comparison of success rates for the multitask simulation (the target position of the handle is unknown except for †).

					ViT-B/32
Task	PPO	Goal†	ViT-B/32	ViT-L/14	(finetune)
drawer-close	0.80	(0.93)	0.59	0.66	0.93
drawer-open	0.00	(0.98)	0.22	0.49	0.94
door-close	0.96	(0.99)	0.96	0.98	0.99
door-open	0.00	(0.89)	0.17	0.21	0.82
window-close	1.00	(0.96)	0.93	0.93	0.93
window-open	0.91	(0.93)	0.87	0.89	0.90

Table 4. Comparison of text accuracy rates for the simulator images.

Task	ViT-B/32	ViT-L/14	ViT-B/32 (finetune)
drawer-close	0.50	0.73	0.94
drawer-open	0.49	0.69	0.95
door-close	0.55	0.73	0.97
door-open	0.63	0.78	0.94
window-close	0.52	0.71	0.90
window-open	0.68	0.73	0.91

Table 5. Texts used in more complex simulation tasks.

Task	Variable	Text
push-button-	$T_1$	press a red button
press-wall	$T_2$	pull a red button
assembly	$T_1$	place an assembly near the bar
	$T_2$	place an assembly away from the bar
box-close	$T_1$	place a lid on the box
	$T_2$	place a lid away from the box
shelf-place	$T_1$	place a blue box onto the top shelf
	$T_2$	place a blue box away from the top shelf

Figure 4. Control results for more complex tasks when applying our method. (a) button-press-wall. (b) assembly. (c) box-close and (d) shelf-place.

Table 6. Success rate for more complex tasks when applying our ViT-L/14.

push-button-press-wall	assembly	box-close	shelf-place
0.81	0.00	0.16	0.00

Figure 5. Experimental configuration.

Table 7. Texts used in the experiment.

Task	Variable	Text
chair-rearrangement	$T_1$	place a green chair under the table
	$T_2$	place a green chair away from the table
box-rearrangement	$T_1$	place a red box next to the yellow box
	$T_2$	place a red box away from the yellow box

Table 8. Comparisons of success rate for the real robot experiment (The target position of the object is unknown except for †).

				ViT-B/32
Task	Goal†	ViT-B/32	ViT-L/14	(finetune)
chair-rearrangement	(1.00)	0.43	0.23	0.83
box-rearrangement	(1.00)	0.27	0.50	0.80

Table 9. Comparison of text accuracy rates for the real images.

			ViT-B/32
Task	ViT-B/32	ViT-L/14	(finetune)
chair-rearrangement	0.66	0.50	0.89
box-rearrangement	0.48	0.73	0.95

Figure 6. An example of control results when applying each method. (a) ViT-B/32. (b) ViT-L/14 and (c) ViT-B/32 (finetune).