Deep-reinforcement-learning-based gait pattern controller on an uneven terrain for humanoid robots

Figures & data

Figure 1. Reinforcement learning.^[22]

Figure 2. Type A fuzzy membership function.

Figure 3. Type B fuzzy membership function.

Figure 4. Discrete wavelet transform process.

Figure 5. System architecture.

Figure 6. Humanoid robot: (a) front view and (b) side view.

Figure 7. State of training with PPO2.

Figure 8. Staircase with steps of different heights.

Table 1. Parameters of gait patterns.

Parameter	Meaning
Swing scale	Swinging extent of the robot
Step scale	Foot height above the floor in each step
Step offset	Knee bend angle
Vx scale	Step length
Vy scale	Distance between the left and right foot

Figure 9. Changes of gait parameters when the robot is walking. (a) Gait parameters without PPO controller and (b) Gait parameters when PPO controller is operating.

Table 2. Parameters of PPO2.

Parameter	Value
Learning rate	0.00025
Discount factor (γ)	0.99
Entropy coefficient	0.01
Value function coefficient	0.5
Maximum gradient norm	0.5
Lambda	0.95
Mini batches per update	4
Epoch per update	4
Clipping range	0.2

Figure 10. Reward values during the training process.

Figure 11. Acceleration record when the reward was −32.

Figure 12. Acceleration record when the reward was 1.5.

Figure 13. Acceleration record when the reward was 49.

Figure 14. Acceleration record after training.

Figure 15. Experimental results before training.

Figure 16. Final results after training.

Figure 17. Acceleration record in the x, y, and z directions processed by the unprocessed sensor data.

Figure 18. Acceleration record in the x, y, and z directions processed by wavelet transform.

Figure 19. Acceleration record in the x, y, and z directions processed by the type A fuzzy membership function.

Figure 20. Acceleration record in the x, y, and z directions processed by the type B fuzzy membership function.

Table 3. Standard deviation of the PPO2 training results.

Method	Acc (x)	Acc (y)	Acc (z)	Average
Original	0.831738	0.976809	2.026566	1.278371
Wavelet	0.836231	0.950575	1.879724	1.222177
Fuzzy (type A)	0.941108	0.949982	2.223176	1.371422
Fuzzy (type B)	0.954038	0.924239	2.244834	1.37437

Table 4. Standard deviation of the TRPO training results.

Method	Acc (x)	Acc (y)	Acc (z)	Average
Original	0.837712	0.948297	2.090171	1.29206
Wavelet	0.858179	0.947015	1.949353	1.251516
Fuzzy (type A)	0.858042	0.941204	2.027458	1.275568
Fuzzy (type B)	0.867374	0.917111	1.88454	1.223008

Table 5. Standard deviation of the A2C training results.

Method	Acc (x)	Acc (y)	Acc (z)	Average
Original	0.866369	0.954105	2.082494	1.300989
Wavelet	0.87761	0.943818	1.988894	1.270108
Fuzzy (type A)	0.922181	0.920244	2.012963	1.285129
Fuzzy (type B)	0.936362	0.946044	1.969335	1.283914

Figure 21. Experimental results on vibrating terrain.

Figure 22. Experimental results with impact disturbance.

Figure 23. Experiment of additional motion on vibrating terrain (turn left).

Figure 24. Experiment of additional motion on vibrating terrain (turn right).

Figure 25. Experiment of additional motion with impact disturbance (turn left).

Figure 26. Experiment of additional motion with impact disturbance (turn right).

Li, D.; Okhrin, O. DDPG car-following model with real-world human driving experience in CARLA. 2021.

 Google Scholar