A big-data-driven matching model based on deep reinforcement learning for cotton blending

Figures & data

Figure 1. Neural network.

An example of a generic Neural Network Chart showing the input/output layers and the pre and post-output positions.

Figure 2. DQN model.

An example of a generic Deep Q Network model showing the neural network arrangement to learn a Q function.

Table 1. DQN algorithm flow table.

DQN Algorithm Flow
First, initialize the Memory Database, which has buffer_size;
Construct Q-neural network (Input_layer = 4, Hidden_layer = 63, Output_ayer = 3, Dropout = 0.5) and Initialize the randomly generated weight ω;
Initialize the target Q network with a weight of ω^−= ω;
Loop over episode =1, 2 … , EPOSODE_COUNT:
Initialization start STATE;
LOOP Step =1,2 … , len(data):
Generate action at with the ε – greedy strategy: select a random action with ε probability, or select at = maxaQ(STATE,a;ω);
Execute action at to receive Reward rt and new state STATE + 1;
Store the transformation samples(STATE,at,rt, STATE + 1) in the Memory Database;
Randomly select a minibatch transfer from the Memory Database (Sj,aj,rj,Sj + 1);
If the step j + 1 is terminal, let yj = rj, otherwise, let yj = rj + γmaxa′Q(St + 1,a′;ω^−);
The (yj−Q(St,aj;ω))^2 about ω is updated using gradient descent method;
Update target Q network every C steps,ω^−= ω;
End For;
End For.

Figure 3. Intelligent manufacturing cotton blending optimisation of a big-data-driven matching model based on deep RL.

An extended version of the big-data-driven model was applied to the cotton matching problem. This model connects the generic models shown in Figures 2 and 3 with additional data sources and business functions to produce the optimisation model.

Table 2. 60 and 80 yarns before and after adjustment variables and average cotton usage (After adjustment and Before adjustment).

Cotton variety 60 yarns	American Cotton	Australian Cotton	Xinjiang Cotton	long-staple cotton
American Cotton	(0.13982, 0.14146)
Australian Cotton		(0.11351, 0.11353)
Xinjiang Cotton			(0.72464, 0.72398)
Long-staple cotton				(0.02202, 0.02102)
Cotton Variety 80 yarns	American Cotton	Australian Cotton	Xinjiang Cotton	long-staple cotton
American Cotton	(0.11667, 0.12999)
American Cotton		(0.18952, 0.18828)
American Cotton			(0.66706, 0.65163)
American Cotton				(0.02675, 0.03010)

Note: The table shows the average of the four kinds of cotton used before and after adjusting 2 units of 60 and 80 yarns.

Table 3. Parameters used in the experiment.

Hyper-Parameter	Search Range	Value	Implication
ϵ	[1e-1,1e-2,1e-3,1e-4]	1e-3	Probability of random action
γ	[0.97,0.98,0.99]	0.99	Incentive discount factor
lr	[5e-1,5e-2,5e-3,5e-4,5e-5]	5e-4	Learning rate of gradient descent algorithm
EPISODE_COUNT	[500,1000,1500]	1000	Iterations
State_size	[4,8,16,32]	8	Number of input neural network nodes
Network layer	[2,3,4,5]	3	Number of Layers
buffer size	[1e3,1e4,1e5,1e6]	1e5	Buffer pool size
UPDATE_EVERY	[2,4,8,16]	4	Buffer pool updates randomly
Dropout	[0.3,0.5,0.8]	0.5	Nodes are discarded according to probability

Figure 4. Stock return prediction results.

An image of stock return prediction results shows one thousand iterations of the data. After 800 iterations, the robot agent learned the principle of buying low and selling high.

Figure 5. Stock buying and selling chart.

An image of a stock buying and selling chart that shows the fluctuations in high and low stock prices and points that show when is best to buy or sell the given stock.

Figure 6. The smoothness of the stock data set.

An image showing how the DQN (RL) model is creating a convergence of the data that shows an oscillatory form.

Table 4. Returns compared with other recent papers.

Author	Time	Paper	Method	Reward (whether to use big data design)	Annual return(%)
Koratamaddi et al.	2021	Market sentiment-aware deep RL approach for stock portfolio allocation	Adaptive		18.84
Xiong et al.	2018	Practical deep RL approach for stock trading	DDPG		25.87
Wu et al.	2020	Adaptive stock trading strategies with deep RL methods	Turtle		27.144
				No	26.790
	DQN(My Model)			Yes	33.865

Figure 7. The loss values of the stock data set.

An image showing that the loss value of the neural network of the DQN (RL) model on the stock trading dataset shows convergence.

Figure 8. Robust test of stock return prediction results.

Based on the DQN (RL) model, the trading data of Yongyi shares are selected to predict the stock results.

Figure 9. Cotton blending experiment results.

An image that shows the experimental results of one thousand iterative cycles for cotton blending and shows the lowest cost cotton blending scheme suitable for 60 yarns without changing the quality is found.

Figure 10. Performance evaluation of cotton blending model.

An image that shows that the DQN (RL) has improved characteristics of other RL methods. Its smoothness between iterations is lower than that of other available models.

Table 5. Optimal cotton blending scheme and price comparison of different cotton blending schemes before and after adjustment.

	Cotton variety	Lower cost scheme			Higher cost scheme
Before Adjustment	Australian Cotton	0	0.0244	0	0.1667
	American Cotton	0	0	0	0.3
	Xinjiang Cotton	1	0.9756	1	0.4667
	Long-staple Cotton	0	0	0	0.0667
After Adjustment	Australian Cotton	0.0238	0	0.0244	0.0833
	American Cotton	0	0	0	0.65
	Xinjiang Cotton	0.9762	1	0.9756	0.2333
	Long-staple Cotton	0	0	0	0.0333
Price		1.2309	1.2312	1.3212	1.7855

Note: The table shows the average of the four kinds of cotton used before and after adjusting two units of 80 yarns.

Data availability statement

The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.