Federated machine learning for privacy preserving, collective supply chain risk prediction

Abstract

The use of Artificial Intelligence (AI) for predicting supply chain risk has gained popularity. However, proposed approaches are based on the premise that organisations act alone, rather than a collective when predicting risk, despite the interconnected nature of supply chains. This yields a problem: organisations that have inadequate datasets cannot predict risk. While data-sharing has been proposed to evaluate risk, in practice this does not happen due to privacy concerns. We propose a federated learning approach for collective risk prediction without the risk of data exposure. We ask: Can organisations who have inadequate datasets tap into collective knowledge? This raises a second question: Under what circumstances would collective risk prediction be beneficial? We present an empirical case study where buyers predict order delays from their shared suppliers before and after Covid-19. Results show that federated learning can indeed help supply chain members predict risk effectively, especially for buyers with limited datasets. Training data-imbalance, disruptions, and algorithm choice are significant factors in the efficacy of this approach. Interestingly, data-sharing or collective risk prediction is not always the best choice for buyers with disproportionately larger order-books. We thus call for further research on on local and collective learning paradigms in supply chains.

Keywords:

• Supply chain
• machine learning
• artificial intelligence
• federated learning
• risk
• privacy preserving

1. Introduction

Complex, international supply chains underpin the survival of much of our economy, supporting critical sectors such as food, defence, aerospace and automotive. Both high-impact-low-frequency and low-impact-high-frequency events that cause supply chain disruption are on the rise due to increased complexity and interdependence.Supply chain disruptions have been identified as a top business risk factor in a variety of industrial sectors and have been prioritised by numerous governments (Edward 2022). Recent studies highlight that efficiency oriented management paradigms are no longer applicable as they fundamentally assume continuous availability of resources, and question whether new paradigms are needed to function in an environment with long term resource shortages (Ivanov and Dolgui 2022a).

The field of supply chain risk management (SCRM) studies risk management techniques that aim to reduce risk exposure in supply chains. Proactive strategies are mainly targeted at risk mitigation and include actions such as increased safety stock, multi-sourcing, information sharing, collaboration, and contingent rerouting. Most of these studies mainly focus on trade offs between enhanced resilience and the costs of adding redundancies or flexibility in supply sources or in-house production. Reactive strategies are deployed after a disruption occurs, and include strategies such as dynamic pricing to counter demand, supply chain reconfiguration, operational rerouting (Ghadge, Dani, and Kalawsky 2012). However, a priori prediction of risk in supply chains remains an unsolved problem that has received little attention.

To that effect, while certain types of risk are impossible to predetermine (e.g. Covid-19), recently, researchers proposed the use of machine learning (ML) methods to predict supply chain risk in contexts where historical data is available (Brintrup et al. 2020). Other proposals included the deployment of ‘digital supply chain risk surveillance (DSCRS)’, where automated algorithms aid in improving supply chain visibility and the extraction of unwanted dependencies (Kosasih and Brintrup 2022; Brintrup et al. 2022), and monitoring news sources (Chae 2015). Applications of ML have thus far included the prediction of late deliveries, delivery windows, and delivery failures mainly using data from enterprise resource planning (ERP) systems (Brintrup et al. 2020). While not as high impact as a one-off, unexpected black swan event, delivery delays have slow and steady impact on business value, by aggregating at the supply chain level.

Extant research has shown that the use of supplier delivery data can achieve performances ranging between 70% and 80% accuracy in use cases that spanned automotive, aerospace, and defence (Brintrup et al. 2020). In all these cases however, risk prediction is conducted on data available to a single company, despite the supply chain being a network of interdependent organisations each of which only have partial visibility of the whole system (Christopher and Peck 2004). It is thus, not unreasonable to postulate increased data visibility could yield better risk prediction. While academics working in the field of SCRM have long argued that data-sharing would help make better forecasts and reduce the risk of stock outs or overproduction, organisations typically do not share data due to the fear of data being misused to gain a competitive advantage by others in the network (Ghosh and Fedorowicz 2008).

As the popularity of ML approaches in supply chain risk management grows, we postulate that a new approach from the field of artificial intelligence (AI) may help mitigate this concern, potentially opening up a new and promising avenue for predicting a priori risk and advancing the field of SCRM. This new approach, named federated learning (FL), was initially proposed by the Google AI team (McMahan and Brendan 2017), and aims to enable edge devices or data servers to collaboratively learn a shared model while keeping training data private. This approach solves the problem of data privacy and security by decoupling the ability of traditional ML from the need to pool all local datasets together in a device or a server. It also reduces communication costs as only model parameters are transmitted rather than raw data.

FL, having originated as a privacy preserving machine learning method in the field of mobile devices, offers methods that can enable collective learning in supply chains. FL enables nodes, in our case buying and selling companies, to share relevant predictive model parameters or intermediate training results to create locally useful sub-models, without the risk of organisational data being exposed and used opportunistically. FL is a new field in computer science, with applications in diverse fields such as healthcare and mobile devices, which achieved differing success rates, however has not yet been applied to supply chain risk prediction. Application of FL into supply chain risk prediction is non-trivial as its conceptualisation necessitates structured problem formulation, and an understanding of when it is better for supply chain members to predict risk alone, and when to do so collectively.

In this paper we aim to explore (i) whether federated learning approaches can be deployed on digital supply chain data effectively to address some of the supply chain risk prediction needs of organisations, and ((ii) when it is better for organisations to predict alone, and when would they benefit from a federated approach. We do so through a case study based approach where empirical datasets from three buyers in the engineering sector are used to predict delivery delays. The case study conducted is in the maritime engineering sector where, on average, buyers share 46% of their suppliers, on whom they would like to develop common risk models. We cross compare FL approaches to local-learning, where buyers develop models using their own data, and to centralised-learning, where buyers pool their data together. We explore how the performance of federated learning is moderated by local dataset size, amount of disruptions faced by a given company during Covid19, and the types of machine learning algorithm used. Our results show that the performance of the FL approach is close to local and centralised learning. However in comparison to centralised learning, FL offers the distinct advantage of data privacy preservation. In comparison to local learning, FL may offer better performance in cases where local datasets are not sufficient. We also find that supply chain members with large and balanced datasets can provide significant help to those members with small and imbalanced datasets through sharing predictive model parameters. Another interesting finding is that contrary to previous studies on supply chain data-sharing, the pooling of datasets is not always beneficial for all network members. For companies with large historical datasets, i.e. a long standing relationship with suppliers, data sharing adds noise to the local learning process, and specialised supplier information is lost, making inferences less accurate. On the other hand, they would still benefit from joining in a federated approach, when there are changes in the supply chain, such as the procurement of a new product from an existing supplier. In such cases, federated approaches may offer a balanced trade off where local and federated models are flexibly swapped, or heterogenously weighted when needed.

2. Background

2.1. Artificial intelligence for supply chain risk management and the role of data

As the field of Supply Chain Risk Management has gained increased attention due to recent world events, so did research that proposes AI based solutions to risk management (Dolgui and Ivanov 2021; Ivanov 2022a; Baryannis et al. 2019; Brintrup 2020; Ganesh and Kalpana 2022). Although in its infancy, extant proposals offer promise in addressing a number of long-standing risk concerns, such as improving visibility through automated supply chain mapping (Wichmann et al. 2020), prediction of disruptions (Brintrup et al. 2020), the use of digital surveillance (Kosasih and Brintrup 2022), bots that negotiate and reconfigure supply chains automatically in the face of disruptions (Xu, Mak, and Brintrup 2021), coupling intelligent event detection with supply chain stress testing (Ivanov and Dolgui 2022b).

Our contribution is primarily positioned within the context of AI in SCRM. Extant approaches within AI-based risk prediction have focussed on the use of datasets available to a singular company, primarily to predict disruptions. These include works of Brintrup et al. (2020) who used ERP data from an aerospace manufacturer to predict order delays with a variety of classification algorithms such as Support Vector Machines, Logistic Regression, and Naïve Bayes. Li and Zeng (2016) deployed Bayesian Network models for failure mode analysis. Rodgers and Singham (2020) constructed a Bayesian Network to predict disruption impacts in a supply network. Badakhshan and Ball (2022) used the CART algorithm to predict payment disruptions and identify the inventory and cash replenishment policies that minimise their impact. Barros et al. (2022) developed a Random Forest based approach to estimate supply lead times for improved Safety Stock keeping.

Initial reports from this body of work are promising and point to increased SCRM capability for predicting disruptions through the use of AI technology. However these advances typically focus on building predictive models based on data from a singular company. As AI driven approaches necessitate the use of large datasets, the use of these advances become inevitably restricted to companies who have enough data to pursue disruption prediction. Companies may not be able to utilise these when they are at the initial stages of a relationship with their supplier, or when there are subtle changes in the relationship such as the procurement of a new product, or ordering from a new location, such that existing datasets become less relevant for new ordering patterns.

In such cases, a naïve proposal may be that companies share their data to enable better resilience. A large body of literature supports the notion of data-sharing in the context of supply chain disruptions (please see Duong and Chong 2020 for a recent review). While original studies on supply chain data sharing date back to bullwhip effect (Lee, Padmanabhan, and Whang 1997) and shortage gaming problems (Cachon and Lariviere 1999), the premise of data sharing has recently been highlighted as an important activity in building supply chain resilience (Pettit, Croxton, and Fiksel 2019).

In the context of SCRM, data sharing has the goal of facilitating learning from previous disruption events, and help assess risk common to all parties (Chen et al. 2019). Through a variety of empirical and simulation based studies, researchers showed that data sharing increases the system's abilities to deal with disruptions (Soni, Jain, and Kumar 2014; Chen and Huang 2021) by monitoring performance and identifying potential problems in supply chains (Macdonald et al. 2018). Having surveyed 157 papers in the last 20 years, Duong et al (2020) found that technological and information sharing practices may become a key factor in enhancing resilience, however, they highlight that technological mechanisms are often costly, and the main reason for not sharing information is underlying trust issues (Duong and Chong 2020). Lotfi et al. (2013) further highlight that despite the reported advantages of data sharing, most practical approaches yielded limited success, because of the desire to maintain competitive advantage, concerns about confidentiality and opportunism, and questions about the reliability and accuracy of available data. Duong and Chong (2020) call for innovations that reduce information sharing costs and help protect the privacy of information could lead to equal benefits for all supply chain partners.

A number of researchers studied how trust can be improved through technological innovations. Most recently, Chen and Huang (2021) suggest the use of digital twins for members of a supply chain to feed data into one another (Chen and Huang 2021), suggesting that trust would emerge if companies observe the benefit of sharing of fast-streamed operational data. Hellwig et al. (2022) proposed the use of blockchain technology to facilitate the sharing of ‘trustless’ information by ensuring anonymity of supply chain partners. Blockchain allows for tamper-proof permanent recording of encrypted data on a decentralised ledger, where records cannot be altered, and participants can remain anonymous. They test their proposals by building DISASTER, a game information sharing platform among competing retailers. A virtual market for trading tokens is then created, which represent claims on suppliers' capacities among (competing) retailers. By trading capacity on the virtual market, distribution of inventory is optimised prior to physical shipments, thereby facilitating more efficient operations may be achieved as compared with traditional transshipments. Reenacting the shortage game, Hellwig et al. (2022) allow competing retailers to access historical information on average orders on their shared supplier. They find that decision makers act more strategically and closely to Nash equilibrium as a result. They also note a downside: increased data sharing leads to order inflation and have adverse consequences, resulting in inefficient planning.

Our contribution is complementary to this emerging strand of thinking. However, contrary to existing efforts, we ask: ‘What if we did not share data at all, but use a common prediction model, updated by competing members on the supply chain?’. The reasoning behind this question is the generalised premise that data-sharing may improve information asymmetric problems in SCRM, and the recent observation that perhaps more data does not create an efficient system after all (Hellwig et al. 2022).

We define information being shared as the common interpretation on supply chain system state. Information sharing allows each member's data to remain private, whilst ensuring that those members who engage in new transactions can benefit from each others' interpretation on supply chain system states. To illustrate this principle, we focus on delivery delays, where each buyer has access to a model on predicted supply delay, based on each members' unique relationship with the supplier so that predictions on risk can be collectively developed.

2.2. Federated learning

Initially proposed by the Google AI team, the key problem federated learning (FL) addressed was that when a centralised model is trained based on data originating from sources such as mobile phones or sensors, the training data typically remains distributed over the clients (McMahan and Brendan 2017). However sending personal data to a central server is problematic for confidentiality reasons. In FL, a mobile device downloads the current model from the cloud and improves it by learning from its own data. The improved model is then sent to the cloud, using encrypted communication, to be averaged with other mobile device updates for the shared model improvement. Thus FL enables collective learning without sharing raw data by decentralising the training of common predictive models. After being initially proposed for mobile keyboard input prediction in Android phones, Smith et al. (2017) proposed to further leverage the relationship between clients, and cast the problem as an instance of multi-task learning (MTL) to derive local client models benefiting from similar ones. A popular follow on application area has been the Internet of Things (IoT) for sensor based predictive analytics (Lu et al. 2019; Savazzi, Nicoli, and Rampa 2020; Kwon et al. 2020; Yuan, Ge, and Xing 2020). In both mobile devices and IoT, local learning process benefits from an existing common model to start with.

Since its inception in 2017, the number of articles on FL research grew exponentially, to more than 5500 articles today. Researchers focused on trade offs between central and local training, prevention of data reconstruction by malicious attackers, reducing vulnerability to inference attacks (Choudhury et al. 2019), expansion of FL to other machine learning tasks. Non-mobile device applications included cross-institutional medical diagnosis (Sheller et al. 2020), forecasting hospitalisations in the USA (Brisimi et al. 2018; Dayan et al. 2021), credit risk (Kawa et al. 2019), fraud detection (Yang et al. 2019; Li et al. 2019), energy demand forecasting (Briggs, Fan, and Andras 2021; Tun et al. 2021), and pharmaceuticals discovery (Choudhury et al. 2019).

In this paper, we aim to explore how and to what extent an FL based approach can facilitate supply chain risk prediction, by cross-comparing scenarios where: buyers predict delays using their own data, buyers pool their data together to make predictions on supplier delays, and an FL approach where buyers train a common prediction model.

3. Federated learning for collective supply chain risk prediction

This section outlines the FL based approach for collective supply chain risk learning (Figure 1). Our problem setting involves a number of buyers each of which have a historical dataset constituting previous transactions with their suppliers. Buyers have exclusive suppliers as well as suppliers they share with one or more other buyers (further detailed in Section 4). In this setting FL is used to create a common prediction model to estimate whether a given order will arrive later than planned, yielding what is known as a ‘classification’ problem (Brintrup et al. 2020). Delay prediction can be carried out using several machine learning problem formulations, which would determine the algorithmic approach that needs to be taken. For example, one could estimate the amount of delay that a given order may incur, the probability with which a delay might occur, or simply whether an order will be delayed or not. In order to contextualise our work within the widest literature possible, we opted to formulate delay prediction as a binary classification problem where a classifier algorithm classifies data into two non-overlapping classes. Hence we need to identify between delayed orders and on-time orders.

Figure 1. The framework of Federated Learning. Graphical illustration of the working principle of federated learning, consisting of a central server with a global model and k local servers with each having a local model and a local dataset. It starts from central server initialising global model's parameters to be downloaded by local servers as their local models' parameters, and then local models are trained on their own datasets before being used to update the global model.

As ordering patterns are heterogeneous, some buyers will have more information on a shared supplier's performance than others. The overarching question is thus, whether buyers who have less information can tap into a common predictive model to make better estimates, and if so, when is it better to use a common model and when to rely on local information, i.e. a predictive model trained by an individual buyer's data only. Thus we seek to explore whether and under what circumstances an FL based approach to order delay prediction yields beneficial results.

More formally, consider a global model stored in a central server and K local models, that are constructed from K buyers using their local datasets, $\{D_1,D_2,\dots ,D_k,\dots ,D_K\}$ (Figure 1 ).

In this setting, each buyer contributes to global model updates, where $D_k$ describes local dataset k collected from the corresponding buyer k.

Each local dataset is further partitioned into M subdatasets with N samples, so as to ensure that each local model is trained on a different subdataset and at every update new data that is generated can be effectively utilised. i.e. $D_k$ is partitioned into $\{D_{k_1},D_{k_2},\dots ,D_{k_m},\dots ,D_{k_M}\}$.

The global model training phase then commences, where:

1. The global model's parameters, $W_0^{glo}$, are initialised on the central server.

2. Each buyer downloads the initialised global model's parameters, mapping them onto their first local model's parameters at t = 0. Each buyer then uses their own dataset to train this local model. Following training, the updated local model parameters are sent to the global model at $t^{th}$ round, defined as $\{W_{1;t}^{loc},W_{2;t}^{loc},\dots ,W_{k;t}^{loc},\dots ,W_{K;t}^{loc}\};(t=0,1,2,\dots )$.

3. After being trained at $t^{th}$ round, local model parameters are updated to $\{W_{1;t}^{loc},W_{2;t}^{loc},\dots ,W_{k;t}^{loc},\dots ,W_{K;t}^{loc}\}$ and sent to a central server for the global model updates. Global model updates are carried out following the FedAvg method, which uses the average of all buyers' local model parameters to update the global model by Equation (1(1) $W_t^{glo}=\frac{W_{1;t}^{loc}+W_{2;t}^{loc}+\cdots +W_{k;t}^{loc}+\cdots +W_{K;t}^{loc}}{K}$(1) ) (1) $W_t^{glo}=\frac{W_{1;t}^{loc}+W_{2;t}^{loc}+\cdots +W_{k;t}^{loc}+\cdots +W_{K;t}^{loc}}{K}$(1)

After the formulation of the above general approach to the application of FL in a supply chain setting, the next steps include the selection of a suitable machine learning algorithm, appropriate optimiser and loss function. As mentioned previously, a number of machine learning algorithms have been experimented with in FL research (Zhang et al. 2021) with differing success rates. Not all algorithms are suitable for FL. For example, tree-based methods create data leakage issues and need additional encryption, which increases computational cost. Algorithms need to be parameter based. We thus opt to use artificial neural networks (ANN) and convolutional neural networks with one dimensional kernels (CNN1D) due to their reported performance in recent FL studies, and logistic regression (LogReg) and multi-layer perceptron (MLP) as further benchmarks. Table 1 shows the structures of four ML models under the framework of FL. As mentioned previously, we shall utilise these in a classification setting.

Table 1. Machine learning parameters used for developing FL.

ANN		CNN1D		LogReg		MLP
Layer name	Parameters	Layer name	Parameters	Layer name	Parameters	Layer name	Parameters
Linear layer^1	100	Linear layer^1	200	Linear layer^1	200	Linear layer^1	100
ReLU layer^1	–	–	–	–	–	ReLU layer^1	–
BatchNorm1D^1	(100)	–	–	–	–	–	–
Linear layer^2	100	CNN-1D layer^2	32, (7)	Linear layer^2	2	Linear layer^2	100
ReLU layer^2	–	ReLU layer^2	–	–	–	ReLU layer^2	–
–	–	Pooling layer^2	7	–	–	–	–
BatchNorm1D^2	(100)	BatchNorm1D^2	(32)	–	–	–	–
Linear layer^3	100	CNN-1D layer^3	64, (7)	–	–	Linear layer^3	2
ReLU layer^3	–	ReLU layer^3	–	–	–	ReLU layer^3	–
–	–	Pooling layer^3	7	–	–	–	–
BatchNorm1D^3	(100)	BatchNorm1D^3	(64)	–	–	–	–
–	–	Flatten layer	–	–	–	–	–
FC layer	2	FC layer	2	–	–	–	–
Sigmoid layer	–	Sigmoid layer	–	Sigmoid layer	–	Sigmoid layer	–

ANN consists of three linear layers with 100 neurons each followed by the activation function of ‘ReLU’ (ReLU layer) and a batch normalisation layer (BatchNorm1D). ‘ReLU’ is used for accelerating the training process and avoiding the vanishing gradient problem (Nwankpa et al. 2021) while a BatchNorm1D is used for normalising the output from the ReLu layer and ensuring that the training process is stable. Extracted features are sent to a fully-connected layer followed by a ‘Sigmoid’ function for the final prediction. We set the number of neurons as 100 based on the best experimental result achieved from 50 to 500 neurons.

CNN1D is composed of a linear layer and two convolutional layers (CNN-1D) followed by the activation function of ‘ReLU’, a ‘AvgPooling’ layer (Pooling layer) and a batch normalisation layer each. The pooling layer aims to reduce the dimension of extracted features from ‘ReLU’ layer. Similar to ANN, a FC layer followed by ‘Sigmoid’ function is used for the final prediction. Considering the computational cost, we only use 32 kernels with kernel size as 7 for the first CNN-1D layer and 64 kernels with kernel size as 7 for the second CNN-1D layer. Due to defining our task as a binary classification, the FC layer for both ANN and CNN1D has 2 neurons.

LogReg contains two linear layers, one with 200 neurons and the other one with 2 neurons, to analyse the relationship between output and input features, followed by a logistic function, ‘Sigmoid’, at the end to squeeze the output of the second linear layer between 0 and 1, aiming to solve binary classification problems. The output in LogReg is a probability between 0 and 1 and can be used for classification problems while linear regression can work well for regression problems but is not suitable for classification problems.

MLP commonly consists of three layers: an input layer, a hidden layer and an output layer. In our case, we consider two linear layers with 100 neurons each as the input layer and hidden layer and a linear layer with 2 neurons as the output layer. Each linear layer is followed by a ‘ReLU’ layer. At the end, it is a ‘Sigmoid’ layer for final prediction.

For optimising local models, we follow (McMahan et al. 2017) and select the stochastic gradient decent (SGD) which has been popularly used in deep learning research (Taheri, Rastgarpour, and Koochari 2021). We use Cross Entropy Loss (Zhang and Sabuncu 2018) as the loss function for local model training while the loss for the global model is updated by averaging losses of all participant local models.

The training process is challenged by training imbalance, which refers to contexts where datasets are heterogeneously distributed across members of the FL. In imbalanced contexts, the classifier has access to more data from one node (ie. company), which may bias the learning process. Considering scenarios in the real world, we opt to train our FL approach in both balanced and imbalanced settings for cross-comparison. For balanced training, all local datasets and corresponding partitioned subdatasets are uniformly sizes. This means that both M and N are fixed, which ensures that each local model is updated equally by being trained on a randomly sampled subdataset of the same size at each round. For imbalanced training, all local datasets and corresponding partitioned subdatasets are heterogeneously sized. This means both M and N are dynamic, which enable each local model to be updated unequally after a given round. Next, we detail our case study.

4. Case study

Our case study originates from the maritime sector where complex engineering assets are produced. Three buyers share 70 suppliers, which constitute 47.54%, 39.04%, and 53.13% of their overall individual supply chains, as shown in Table 2. Each buyer would like to create a predictive model for estimating the delays associated with orders to aid production and inventory planning as well as conducting what-if scenarios. Although each buyer may develop a local predictive model using historical data that is available to them, they would like to explore whether a common predictive model on shared suppliers could improve predictive performance.

Table 2. Common statistical analysis of used datasets.

	NC-Buyer1	C-Buyer1	NC-Buyer2	C-Buyer2	NC-Buyer3	C-Buyer3	Total
Number of orders	27,757		31,639		18130		77,526
Number of orders	19,708	8,049	25,688	5,951	8,989	9,141	77,526
Number of suppliers	122		146		96		268
Number of shared suppliers (x%)	58 (47.54%)		57 (39.04%)		51 (53.13%)		70 (26.12%)
Delayed rate	69.53%	51.62%	51.88%	61.38%	78.30%	61.13%	62.31%
On-time rate	34.07%	48.38%	48.12%	38.62%	21.70%	38.87%	37.69%
Maximum of delays (days)	1126	597	989	563	413	587	1126
Mean of delays (days)	69.87	28.87	34.97	27.69	50.67	21.90	39.00
Standard deviation of delays (days)	135.34	94.94	92.09	71.20	74.42	69.37	89.56

The data used spans orders delivered between 2015 and 2022. Considering the challenges brought by the Covid19 pandemic to global supply chains (Guan et al. 2020), we split each buyer's data into two phases, post-Covid19 and pre-Covid19, in order to explore how the results obtained by the FL approach are moderated by this large-scale disruption.

All buyers are located in United Kingdom where the first national lockdown started from March of 2020. Therefore, we define the duration of 12 March 2015 and 01 March 2020 as pre-Covid19 period and the duration of 02 March 2020 and 01 March 2022 as post-Covid19 period, and split data accordingly. As there are three buyers in total, six datasets are generated and used: three post-Covid19 datasets are labelled as C-Buyer1, C-Buyer2 and C-Buyer3 and three pre-Covid19 datasets are labelled as NC-Buyer1, NC-Buyer2 and NC-Buyer3, respectively.

4.1. Data preprocessing

First, a number of data-preprocessing steps have been applied to remove incomplete, inconsistent, redundant and noisy data. Specifically, this included the removal of orders without actual delivery dates, features with identical values, and duplicated orders. Suppliers with less than 10 orders have been removed from the dataset in order to reduce information diversity and thus dimensionality of encoded features.

‘Delay Delta’, labelled as $\mathrm{\Delta }T$, was then calculated by subtracting contracted lead time from the actual delivery lead time. To explore the delivery performance differences pre and post Covid19, we visualise distributions of delay delta and contractual lead time in Figure 2. For visualisation purposes we randomly select a supplier for each buyer during pre-Covid19 (Figure 2(a–c)) and post-Covid19 (Figure 2(d–f)). A non-linear relationship can be observed between contractual lead time and delay delta. Delay delta does not obviously increase due to Covid19. For example, delayed deltas from C-Buyer1 (d), C-Buyer2 (e) and C-Buyer3 (f) are not larger than that in NC-Buyer1 (a), NC-Buyer2 (b), and NC-Buyer3 (c). However, the distributions of both delivery delta and contractual lead time are very different. For instance, the number of delayed orders decrease with longer contractual lead times, and decreases with larger delay delta. Hence it is reasonable to carry out the FL approach in separate time periods, leading to six datasets.

Figure 2. The distributions of delayed orders alighted with its contracted delivery duration and delayed delta. A plot that consists of six joint correlation subplots corresponding to six datasets showing the relationship of contracted delivery duration and delayed delta, the distribution of contracted delivery duration, and the distribution of delayed delta.

Statistical information pertaining the number of orders, suppliers and shared suppliers, delayed order rate, on-time order rate, values of maximum, mean and standard deviation of delays, are displayed on Table 2. It can be observed that the data size of buyer 3 is smaller than buyers 1 and 2, however its average delay rate during both periods is larger than the other two.

The intersecting supply chains of the buyers are shown in Figure 3. Buyers are depicted in dark grey, sole suppliers that serve to a given buyer only are depicted in light grey and shared suppliers are depicted in blue. Edges between suppliers and buyers represent their relationships, weighted by the proportion of orders between a given supplier and buyer, over all orders a buyer gives. The label on each edge represents the ratio of delays a given buyer-supplier relationship endures over all of a given buyer's orders.

Figure 3. The visualisation of the network-wide relationships between suppliers and buyers. A supplier-buyer network plot showing buyers (dark grey nodes), sole suppliers (light grey nodes), shared suppliers (blue nodes), and the ratio of delays for a given buyer-supplier relationship over all of a given buyer's orders labelled on the edge.

The next step in our analysis is the selection of features that will yield a predictive outcome, namely whether an order will be delayed or on-time. Feature selection in an FL setting is constrained to features that are accessible by all suppliers. Heterogeneity (non-IID data) across the feature space in FL is currently not well-established and presents a constraint for a collective learning approaches (Zhang et al. 2021). Common features include those presented in Table 3.

Table 3. The descriptions of selected features.

Feature name	Description
Supplier Name	Legal name of supplier
Project Name	Name of project that the ordered product is used for
Part Description	Characteristic description of the ordered product
Quantity	The number of the ordered product
Created Ordered Date	Date when the order was placed
Promised Delivery date	Contracted lead time date

One-hot encoding technique is a widely used encoding technique (Potdar, Pardawala, and Pai 2017) for feature embedding whereby each unique variable is encoded to a new binary variable. One-hot encoding can help machine learning algorithms avoid the problem of bias across differently scaled features (Ul Haq et al. 2018). We specifically choose one-hot encoding over other methods such as Ordinal, Polynomial or Binary encoding, as it has been shown to outperform these encoding methods when input data does not have ranking, as is in our case (Potdar, Pardawala, and Pai 2017).

Due to the large number of unique variables in datasets, input data becomes high dimensional after one-hot encoding, which in turn increases the demand for computational resources. To address this problem, principle component analysis (PCA) was used for dimensionality reduction, which involves calculating the eigenvectors of the covariance matrix of the original inputs and then extracting those eigenvectors corresponding to large eigenvalues (Reddy et al. 2020).

4.2. Performance metrics

Common metrics used to evaluate the performance of a classification approach are Accuracy, Precision, Recall and F-Score, calculated by confusion matrix (Kulkarni, Chong, and Batarseh 2020). Table 4 describes confusion matrix. Samples that are recognised correctly are defined as TP and TN. Otherwise, they are defined as FP and FN. In our case, TP represents the number of orders that have been correctly classified as delayed deliveries while TN describes the number of orders that have been correctly classified as on-time deliveries. FP refers to the number of orders that have been wrongly recognised as delayed orders while FN means the number of orders that are incorrectly recognised as on-time deliveries.

Table 4. Confusion Matrix.

	Predicted positive	Predicted negative
Real positive	True Positive (TP)	False Negative (FN)
Real negative	False Positive (FP)	True Negative (TN)

Based on the confusion matrix, Accuracy, Precision, Recall and F-Score are calculated as follows:

Accuracy is represented as the ratio of accurate predictions to the total number of orders, $\frac{\mathit{T}\mathit{P}\mathit{+}\mathit{T}\mathit{N}}{\mathit{T}\mathit{P}\mathit{+}\mathit{T}\mathit{N}\mathit{+}\mathit{F}\mathit{P}\mathit{+}\mathit{F}\mathit{N}}$.

Precision is defined as the ratio of accurate predictions of real delays to the total number of predicted delays, $\frac{\mathit{T}\mathit{P}}{\mathit{T}\mathit{P}\mathit{+}\mathit{F}\mathit{P}}$.

Recall describes the ratio of accurate predictions of real delays to the total number of real delays, $\frac{\mathit{T}\mathit{P}}{\mathit{T}\mathit{P}\mathit{+}\mathit{F}\mathit{N}}$.

F-Score represents the equilibrium between the precision and the recall, $\frac{2\times \mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}\times \mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}}{\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}\mathit{+}\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}}$.

For imbalanced datasets, Accuracy is not a reasonable metric to measure the performance of a classification algorithm because the class with few number of samples have a little impact on Accuracy compared to the class with large number of samples (Hamad et al. 2021). Considering that both balanced and imbalanced datasets are tested in our case, an additional metric, named weighted F-Score ($F_w$-Score), is additionally used to evaluate our proposed approach. $F_w$-Score is defined as (2) $F_w-\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\sum _{c}2\times w_c\frac{{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_c\times {\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_c}{{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_c+{\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_c}$(2) where $w_c$ is the proportion of orders for class c and is equal to $\frac{n_c}{N}$. N is the total number of orders while $n_c$ is the number of orders in class c.

4.3. Establishing benchmarks

In order to test the efficacy of a federated approach to risk prediction, two benchmarks were created. First of these is ‘local learning’ where buyers use historical data available only to them to predict the deliver state of an order (i.e. delayed or on-time) if given to a particular supplying agent. This would help answer the question whether and under what circumstances would a federated approach be more beneficial to a company, than simply using their own data. One might expect that availability of sufficient data on a given supplier or material being procured would be a differentiating factor here. If a buyer has recently started a relationship with a supplier,or has recently started outsourcing a particular type of product, they would have little past data available and thus a federated approach to access market based-prediction may be more beneficial. On the other hand, as a buyer's own experience with a supplier grows, it might be better off developing predictive models using its own data, which would inevitably be tailored to its own relationship with a supplier and may capture nuances that are specific to their unique, dyadic relationship. Our inquiry thus becomes centred around the extraction of this trade off.

Another benchmark is that of ‘centralised learning’ where an algorithm is trained on the entire dataset of all three buyers. Clearly this is unrealistic, as the starting premise of a federated learning approach is that buyers will not share their data. However such a benchmark could inform us whether centralised learning is indeed ‘optimal’, and if so, how far from an ‘optimal’ performance a federated approach can be expected to perform, which may be useful in companies determining whether effort spent on building a federated approach is worthwhile. We will further discuss these considerations in the following sections.

4.4. Experimental settings

Experimental settings include those concerning data sampling and algorithmic parameter settings during the training phase.

As the datasets among companies show a degree of imbalance Table 2, we conduct an experiment that involves balancing the datasets by undersampling data to train FL. This undersampling data operation here aims to balance data size among companies not as common research works for balancing samples of different classes in a dataset. After undersampling, the dataset in each company has same size with others. In a round, each local model is updated on the same sample size and contributes to the global model equally. Hereafter, we name this balanced training where every local model used to update the global model is trained on the same sample size.

In contrast, we define Imbalanced training where every local model unequally contributes to the global model by having been trained on a different sample size. Imbalanced training is then implemented in two ways, the first of which involves training every local model on the same sized local sub-sample in a round, however the number of these in each local model are different (let us call this mode of training Type-1). In Type-2, the number of subsamples for each local dataset are the same, however the number of orders in each subsample is different. Type-1 thus aims to investigate how heterogeneity in the size of datasets used to train local models affect the performance of the global model, while Type-2 explores how local models that are trained on imbalanced datasets impact the performance of the global model.

The parameter values that are used to implement balanced training and imbalanced training are displayed in Table 5. For example, for balanced training on pre-Covid19 datasets, we undersample NC-Buyer1 and NC-Buyer2 so they have the same sample size as NC-Buyer3, who has the smallest. Here each buyer has 8989 orders. We then partition each dataset to several subsamples with 600 orders of each based on McMahan et al. (2017). Therefore, each dataset has M=14 (≈8,989/600) subsamples. This ensures that every local model used to update the global model is trained on the same sized dataset. For orders given during Covid19, C-Buyer1 and C-Buyer3 are undersampled to have 5,951 orders, which is the same as C-Buyer2, and each dataset has M=9 (≈5951/600). For imbalanced training, all orders in each dataset join in the corresponding local model training. By Type-1, N is set as 600 and M varies with the size of different datasets; while by Type-2, M is set as 25 and 14 for pre-Covid19 and post-Covid19 datasets, respectively. These settings depend on two conditions. One is to make sure each subsample in the smallest dataset has more than 300 orders so as to avoid being unable to train a local model due to there not being enough orders. The other condition is that the subsample from the largest dataset is balanced from the smallest dataset.

Algorithmic parameters that need to be set during the training phase include the number of epochs, E, batch size, B, and learning rate, r, the number of components, $N_p$, and the number of rounds, R. We set B as 16 after considering the small size of partitioned subsamples for local model training. r is defined as 0.001 following guidelines given in Senior et al. (2013) so as to compromise between large learning rates that result in over-fitting and small learning rates that result in long computational run times. R, is variable and further discussed on Section 4.5.2. In both local learning and centralised learning we use the $stopearly$ strategy to optimise the number of epochs, E, and avoid over-fitting. Using stop−early means that the training process is stopped when the training loss decreases in 10 consecutive epochs while the validation loss increases. This similar experimental environment setting aims to ensure the uniformity of experiments.

Table 5. Data settings for both balanced training and imbalanced training by Type-1 and Type-2.

	NC-Buyer1	NC-Buyer2	NC-Buyer3	C-Buyer1	C-Buyer2	C-Buyer3
Number of orders	19,708	25,688	8,989	8,049	5,951	9,141
Balanced training	N=600; M=14	N=600; M=14	N=600; M=14	N=600; M=9	N=600; M=9	N=600; M=9
Imbalanced training (Type-1)	N=600; M=32	N=600; M=42	N=600; M=14	N=600; M=13	N=600; M=9	N=600; M=15
Imbalanced training (Type-2)	N=788; M=25	N=1027; M=25	N=359; M=25	N=574; M=14	N=425; M=14	N=652; N=14

Following convention (e.g. see Zheng 2021), we use 70%, 10% and 20% of each dataset for training, validating and testing, respectively. All experiments are implemented on a machine equipped with a GeForce RTX 2080 Ti GPU card with 11 GB physical memory and PyTorch is used to build and train all models.

4.5. Experimental results and discussion

4.5.1. Trade offs between local learning and centralised data sharing

Table 6 shows values of four performance metrics, including Accuracy, Precision, Recall and F-Score , achieved by local training, centralised training and FL. Table 7 presents F-Score of each class. $F_w$-Score of benchmarks and FL obtained on all datasets are presented in Figure 4. Based on Tables 6, 7 and Figure 4, our findings are as follows.

Figure 4. Achieved $F_w$-Score by all models on all datasets. A bar chart showing one of performance metrics, $F_w$-Score, achieved on eight datasets used for experiments.

Table 6. Accuracy, Precision, Recall and F-score of local learning, centralised learning and the proposed FL.

Dataset	NC-Buyer1				NC-Buyer2				NC-Buyer3
Model	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP
Local Learning
Accuracy	0.8000	0.8104	0.7498	0.8040	0.8174	0.8268	0.7909	0.8222	0.8305	0.8311	0.7923	0.8133
Precision	0.7831	0.7938	0.7221	0.7890	0.8234	0.8313	0.8011	0.8268	0.7479	0.7536	0.7110	0.7563
Recall	0.7609	0.7760	0.7011	0.7610	0.8146	0.8244	0.7895	0.8214	0.7094	0.6935	0.6097	0.6551
F-score	0.7693	0.7831	0.7084	0.7712	0.8154	0.8253	0.7885	0.8213	0.7249	0.7147	0.6254	0.6785
Centralised Learning (ANN / CNN1D / LogReg / MLP)
Accuracy	0.8033 / 0.8192 / 0.7453 / 0.8085
Precision	0.8013 / 0.8160 / 0.7557 / 0.8088
Recall	0.7773 / 0.7971 / 0.6971 / 0.7807
F-score	0.7851 / 0.8040 / 0.7045 / 0.7895
The Proposed Federated Learning
Accuracy	0.7758	0.7916	0.7160	0.8120	0.8074	0.8037	0.7788	0.8102	0.7455	0.7969	0.7161	0.7710
Precision	0.7451	0.7599	0.7068	0.8002	0.8181	0.8132	0.8358	0.8096	0.6647	0.7165	Nan	0.6844
Recall	0.7233	0.7369	0.6384	0.7786	0.8059	0.8014	0.7641	0.8067	0.6871	0.6770	0.4932	0.6520
F-score	0.7234	0.7401	0.6350	0.7832	0.8030	0.7984	0.7590	0.8049	0.6651	0.6853	Nan	0.6551
Dataset	C-Buyer1				C-Buyer2				C-Buyer3
Model	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP
Local Learning
Accuracy	0.7921	0.7940	0.7471	0.8029	0.7592	0.7647	0.7286	0.7560	0.7416	0.7535	0.6686	0.7297
Precision	0.7922	0.7944	0.7481	0.8031	0.7485	0.7551	0.7146	0.7420	0.7244	0.7377	0.6484	0.7194
Recall	0.7921	0.7941	0.7475	0.8030	0.7473	0.7490	0.7203	0.7407	0.7227	0.7305	0.6382	0.6980
F-score	0.7920	0.7940	0.7470	0.8028	0.7479	0.7514	0.7167	0.7408	0.7233	0.7333	0.6407	0.7033
Centralised Learning (ANN / CNN1D / LogReg / MLP)
Accuracy	0.7458 / 0.7644 / 0.6161 / 0.7611
Precision	0.7393 / 0.7590 / 0.6030 / 0.7564
Recall	0.7369 / 0.7560 / 0.6001 / 0.7517
F-score	0.7379 / 0.7572 / 0.6005 / 0.7534
The Proposed Federated Learning
Accuracy	0.7319	0.7457	0.6050	0.7828	0.7501	0.7635	0.6813	0.7469	0.7530	0.7525	0.5839	0.6969
Precision	0.7356	0.7434	0.6234	0.7855	0.7384	0.7547	0.6532	0.7280	0.7442	0.7379	0.5329	0.6815
Recall	0.7270	0.7421	0.6010	0.7806	0.7417	0.7586	0.6484	0.7329	0.7180	0.7259	0.5295	0.6839
F-score	0.7255	0.7409	0.5803	0.7792	0.7327	0.7476	0.6479	0.7233	0.7213	0.7270	0.5181	0.6788

Table 7. F-score of delayed and on-time orders from local learning, centralised learning and the proposed FL.

Dataset	NC-Buyer1				NC-Buyer2				NC-Buyer3
Model	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP
Local Learning
On-time	0.6852	0.7063	0.5985	0.6846	0.7962	0.8088	0.7663	0.8088	0.5544	0.5324	0.3754	0.4705
Delays	0.8534	0.8600	0.8182	0.8578	0.8345	0.8417	0.8107	0.8338	0.8954	0.8969	0.8754	0.8866
Centralised Learning (ANN / CNN1D / LogReg / MLP)
On-time	0.7226 / 0.7494 / 0.5949 / 0.7264
Delays	0.8476 / 0.8585 / 0.8142 / 0.8526
The Proposed Federated Learning
On-time	0.6604	0.6745	0.6317	0.6784	0.7901	0.7963	0.7802	0.8037	0.5209	0.5403	0.5120	0.5317
Delays	0.8203	0.8347	0.7912	0.8361	0.8267	0.8379	0.8123	0.8403	0.8013	0.8271	0.7913	0.8079
Dataset	C-Buyer1				C-Buyer2				C-Buyer3
Model	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP	ANN	CNN1D	LogReg	MLP
Local Learning
On-time	0.7924	0.7955	0.7489	0.8014	0.6944	0.6938	0.6584	0.6784	0.6525	0.6602	0.5406	0.6151
Delays	0.7917	0.7924	0.7451	0.8043	0.8013	0.8088	0.7749	0.8032	0.7942	0.8065	0.7408	0.7916
Centralised Learning (ANN / CNN1D / LogReg / MLP)
On-time	0.6926 / 0.7157 / 0.5223/ 0.7098
Delays	0.7833 / 0.7988 / 0.6788 / 0.7970
The Proposed Federated Learning
On-time	0.7301	0.7501	0.6103	0.7763	0.6991	0.6993	0.6132	0.6779	0.6452	0.6673	0.5317	0.6203
Delays	0.7217	0.7434	0.6079	0.7601	0.7923	0.8147	0.7246	0.7821	0.7883	0.8196	0.7231	0.7634

Convolutional Neural Network with one dimensional kernels (CNN1D) consistently outperforms other three models for all buyers and in all cases. CNN1D can achieve more accurate results because CNN1D is a kernel based model which can extract a variety of features by multiple kernels with different parameters in each kernel. On the other hand, LogReg is the worst for all buyers and in all cases.

All models perform better on pre-Covid19 datasets than post-Covid19 datasets. This is expected as post-Covid19 ordering patterns are more irregular making it difficult to derive consistent inference during this period, as shown in Figure 2(d–f).

All models, in all cases, are able to predict delayed orders with more success than on-time orders (see Table 7). Consider NC-Buyer3, where the F-score for delayed orders is over 0.87, while the F-score for on-time orders is lower than 0.56. This is because there is a class imbalance towards delayed orders, which is most prominent in the case of NC-Buyer3, with 78% of its orders being delayed. Most notably during centralised learning, the F-score s are 0.81 for delayed orders and lower than 0.75 for on-time orders pre-Covid19, while they are 0.78 and 0.72 post-Covid19, except results achieved by LogReg.

Comparing centralised and local learning is not straightforward as performance is impacted by multiple factors. However, important lessons emerge.

During the more stable, pre-Covid19 period, centralised learning for buyers 1 and 3 performs better than these buyers learning using their own local dataset, whereas the opposite is true for buyer 2. However, note that in this case, datasets from buyers 1 and 3 are smaller and more imbalanced than buyer 2. Hence companies with small and imbalanced datasets benefit from pooling data together than using their own datasets. Companies that already have a longer order book with a supplier, seem to be better off using their own predictions.

Interestingly, the same observation holds post-Covid19 datasets, where centralised learning is better than local learning for buyers 2 and 3, both of whom have smaller and imbalanced datasets, but buyer 1 is better off using its own larger dataset.

In both of these cases, the dataset of one buyer is more ‘specialised’ in the specific ordering relationship with its own suppliers, and additional data from other buyers add noise, making its predictions worse. This points to a potential ‘sweet spot’ where the amount of data samples and data imbalance will determine whether a company is better off using its own dataset for predictive analysis or pool its data together with others.

In the next section we study the results of the FL approach.

4.5.2. Federated learning achieves comparable results

Expectedly, the FL approach is slightly worse than local learning for all buyers but surpasses centralised training under certain conditions. This finding has been reported by many existing works (McMahan et al. 2017; Yu, Bagdasaryan, and Shmatikov 2020), who advocated FL to cater to those use cases where local training data is not sufficient, and centralised training is not possible due to data sharing concerns. One must thus explore how FL performance can be maximised in these cases. Our results offer similar insights.

Although FL is not the highest performing approach, its performance approaches remarkably close to that of local and centralised learning (except Logistic Regression based FL). The maximum difference between FL and local learning; and between FL and centralised learning are 0.0670 and 0.0634, respectively, while their averaged differences are 0.0212 and 0.0148 Figure 5. In Figure 5 the dark bars illustrate the differences between FL and local learning and the red bars illustrate the differences between FL and centralised learning, where positive values indicate that FL is better than local learning or/and centralised learning, and vice versa.

Figure 5. The differences of achieved $F_w$-Score between FL and local learning and between FL and centralised learning for all datasets. A bar chart showing the performance differences of FL and local learning and of FL and centralised learning on all eight datasets.

Performance evaluation of FL conducted using Artificial Neural Networks (FL-ANN), Convolutional Neural Networks (FL-CNN1D), Logistic Regression (FL-LogReg), and Multi-Layer Perceptron (FL-MLP) during balanced training are shown in Figure 6 where the x-axis represents the number of rounds, R, and the y-axis represents $F_w$-Score . Here, all models provide more accurate predictions on pre-Covid19 datasets (i.e. NC-Buyer1, NC-Buyer2 and NC-Buyer3) consistently arriving or exceeding $F_w$-Score scores near 0.80.

Figure 6. Results of the FL approach achieved by the balanced training. A line-based plot that consists of three subplots corresponding to three buyers showing the performances of different machine learning algorithm-based FL approaches achieved by the balanced training.

The difference between pre-Covid19 and post-Covid19 datasets is smaller for buyer 1 while they are larger for buyers 2 and 3. The reason for this is more scattered distribution of delayed deltas from buyer 1 shown in Figure 2(a) and (d) compared to buyers 2 and 3.

One can observe how FL based models improve in performance, as the results of inter-buyer information sharing begins to take effect. This is quite noticeable when FL is undertaken on imbalanced datasets. Here, both FL-ANN and FL-CNN1D take longer to converge (Figure 6). Consider the more imbalanced pre-Covid19 dataset of buyer 3, where convergence takes longer than on the less imbalanced post-Covid19 dataset of buyer 3. This performance degradation is most probably caused by orders in the sample being predicted to the over-represented class. For instance, if there are no delayed orders being correctly predicted as delayed orders, TP would be equal to 0; resulting in $Precision=\frac{TP}{TP+FP}$ and $Recall=\frac{TP}{TP+FN}$ being 0. If Precision and Recall were both 0, the F-Score would be infinite. However, interestingly, both FL-ANN1 and FL-CNN1D recover after 2000 rounds as they begin to learn more information from other datasets, providing evidence on how an imperfect model starts to improve due to FL based training.

Exploring algorithmic approaches, consistent with centralised and local learning, FL-CNN1D performs better than FL-ANN in most cases. FL-CNN1D achieves 0.775 and 0.700 $F_w$-Score while FL-ANN obtains 0.760 and 0.685 $F_w$-Score on NC-Buyer1 and C-Buyer3, respectively. For a small number of datasets (i.e. C-Buyer2 and NC-Buyer3), FL-ANN and FL-CNN1D achieve similar results. Among FL-CNN1D and FL-MLP, the former performs better on small datasets (i.e. C-Buyer2 and NC-Buyer3) while the later achieves higher $F_w$-Score on large NC-Buyer1 and NC-Buyer2. FL-LogReg is the worst algorithm in all cases.

For FL-ANN and FL-CNN1D in imbalanced training, recall that two types were pursued: one where the number of model results on localised individual samples that were sent to the central server sequentially varied in size but individual sample sizes were heterogeneous (Type-1), and one where both numbers of model results and individual sample sizes varied (Type-2). Results are shown in Figure 7. In Figure 7, red and black lines represent results achieved by FL-CNN1D and FL-ANN respectively while blue and green lines represent FL-LogReg and FL-MLP. Solid lines represent models trained by imbalanced Type-2 and dashed lines are models trained by imbalanced Type-1.

Figure 7. Results of the FL approach achieved by the imbalanced training. Noting that (c) does not show the results achieved by FL-LogReg due to being Nan (Table 6). A line-based plot that composes of six subplots corresponding to six datasets showing the performances of different machine learning algorithm-based FL approaches achieved by the imbalanced training.

For the two imbalanced training types, it appears all models trained by Type-1 are more stable. For example, on all three post-Covid19 datasets, achieved $F_w$-Score s by Type-1 are all around 0.750 while, for Type-2, differ from buyer to buyer. Hence one can conclude that heterogeneity in the size of datasets used to train local models has less negative impact on the performance of the global model, than training local models on imbalanced datasets.

Overall, imbalanced training helps all models provide slightly better predictions, compared to balanced training. For example, the highest $F_w$-Score achieved by balanced training on C-Buyer1, C-Buyer2 and C-Buyer3 are around 0.750, 0.750, 0.700 while by imbalanced training are approximately 0.770, 0.750 and 0.750, respectively. This is good news, as this mode of training also facilitates real world scenario where datasets and their balance levels differ from company to company.

5. Conclusions and managerial implications

An increasingly volatile global economy has made researchers question the long-standing efficiency focus in supply chains, and considered how supply chains that are frequently exposed to shortages can be made viable (Ivanov and Dolgui 2022a). Recent conceptualisations of supply chain viability go beyond the notion of resilience, and propose a holistic framework encompassing preparedness, anticipation, recovery and sustainability (Ivanov and Dolgui 2020). Our work directly contributes to these efforts by providing a privacy preserving, data-driven risk prediction method to enable organisations anticipate supply disruptions collectively.

Extant research in data-driven supply chain risk prediction focused on siloed approaches in which companies attempt to examine risk alone, despite the supply chain being an interdependent system. Although data-sharing has long been proposed as a means to collectively assess supply chain risk, practical approaches to do so have not been successful, largely due to concerns about confidentiality. In this paper, we formulated new approach to collective risk prediction based on a new development from the field of Artificial Intelligence called Federated Learning. FL can enable information sharing without sharing raw data itself, where each supply chain member shares their predictive model parameters instead. The approach was tested using an empirical case study in the maritime engineering sector, in which buyers who shared a number of their suppliers developed delivery delay prediction models.

Recognising that companies will have heterogeneous ordering patterns, different levels of exposure to risk, and specialised relationships with their suppliers, we then asked the question ‘When is it better for companies to learn from their own historical data, and when should they undertake a federated learning approach?’ To explore this question, two benchmarks were created: local learning during which each member uses its own data, and centralised learning during which a common model is created from all members' data.

Our results indicate the following:

• The performance difference between a federated approach and benchmarks are minimal, hinting that FL is a viable approach for supply chain data-sharing in contexts where privacy preservation hinders collective learning.

• FL enables supply chain members with small and imbalanced datasets to tap into collective information, and improves their predictive performance. On the other hand, members with large datasets are better off developing their own models. However, for the greater good of the supply chain, they should bear responsibility in participation in collective learning.

• Expectedly, federated approaches worked better during the pre-Covid19 period, when variance in delivery delays were smaller than during the post Covid19 period. However, interestingly, what matters more to the success of federated learning is dataset size and dataset imbalance.

• Algorithm choice is a small yet significant factor in delay prediction.Among the four machine learning algorithms, used, namely Artificial Neural Network (ANN), Convolutional Neural Network with one dimensional kernels (CNN1D), Logistic Regression (LogReg), and Multi-Layer Perceptron (MLP), CNN1D on average performed 0.34%, 1.04% and 16.90% better than MLP, ANN and LogReg across all cases.

Our research has limitations, which in turn pose avenues for future research. The case study based nature of the study restricts our findings to the domain of inquiry, hence more research needs to be conducted to explore the efficacy of FL in other industrial domains. In particular, trials on supply chains with different underlying dynamics, and frequencies of disruptions need to be studied. Exploration of different model update mechanisms such as weighted averages could be explored. This in turn would bring on questions of model bias by those members of the supply chain who have disproportionately more data on certain supply chain members.

While we have concluded that collective risk prediction is better for companies that suffer from small dataset sizes and data imbalance, the exact trade offs between self-learning and collective learning still needs to be investigated. Thus, an important question that remains is how can companies know when to tap into the collective knowledge, and when to trust their own datasets?. It is also worth noting that large complex supply chains will involve frequent supplier changes, hence, although a buyer might not immediately benefit from collective learning approaches by joining in the FL approach, it might do so in the future. One means of persuading buyers with already existing large datasets to join in a federated approach may include payment based incentivisation. More research needs to be conducted in how companies may be incentivised to take part in collective learning.

Moreover, while we have applied FL in a use case on supply chain delivery delays, in principle FL may present opportunities for addressing other collective supply chain learning problems, which in turn would help improve coordination and planning. These include but are not limited to demand forecasting, and inventory prediction, as well as potentially finding remedies for long standing challenges such as demand amplification and shortage gaming.

Lastly, we also see an opportunity for this work to inform supply network reliability optimisation (e.g. Brintrup and Puchkova 2018; Ivanov 2022b) as the methods presented herein allow a collectively informed means to quantify network risk.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data Availability Statement

Due to the commercially sensitive nature of this research, supporting data is not available.

Additional information

Funding

This work was supported by the Engineering and Physical Sciences Research Council (grant number EP/W019868/1).

Notes on contributors

Ge Zheng

Dr. Ge Zheng currently is a Research Associate in the Supply Chain AI Lab at the Institute forManufacturing (IfM), Department of Engineering, University of Cambridge, UK. She received her PhD degree with full PhD scholarship in the Department of Computing and Informatics at the University of Bournemouth, UK, and MSc degree with Academic Excellence International Masters Scholarship in Electronic Engineering from the School of Computer Science and Electronic Engineering at the University of Essex. Her interested areas involve supply chain risk prediction, pattern recognition and/or classification, intelligent transportation systems, and healthcare applications.

Lingxuan Kong

Lingxuan (Leo) Kong is a PhD student from the Supply Chain AI LAB (SCAIL) Group in Institute of Manufacturing at Department of Engineering, University of Cambridge. His current research area is learning-based risk prediction methods for supply chain financing. Lingxuan join the SCAIL Group in 2021. He received his Bachelor of Engineering (Honours) degree from Australian National University. He majored in mechatronics system in his bachelor's degree. His previous research projects during bachelor study mainly focus on the distributed clustering algorithm and graph partitioning methods. These algorithms can be applied to multi-robot system and data processing problems. He published these proposed algorithms in International Conference on Intelligent Robotics and Systems (IROS).

Alexandra Brintrup

Prof. Alexandra Brintrup is Professor of Digital Manufacturing and head of the Supply Chain AI Lab at the Institute for Manufacturing (IfM), Department of Engineering, University of Cambridge, UK. She has a PhD in Artificial Intelligence, MSc in Applied Maths and Computing and a BEng in Manufacturing Engineering. She specialises in distributed negotiation, machine learning, autonomous systems, and nature-inspired optimisation in complex supply networks.