Comprehensive evaluation of the intelligence levels for unmanned swarms based on the collective OODA loop and group extension cloud model

Abstract

Considering the increasing complexity of application scenarios, the evaluation of the intelligence levels for unmanned swarms has attracted scholarly attention. However, the existing evaluation studies cannot suitably reflect the intelligence of unmanned swarms, and they rarely provide a specific evaluation process. By introducing the collective intelligent behavior model for unmanned swarms based on the collective Observe–Orient–Decide–Act (OODA) loop, this study constructs a comprehensive evaluation index system that can systematically reflect the overall intelligence of unmanned swarms in complex scenarios. Considering the fuzziness and randomness in the processes of weight calculation and level evaluation, this study proposes a comprehensive evaluation method of the intelligence levels for unmanned swarms based on a group extension cloud model. This method calculates the weights of various evaluation indexes at different layers by adopting the group extension analytic hierarchy process. Moreover, it obtains the comprehensive evaluation conclusion of the intelligence levels for unmanned swarms by adopting the cloud model. Applying the proposed method, this study evaluates two specific types of unmanned aerial vehicle swarms. The results show that the proposed method can more flexibly and accurately evaluate the intelligence levels of unmanned swarms than the previous fully qualitative methods.

KEYWORDS:

• Comprehensive evaluation
• intelligence levels
• unmanned swarms
• collective oODA loop
• group extension cloud model

1. Introduction

An unmanned swarm is an artificial swarm system comprising a certain number of homogeneous or heterogeneous unmanned vehicles and other auxiliary units (Bai et al., 2016) and has become an important development direction for unmanned systems. As a combination of unmanned systems and swarm intelligence, an unmanned swarm is not a simple aggregation of multiple simple unmanned vehicles. Nonetheless, it makes them cooperate closely to form a self-organised and highly stable distributed system to stimulate individual and swarm intelligence (Sun et al., 2020). Owing to their multiple advantages (including intellectualisation, collectivisation, and unmanned operation), unmanned swarms have wide application prospects in complex scenarios, such as cooperative operation, logistics transmission, industrial production, and emergency response (Liang et al., 2018). The basic forms of unmanned swarms mainly include unmanned aerial vehicle (UAV), unmanned ground vehicle, unmanned sea vehicle, unmanned underwater vehicle, and some other hybrid unmanned swarms (Liu et al., 2017). Currently, cooperative operation is the most important application scenario for unmanned swarms, and the UAV swarm is the primary development form (Jia et al., 2019). Countries all over the world are committed to the research and development of UAV swarms, and several representative projects, such as Gremlins (Daniel, 2016), Offensive Swarm-Enabled Tactics (DARPA Public Affairs, 2016), Perdix (Zhang, 2015) and Low-Cost UAV Swarming Technology (Office of Naval Research, 2015), have emerged.

Previous studies have demonstrated that the effective evaluation of the capability, effectiveness, or performance for unmanned systems will play a leading role in technology. Recently, the intelligence realisation of unmanned swarms has begun to receive increasing attention (Zhou et al., 2020). Every type of intelligence is acquired progressively (Yang, 2018). The evaluation of the intelligence levels for unmanned swarms has become an important part of studying and understanding their performance (Dai et al., 2020). First, it helps policy makers to scientifically measure the intelligence levels of the existing unmanned swarms and point out the right way to develop and realise their intelligence from low to high when formulating development plans and polices for unmanned swarms and its related industries, respectively. Second, it enables developers and designers to reasonably determine whether the expected intelligence level is achieved. Third, it helps users to systematically describe the intelligence requirements for unmanned swarms and effectively evaluate the intelligence level of multiple similar unmanned swarms.

Although the evaluation of the intelligence levels for unmanned swarms is still a new research problem, there are many alternative methods that can be used to approximately evaluate the intelligence levels of unmanned swarms. However, most previous related studies only provided some fully qualitative evaluation methods based on different evaluation criteria and did not provide a specific evaluation process. A cloud model is a cognitive model that investigates the uncertainties in the transformation between qualitative concepts and quantitative values and is not disturbed by fuzziness and randomness (Harrod, 2009; Zhao & Song, 2016). Considering the development of collective intelligence, the constructed evaluation index cannot suitably reflect the overall intelligence of unmanned swarms because the previous evaluation model cannot suitably describe the cooperation between individuals in an unmanned swarm. The commonly used weight determination method in previous studies is still the traditional analytic hierarchy process (AHP). This method requires that the elements in the importance judgment matrix are exact numbers, and it does not consider the complexity and uncertainty of each index (Li & Qiu, 2019). Moreover, owing to the differences in knowledge structure, preference, and cognitive ability of different decision-makers, the relative importance of the evaluation indexes often deviates from the objective reality. Therefore, it is difficult to obtain more accurate weights using the traditional AHP.

In this study, we propose a quantitative evaluation solution for the intelligence levels of unmanned swarms based on the collective Observe-Orient-Decide-Act (OODA) loop and group extension cloud model. The contributions of this study are enumerated below.

• The constructed intelligence evaluation index system covers the collective observation, orientation, decision, and action competencies. It can more suitably and comprehensively reflect the overall intelligence of unmanned swarms owing to the introduction and reference of the collective intelligent behavior model of unmanned swarms based on the collective OODA loop.

• The proposed group extension AHP method replaces the exact numbers with interval numbers, integrates the judgments of multiple decision experts, and it does not require a consistency check. It can more flexibly, accurately, and conveniently determine the weights of the constructed intelligence evaluation index system than the traditional AHP method because of the integration of the extension and group decision-making methods.

• The designed comprehensive evaluation method based on the cloud model simplifies the measurement and quantification of intelligence evaluation indexes. Furthermore, it fully describes the fuzziness and randomness in the intelligence level evaluation process of unmanned swarms. It can obtain more objective and accurate evaluation results owing to the transformation from a qualitative evaluation to a quantitative evaluation.

The rest of this study is organised as follows. Section 2 reviews the related studies. Considering Section 3, a comprehensive intelligence evaluation index system is constructed for unmanned swarms in complex scenarios by introducing a collective intelligent behavior model based on the collective OODA loop. Section 4 describes the basic principles and detailed design processes of the comprehensive evaluation method for the intelligence levels of unmanned swarms based on the group extension cloud model. To validate the proposed method, a case study is presented in Section 5. Regarding Section 6, the conclusions of this study are presented, and future studies are recommended.

2. Related studies

Researchers have been paying attention to the autonomy level evaluation of unmanned systems for a long time. Because unmanned swarms are also unmanned systems, most of the previous studies included the evaluation of unmanned swarms. Although autonomy and intelligence are two different concepts, autonomy level evaluation methods can also be used to evaluate their intelligence level. This is because the autonomy of an unmanned system is the external manifestation of its intelligence (Duan & Qiu, 2018; Fan & Zhang, 2017).

Early evaluation studies on the autonomy level of unmanned systems were mainly achieved through a set of extremely simple qualitative languages. To investigate the autonomous operation capability of unmanned systems, the Department of Defense (DoD) of the United States has proposed the definition of autonomy control level (ACL) and has successively released multiple versions of the unmanned system roadmap. Considering the earlier versions (Office of the Secretary of Defense, 2000, 2002, 2005), UAV autonomy is classified into ten levels. These include: remotely guided, real-time health diagnosis, adaptation to failures and flight conditions, onboard route re-planning, group coordination, group tactical re-plan, group tactical goals, distributed control, group strategic goals, and fully autonomous swarms. Regarding this classification, levels one to four describe monomer autonomy, and levels five to ten describe swarm autonomy. Although other research institutions later updated the autonomous capability description of UAVs from different perspectives, they generally did not deviate from the description scope.

Several qualitative evaluation methods are based on multiple evaluation indexes. Draper (Goodrich et al., 2003) divided unmanned system autonomy into four levels based on three dimensions: motion control, task planning, and state understanding. The National Aeronautics and Space Administration (NASA) proposed a simplified ACL that classified UAV autonomy into six levels: remote control, simple automatic operation, remote operation, highly automated (semi-autonomous), and fully autonomous and collaboration (Larry et al., 2005). One intuitive data is used to show the autonomy level, which indicates the amount of time the person controls during the flight course of the UAV. The Air Force Research Laboratory (AFRL) further refines the evaluation requirements of the ten-level ACL by introducing the classical OODA loop (Clough, 2002). Based on the actual development of the UAV autonomy in China, Chen et al. (2011) proposed a more suitable UAV ACL based on four evaluation indexes: autonomous function, autonomous type, intelligent attributes, and communication level.

Some institutions have also proposed few generic evaluation methods. The National Institute of Standard and Technology (NIST) proposed a generic autonomy levels for unmanned systems (ALFUS) framework that comprehensively evaluated the autonomy level based three dimensions: task complexity, environment complexity, and human independence (Huang et al., 2005). These three dimensions contain series of influencing factors and a set of measurement methods. The ALFUS framework model can effectively reflect the task execution ability of unmanned systems; nonetheless, the model evaluates unmanned system autonomy completely from an external perspective. Furthermore, there are series of problems that cannot be ignored in practical applications. First, the process of measuring the factors in each dimension and allocation of weights is not provided. Second, the evaluation of the autonomy level is undetermined. The Shenyang Automation Research Institute proposed a universal cobweb model (Wang & Liu, 2012) to evaluate the autonomy level of unmanned systems. The model has strong generality; however, it has the same problems as the ALFUS framework model in practical applications. A comparison of these autonomy-level evaluation methods is shown in Table 1.

Table 1 Comparison of various autonomy level evaluation methods

			Evaluation Dimension
Proposer	Evaluation Object	Evaluation Metric	Ob	Or	D	A	C	T	E	I
Draper	UMS	4	√		√	√
NASA	UAV	6	√		√		√
AFRL	UAV	11	√	√	√	√	√			√
NIST	UMS	10						√	√	√

“UMS” represents unmanned systems. “Ob”, “Or”, “D”, and “A” represent observation, orientation, decision, and action, respectively. “C” represents cooperation. “T”, “E”, and “I” represent task, environment, and interaction, respectively.

Recently, few researchers have proposed special evaluation studies on the intelligence levels of unmanned swarms. Considering the development of marine robots and the history of artificial intelligence, Zhang et al. (2017) divided swarm intelligence into six levels: formation control, task planning, cooperation, task re-planning, cooperative exploration, and duty. The level classification standards are common to other fields; however, the study does not provide detailed explanations for each level. Moreover, there is no further study on how to evaluate the intelligence level of unmanned swarms. Wu and Zhou (2020) proposed a generic intelligence level measurement model for unmanned swarms, and the model is composed of three dimensions: scene complexity, system adaptability, and system effectiveness. The model has a strong generality; however, it also has the same problems as the ALFUS framework model in practical applications. In addition, some researchers have proposed some fully quantitative methods (Chmait, 2017; Chmait et al., 2015; Commuri et al., 2004; Dai et al., 2020); nonetheless, these methods can only evaluate the intelligence of a certain dimension of unmanned swarms.

In summary, most previous studies are still at the preliminary cognition stage and only provide some qualitative evaluation methods based on different evaluation criteria. However, they do not provide a specific quantitative evaluation process. Moreover, it is difficult for them to reasonably reflect the overall intelligence of unmanned swarms owing to the lack of relation to the corresponding reasonable theoretical models and the over-generalization of evaluation objects. Several researchers have proposed operable evaluation methods for the autonomy level of unmanned systems through mathematical models and quantitative language (Shuai et al., 2016; Yan & Tang, 2012; Yang & Zhang, 2009). These methods can also be used to evaluate the intelligence level of unmanned swarms. However, they still need to be further improved because they all use the traditional AHP when determining the weights of the evaluation indexes. There are recently some emerging related evaluation methods, such as expertise-based bid evaluation with generalised comparative linguistic ELECTRE III (Chen et al., 2021a), online review analysis based on large-scale group decision-making (Zsca et al., 2021), power-average-operator-based hybrid multi-attribute online recommendation model (Chen et al., 2021b), and third-party selection approach based on hybrid-information multi-criteria decision making and cumulative prospect theory (Li et al., 2018). Although these advanced methods aim at different objects and have not yet been used in this field, they are not available to improve the comprehensive evaluation process for the intelligence levels of unmanned swarms in the future.

3. Construction of an intelligence evaluation index system for unmanned swarms

3.1. Collective intelligent behavior modelling based on the collective OODA loop

The construction of an intelligence evaluation index system for unmanned swarms should be closely related to the collective intelligent behavior model of unmanned swarms. The traditional collective intelligent behavior modelling methods are realised by simulating the collective intelligent behaviors of simple social biological swarms (Barbara, 2002). A typical representative is the Boids model proposed by Reynolds (1987). This modelling method usually needs to make appropriate simplification, abstraction, and assumptions for individual behavior ability and rules in advance (Amanda, 2007; Fidel et al., 2012). Nonetheless, it is only suitable for some relatively simple scenarios owing to the limitation of individual ability. Considering some complex scenarios, the intelligent behavior of unmanned monomers is often modelled as a classical OODA loop (Boyd, 1996). The loop was proposed by Boyd in 1996 and was used to describe the entire process of military system decision-making. However, it only has a good guiding role in analyzing the intelligent behavior of unmanned monomers and is insufficient for describing the cooperation and interaction between multiple unmanned vehicles.

In summary, the traditional collective intelligent behavior modelling method based on simple rules and the classical OODA loop cannot be applied to describe the collective intelligent behavior of unmanned swarms in complex and changeable battlefield environments. Therefore, this study introduces a collective intelligent behavior modelling method for unmanned swarms based on the collective OODA loop as shown in Figure 1. The collective OODA loop is a type of collective extension for the classical OODA loop. From a micro perspective, each unmanned agent has a highly intelligent behavior, and its intelligent behavior can be regarded as a classical OODA loop. From a macro perspective, the collective intelligent behaviors of an unmanned swarm composed of multiple unmanned agents with highly intelligent behaviors and other agents can be regarded as a collective OODA loop. Therefore, the overall intelligence of unmanned swarms can be realised and analyzed based on the collective OODA loop.

Figure 1 Collective intelligent behavior model of unmanned swarms based on the collective OODA loop

Considering the collective intelligent behavior model based on the collective OODA loop, unmanned swarms complete a given target task through collective observation, orientation, decision, and action functions or competencies between the unmanned agents and between unmanned agents and other agents. These four collective functions support an unmanned swarm having a stronger operational competence in each stage of task execution to promote a significant improvement in the overall task competence. The collective observation competence, for example, can widen the perception range of the whole system. Moreover, the perception types are more diversified, and the individual perception error or loss can be corrected and compensated. When we regard the whole unmanned swarm as a unified whole, it is equivalent to forming a virtual agent from a global perspective. Considering the processes of the collective observation, orientation, decision, and action, the virtual global agent makes a comprehensive trade-off from the overall layer of the swarm based on the global perspective. This ultimately reflects that the behavior of individual unmanned agents in the swarm is not necessarily optimal; however, the effect of the overall behavior of the swarm is optimal. This reflects swarm intelligence. The virtual global agent is not a top-down global control, and its role is not to replace the unmanned agents in the swarm to make decisions. Nonetheless, it is to make individual decision-making with its assistance to have a more macro perspective. Thus, it assists unmanned agents to obtain and process global information and coordinate within the swarm.

3.2. Construction of the evaluation index system based on the collective OODA loop

It is indisputable that the evaluation of the intelligence level of unmanned swarms is a multi-objective and -layer systematic project. To establish a reasonable and comprehensive evaluation index system, we need to consider various factors that have a significant impact on the intelligence level of unmanned swarms as comprehensively as possible using the hierarchical analysis process. Based on the collective intelligent behavior model of unmanned swarms based on the collective OODA loop, it can be concluded that the overall intelligence of unmanned swarms mainly depends on four basic collective competencies: collective observation, orientation, decision, and action. These four basic collective competencies can be described and analyzed from different perspectives. For example, considering the inherent competencies of unmanned swarms, collective observation competence mainly includes radar detection, photoelectric detection, and electronic reconnaissance competencies. However, these inherent competencies are only some basic conditions that generate the intelligence of unmanned swarms, and they do not play a decisive role in evaluating the intelligence level of unmanned swarms.

The intelligence of unmanned swarms can be more reflected in whether it has collective intelligence competencies that are mainly obtained by simulating, extending, and expanding the biological swarm or even human team intelligence (Donna, 2017), such as collective perception, navigation, recognition, assessment, diagnosis, prediction, planning, etc. These collective intelligent competencies can distinguish the intelligence level of unmanned swarms, and they may involve more specific classifications. By further sorting out some specific collective intelligent competencies that can reflect the intelligence of unmanned swarms based on the four basic collective competencies described in the collective intelligent behavior model of unmanned swarms based on the collective OODA loop, a multi-dimensional hierarchical intelligence evaluation index system of unmanned swarms can be constructed as shown in Figure 2.

Figure 2 Intelligence evaluation index system of unmanned swarms based on the collective OODA loop

4. Comprehensive evaluation method based on the group extension cloud model

4.1. Group extension AHP

The extension theory is widely applied to solve comprehensive evaluation problems aimed at multiple objectives (Yang & Cai, 2013). The extension set is a quantitative tool used to describe the variability of things in the extension methods. This can determine the degree of things belonging to a certain set based on their quantity values about a certain characteristic. The group extension AHP introduces group decision-making and extension theories into the traditional AHP, replaces exact numbers with interval numbers, and gathers the judgment of multiple experts when constructing the importance judgment matrix of the evaluation indexes. The processes for determining the weights of the evaluation indexes by the group extension AHP are demonstrated below.

Step 1: Construction of the comprehensive extension judgment matrix. The construction of the comprehensive extension judgment matrix is a formal process that transforms various original source data into data that can be directly compared. The general extension AHP (Wu et al., 2016; Zhang et al., 2014) usually invites a single domain expert to compare the relative importance of the indexes at the $k$-th layer in pairs on a certain index at its upper layer. Thereafter, all the results are combined to form an extension judgment matrix: $A=(a_{ij}{)}_{n\times n}(i,j=1,2,\cdots ,n)$. A is a positive reciprocal matrix, which indicates that all the elements obey the conditions $a_{ii}=1$ and $a_{ij}=a_{ji}^{-1}$. All the elements of A are the interval numbers. $a_{ij}=⟨a_{ij}^{-},a_{ij}^{+}⟩$ indicates its element in the $i$-th row and $j$-th column. $a_{ij}^{-}$ and $a_{ij}^{+}$ indicate the lower and upper endpoints of the element, respectively. The group extension AHP invites $T(T\ge 2)$ domain experts to construct a comprehensive A using the group decision-making method. To avoid the subjective preference of the invited experts, the value of T should theoretically be as large as possible. This indicates that the invited experts should as many as possible. However, the actual number needs to be determined in combination with other conditions, such as the authority, representativeness, accessibility of experts, and consulting costs. The extension judgment matrix provided by the $t$-th domain expert is indicated as $A^t=(a_{ij}^t{)}_{n\times n}(i,j=1,2,\cdots ,n)$ and its element in the $i$-th row and $j$-th column is indicated as $a_{ij}^t=<a_{ij}^{t-},a_{ij}^{t+}>$. Considering the comprehensive A constructed by T domain experts, its element in the $i$-th row and $j$-th column can be calculated as: (1) $a_{ij}=⟨a_{ij}^{-},a_{ij}^{+}⟩=⟨\sqrt[T]{\prod _{t=1}^T{a}_{ij}^{t-}},\sqrt[T]{\prod _{t=1}^T{a}_{ij}^{t+}}⟩$(1) Step 2: Calculation of the extension weight vector. The lower and upper endpoint matrices $A^{-}=(a_{ij}^{-}{)}_{n\times n}$ and $A^{+}=(a_{ij}^{+}{)}_{n\times n}$, respectively, can be easily obtained based on these extension interval elements. The processes for calculating the extension weight vector of $A=[A^{-},A^{+}]$ satisfy the consistency condition based on the following.

First, the normalised eigenvectors $x^{+}$and $x^{-}$ with positive components corresponding to the maximum eigenvalues of A⁺ and A^- can be easily calculated.

Second, the values of the two intermediate variables $l$ and $m$ can be calculated as: (2) $\{\begin{array}{c}l=\sqrt{\sum _{j=1}^n{\left(\sum _{i=1}^n{a}_{ij}^{+}\right)}^{-1}}\\ m=\sqrt{\sum _{j=1}^n{\left(\sum _{i=1}^n{a}_{ij}^{-}\right)}^{-1}}\end{array}$(2) where $l$ and $m$ meet condition $l{x}_i^{-}\le m{x}_i^{+}$. If $0\le l\le 1\le m$, it can be concluded that the constructed A is consistent.

Finally, the extensive weight vector corresponding to matrix A can be calculated as: (3) $w=(w_1,w_2,\cdots ,w_{n_k}{)}^T=⟨l{x}^{-},m{x}^{+}⟩$(3) where $w_{n_{\text{k}}}$ indicates the extension interval weight of the $n$-th element at the $k$-th layer on a certain factor at its upper layer.

Step 3: Calculation of the single-ordering weights. Considering the hierarchical evaluation index system, each element at the $k$-th layer has a weight expressed by the interval number $w_i^k=⟨l{x}^{-},m{x}^{+}⟩=⟨w_i^{k-},w_i^{k+}⟩(i=1,2,\cdots ,n_k)$, and the relative importance of the two elements $w_i^k$and $w_j^k(i\ne j)$ at the$k$-th layer can be calculated as. (4) $V(w_i^k\ge w_j^k)=\frac{2(w_i^{k+}-w_j^{k-})}{(w_i^{k+}-w_i^{k-})+(w_j^{k+}-w_j^{k-})}$(4) When a fixed number $j$ meets the conditions that $w_i^k\ge w_j^k\ge 0$ for $\mathrm{\forall }i=1,2,\cdots ,n_k,i\ne j$, $p_1^k=1$, and $p_i^k=V(w_i^k\ge w_j^k)(i=1,2,\cdots ,n_k;i\ne j)$, the single ordering weights of $n_k$ elements at the $k$-th layer on the $h$-th element at its upper layer can be normalised as: (5) $p_h^k=(p_{1h}^k,w_{2h}^k,\cdots ,p_{n_{k}h}^k{)}^{\text{T}}$(5) Step 4: Calculation of the total ordering weights. Considering the corresponding $p_h^k$ of each element at the $k$-th layer on all elements at its upper layer, an $n_k\times n_{k-1}$ matrix $p^k=(p_1^k,p_2^k,\cdots ,p_{n_{k-1}}^k{)}^{\text{T}}$can be obtained. If the weight sort of the elements at the $k$−1th layer on the total objective is indicated as $w^{k-1}=(w_1^{k-1},w_2^{k-1},\cdots ,w_{n_{k-1}}^{k-1}{)}^{\text{T}}$, then the weight sort of the elements at the $k$-th layer on the total objective can be defined as: (6) $w^k=(w_1^k,w_2^k,\cdots ,w_{n_k}^k{)}^{\text{T}}=p^k{w}^{k-1}$(6) Finally, it can be calculated as $w^k=p^k{p}^{k-1}\cdots p^3{w}^2$.

4.2. Comprehensive evaluation method based the cloud model

(1) Cloud model

The cloud model is a new cognitive model based on the fuzzy set and probability theories (Li et al., 2009), and it can effectively reflect the fuzziness and randomness of things or human knowledge. The model skilfully integrates the fuzziness and randomness of things themselves and uses fixed mathematical expressions to describe them uniformly. Currently, the cloud model has developed various forms, such as normal, triangular, and trapezoidal clouds, among which the first model is the most widely used.

The normal cloud model is indicated as triple C(E_x, E_n, H_e). Considering the triple, E_x is indicated as the expectation of the membership cloud. This is the central position of the membership cloud, which is the most representative point of a qualitative concept. E_n (also known as entropy) indicates the width of the membership cloud, which is a measure of the uncertainty of the qualitative concept. H_e (also known as super entropy) indicates the variance of the membership cloud, which is a measure of the uncertainty of the entropy, and it reflects the dispersion degree of the membership cloud.

(2) Cloud model algorithm

The cloud model algorithm is also referred to as a cloud generator, and it can effectively establish the relationship between qualitative and quantitative attribute values. The cloud generator mainly includes a forward generator, reverse generator, and index approximation method. The algorithm generating two-dimensional points of the cloud is known as a forward generator. One point in the cloud is referred to as a cloud droplet and is indicated as a drop(x_i, u_i). Considering the three mathematical characteristics of the cloud model, the implementation algorithm for the normal cloud is described below.

Step 1: Generate a normal random number with E_n as the expectation and H_e as the variance as expressed below: (7) ${\sigma }_i=G(E_n,H_e)$(7) Step 2: Generate a normal random number with E_x as the expectation and ${\sigma }_i$ as the variance: (8) $x_i=G(E_x,{\sigma }_i)$(8)

Step 3: Calculate the membership degree: (9) ${\mu }_i=\text{exp}[-{(x_i-E_x)}^2/2{\sigma }_{n1}^2]$(9) Obtain a cloud droplet with ${\mu }_i$ as the membership degree.

Step 4: Repeat Steps 1 to 3 until n cloud droplets are generated.

(3) Cloud model operation rules

A new cloud model can be constructed by adopting the fitting operation rules for two cloud models. The fitting operation rules of the mathematical characteristics of the two normal cloud models are shown in Table 2.

Table 2 Fitting operation rules for two normal clouds

$E_x$	$E_n$	$H_e$
$E_{x1}+E_{x2}$	$\sqrt{E_{n1}^2+E_{n2}^2}$	$\sqrt{H_{e1}^2+H_{e2}^2}$
$E_{x1}\times E_{x2}$	$|E_{x1}{E}_{x2}|\times \sqrt{{(E_{n1}/E_{x1})}^2+{(E_{n2}/E_{x2})}^2}$	$|E_{x1}{E}_{x2}|\times \sqrt{{(H_{e1}/E_{x1})}^2+{(H_{e2}/E_{x2})}^2}$
$E_{x1}÷E_{x2}$	$|E_{x1}/E_{x2}|\times \sqrt{{(E_{n1}/E_{x1})}^2+{(E_{n2}/E_{x2})}^2}$	$|E_{x1}/E_{x2}|\times \sqrt{{(H_{e1}/E_{x1})}^2+{(H_{e2}/E_{x2})}^2}$

(4) Comprehensive evaluation process of reference levels based on cloud model

Considering actual applications, a comprehensive evaluation of the total objective may be obtained either by single- or multiple-dimensional indexes. The basic processes for establishing a comprehensive evaluation model of objective levels based on the cloud model (Pan et al., 2012; Zhang et al., 2019) based single-dimensional evaluation indexes are demonstrated below.

Step1: Evaluation index set establishment of the single dimension. The evaluation index set is established by adopting the hierarchical analysis process and is indicated as $U=\{u_1,u_2,\cdots ,u_m\}$, where $u_i(i=1,2,\cdots ,m)$ represents the $i$-th evaluation factor and $m$ indicates the number of the evaluation indexes.

Step 2: Weight cloud model establishment of the evaluation indexes. The weight set of the evaluation indexes is obtained by adopting the group extension AHP. It is also further transformed into a weight cloud set as: (10) $\begin{array}{rl}W& =[w_1,w_2,\cdots ,w_n]\\ & ={\left[\begin{array}{ccc}E{x}_{w1}& E{n}_{w1}& H{e}_{w1}\\ E{x}_{w2}& E{n}_{w2}& H{e}_{w2}\\ ⋮& ⋮& ⋮\\ E{x}_{wn}& E{n}_{wn}& H{e}_{wn}\end{array}\right]}^T\\ & =C_w(E{x}_w,E{n}_w,H{e}_w{)}^T\end{array}$(10) where Ex_w indicates the weight, and $E{n}_w=1/3E_w$, E_w and He_w indicate the maximum deviation from the expectation and randomness when the expert panel scores.

Step 3: Evaluation cloud model establishment of the evaluation indexes. The evaluation indexes are divided into several qualitative levels based on the actual requirements. The evaluation result set is obtained by adopting the expert rating method and is further transformed into a normal evaluation cloud set as: (11) $\begin{array}{rl}R& =[r_1,r_2,\cdots ,r_n{]}^T\\ & =\left[\begin{array}{ccc}E{x}_{e1}& E{n}_{e1}& H{e}_{e1}\\ E{x}_{e2}& E{n}_{e2}& H{e}_{e2}\\ ⋮& ⋮& ⋮\\ E{x}_{en}& E{n}_{en}& H{e}_{en}\end{array}\right]\\ & =C_e(E{x}_e,E{n}_e,H{e}_e)\end{array}$(11) where Ex_e indicates the score allotted by the expert panel, and En_e= 1/3E_e, E_e and He_e indicate the maximum deviation from the expectation and the randomness when the expert panel scores.

Step 4: Comprehensive evaluation cloud model establishment of the total objective. The comprehensive evaluation cloud model of single-dimension evaluation indexes on the total objective can be obtained by adopting the fitting operation rules on their weight and evaluation clouds as shown in Table 2. (12) $\begin{array}{rl}C(E_x,E_n,H_e)& =C_w(E{x}_w,E{n}_w,H{e}_w{)}^T{C}_e(E{x}_e,E{n}_e,H{e}_e)\\ & ={\left[\begin{array}{ccc}E{x}_{w1}& E{n}_{w1}& H{e}_{w1}\\ E{x}_{w2}& E{n}_{w2}& H{e}_{w2}\\ ⋮& ⋮& ⋮\\ E{x}_{wn}& E{n}_{wn}& H{e}_{wn}\end{array}\right]}^T\left[\begin{array}{ccc}E{x}_{e1}& E{n}_{e1}& H{e}_{e1}\\ E{x}_{e2}& E{n}_{e2}& H{e}_{e2}\\ ⋮& ⋮& ⋮\\ E{x}_{en}& E{n}_{en}& H{e}_{en}\end{array}\right]\\ & ={\left[\begin{array}{c}\sum _{i=1}^{n}E{x}_{wi}\times E{x}_{ei}\\ \sqrt{\sum _{i=1}^n{(E{x}_{wi}E{n}_{ei})}^2+{(E{x}_{ei}E{n}_{wi})}^2}\\ \sqrt{\sum _{i=1}^n{(E{x}_{wi}H{e}_{ei})}^2+{(E{x}_{ei}H{e}_{wi})}^2}\end{array}\right]}^T\end{array}$(12) Similarly, the comprehensive evaluation cloud model of multiple-dimension evaluation indexes on the total objective can be obtained by adopting the fitting operations rules on the weight and evaluation clouds of multiple sub-objectives. Therefore, a multidimensional hierarchical comprehensive evaluation process based on a cloud model can be realised.

5. Case study

In this study, we mainly consider two specific types of UAV swarms as an example to introduce the process of evaluating the intelligence levels of unmanned swarms by adopting the group extension cloud model. First, the evaluation index weights of all the layers were obtained by adopting the group extension AHP. Second, an intelligence evaluation level classification of unmanned swarms based on the collective OODA loop is presented. Thereafter, the cloud model is used to obtain the comprehensive evaluation results of the intelligence levels of unmanned swarms by fitting it layer-by-layer.

In this section, as shown in Figure 3, u indicates the overall intelligence of an unmanned swarm, and u₁–u₄ indicate collective observation, orientation, decision, and action competencies, respectively. Moreover, u₁₁–u₁₃ indicate the environment, target, and self-state perception competencies, respectively. Furthermore, u₂₁ and u₂₂ indicate situation assessment and fault diagnosis and prediction competencies, respectively; u₃₁ and u₃₂ represent the task allocation and path planning competencies, respectively; u₄₁ and u₄₂ indicate the task action competence and human independence, respectively. The interpretation of all the other variables is consistent with the descriptions in Section 4.

Figure 3 Evaluation cloud chart of six reference intelligence levels

5.1. Weight calculation based on group extension AHP

Considering the constructed intelligence evaluation index system of unmanned swarms, three experts who have been engaged in the research and development of unmanned swarms for a long time are invited to construct the extension judgment matrices at all layers by adopting the one-nine scale method proposed by Satty (Liu, 2012). A value of 1 indicates that an element is as important as another element, while the values of 3, 5, 7, and 9 indicate that one element is slightly, obviously, strongly, and extremely important than another element, respectively. The relative importance of the primary indexes on the overall intelligence of an unmanned swarm is summarised in Table 3.

Table 3 Relative importance of the primary indexes on the overall intelligence of an unmanned swarm

Expert	Index	u1	u2	u3	u4
Expert 1	u1	[1, 1]	[0.5, 0.67]	[0.67, 1]	[1.5, 2]
	u2	[1.5, 2]	[1, 1]	[1, 1.5]	[2, 2.5]
	u3	[1, 1.5]	[0.67, 1]	[1, 1]	[1.5, 2]
	u4	[0.5, 0.67]	[0.4, 0.5]	[0.5, 0.67]	[1, 1]
Expert 2	u1	[1, 1]	[0.33, 0.5]	[0.4, 0.67]	[1.5, 2.5]
	u2	[2, 3]	[1, 1]	[1.5, 2]	[2, 3]
	u3	[1.5, 2.5]	[0.5, 0.67]	[1, 1]	[1.5, 2.5]
	u4	[0.4, 0.67]	[0.33, 0.5]	[0.4, 0.67]	[1, 1]
Expert 3	u1	[1, 1]	[0.67, 1]	[0.67, 1]	[1.5, 2]
	u2	[1, 1.5]	[1, 1]	[1, 1.5]	[2, 2.5]
	u3	[1, 1.5]	[0.67, 1]	[1, 1]	[1.5, 2]
	u4	[0.5, 0.67]	[0.4, 0.5]	[0.5, 0.67]	[1, 1]

The lower and upper endpoint matrices $A^{-}$ and $A^{+}$ of the comprehensive extensive judgment matrix $A$, respectively, can be obtained using Eq.(1) as shown below: $\begin{array}{rl}{A}^{-}& =\left[\begin{array}{cc}\begin{array}{cc}1.000& 0.500\\ 1.500& 1.000\end{array}& \begin{array}{cc}0.580& 1.500\\ 1.167& 2.000\end{array}\\ \begin{array}{cc}1.167& 0.613\\ 0.467& 0.377\end{array}& \begin{array}{cc}1.000& 1.500\\ 0.467& 1.000\end{array}\end{array}\right],\end{array}$ $\begin{array}{rl}{A}^{+}& =\left[\begin{array}{cc}\begin{array}{cc}1.000& 0.723\\ 2.167& 1.000\end{array}& \begin{array}{cc}0.890& 2.167\\ 1.667& 2.667\end{array}\\ \begin{array}{cc}1.833& 0.890\\ 0.670& 0.500\end{array}& \begin{array}{cc}1.000& 2.167\\ 0.670& 1.000\end{array}\end{array}\right],\end{array}$

The normalised eigenvectors $x^{+}$and $x^{-}$ with positive components corresponding to their maximum eigenvalues can be easily obtained as demonstrated below: $\begin{array}{rl}{x}^{-}& =(0.4152,0.6958,0.5182,0.2737{)}^T,\end{array}$ $\begin{array}{rl}{x}^{+}& =(0.4210,0.6837,0.5330,0.2669{)}^T,\end{array}$

The values of the two intermediate variables can be obtained using Eq.(2), $l=0.9269$ and $m=1.0291$. They meet the conditions $0\le l\le 1\le m$, which indicates that the constructed A is consistent.

The weights expressed by the interval numbers can be obtained using Eq.(3) asindicated: $\begin{array}{rl}{w}_1& =⟨0.3848,0.4333⟩,\end{array}$ $\begin{array}{rl}{w}_2& =⟨0.6449,0.7036⟩,\end{array}$ $\begin{array}{rl}{w}_3& =⟨0.4803,0.5485⟩,\end{array}$ $\begin{array}{rl}{w}_4& =⟨0.2537,0.2747⟩,\end{array}$

The relative importance between elements $w_i$ and $w_j(i\ne j)$ can be obtained using Eq.(4) as provided below: $\begin{array}{rl}V(w_1\ge w_4)& =5.1683,\end{array}$ $\begin{array}{rl}V(w_2\ge w_4)& =11.2898,\end{array}$ $\begin{array}{rl}V(w_3\ge w_4)& =\mathrm{6.6099.}\end{array}$

Through normalisation, the single ordering weights of the four primary indexes on the overall intelligence of an unmanned swarm can be obtained as shown below: $p=(0.2240,0.4894,0.2865,0.0434{)}^T.$Similarly, the single ordering weights of the nine secondary indexes on the four primary indexes can be obtained as indicated below:   $\begin{array}{rl}{p}_1& =(0.4414,0.4321,0.1265{)}^T,\end{array}$ $\begin{array}{rl}{p}_2& =(0.6425,0.3575{)}^T,\end{array}$ $\begin{array}{rl}{p}_3& =(0.5782,0.4218{)}^T,\end{array}$ $\begin{array}{rl}{p}_4& =(0.7822,0.2178{)}^T.\end{array}$

5.2. Intelligence level classification based on the collective OODA loop

Currently, there is no authoritative standard for classifying the intelligence levels of unmanned swarms. By referring to and combining the classification standard of the intelligence levels of unmanned swarms proposed by Zhang et al. (2017) and the classical ACL distributed by DoD in 2005 (Office of the Secretary of Defense, 2005), we divided the intelligence of unmanned swarms into six levels: formation control, task planning, cooperative tasking, task re-planning, cooperative exploration, and fully rational from low to high. This is generally consistent with the intelligent development trend in the two classifications mentioned above; however, they are more reasonable in naming or more suitable for unmanned swarms. The intelligence of unmanned swarms is in the rapid development stage. Moreover, we investigate the existence of one or some specific collective competencies that can reflect the intelligence of unmanned swarms rather than their quantity. Therefore, we have further refined the specific evaluation requirements of different intelligence levels for unmanned swarms on the collective OODA loop as shown in Table 4. The structure of the classification is similar to the UAV ACL classification based on the OODA loop proposed by AFRL (Clough, 2002). Furthermore, it provides comprehensive evaluation levels of the nine secondary indexes on the four primary indexes and the comprehensive evaluation levels of four primary indexes on the overall intelligence of an unmanned swarm. However, the classification is only a type of relative reference because there are some uncertainties in the evaluation results of different evaluation indexes. The impact of different evaluation indexes on the overall evaluation results is also different. To obtain a reasonable intelligence level evaluation result, special methods are required.

Table 4 Classification for the intelligence level of unmanned swarms based on the collective OODA loop

Level	u	u1	u2	u3	u4
v6	fully rational	cognizant of all elements	coordinates as necessary	capable of total independence	requires little guidance
v5	cooperative exploration	tracks targets	infers strategic intention	accomplishes strategic planning	accomplishes strategic goal with minimal supervisory assistance
v4	task re-planning	tracks targets in limited ranged	infers tactical intention	accomplishes tactical planning	accomplishes tactical goal with minimal supervisory assistance
v3	cooperative tasking	identifies target status	evaluates and predicts target behavior	capable of cooperative trajectory planning	accomplishes tactical goal with minimal supervisory assistance
v2	task planning	perceives self status, identifies targets	diagnoses and predicts faults	re-planning on-board trajectory	accomplishes tactical plan as externally assigned
v1	formation control	co-localization, detects obstacles	cooperative guidance	chooses the right form in real time	keeps and reconstructs formation, avoids obstacles

5.3. Comprehensive evaluation of intelligence level based on the cloud model

In this study, we divide the qualitative evaluation conclusions into six intelligence levels as shown in Table 4. The corresponding expectation can be respectively set to 0.1 -0.6 in the cloud models. The greater the expectation is, the better the evaluation result. The mathematical characteristics of the normal cloud transforming the qualitative evaluation are expressed as C_e(Ex_e, En_e, He_e), where expectation Ex_e is the average score allotted by the expert panel, entropy En_e= 1/3E_e, E_e is the maximum deviation of the score provided by each expert relative to the expectation, and super entropy He_e is the randomness covered in the scores provided by the experts. Figure 3 shows a cloud chart of six reference intelligence levels.

The weights of the primary and secondary indexes are converted into the corresponding weight clouds, and their mathematical characteristics are shown in Table 5 and 6, respectively. Considering the proposed intelligence level classification method and the corresponding scoring standard mentioned above, a special expert panel is invited to evaluate the levels of the secondary indexes for the two specific types of UAV swarms by referring to the available description of system functions and observing its real behaviors. Thereafter, the evaluation results of the secondary indexes for the two UAV swarms are converted into the corresponding evaluation clouds, and their mathematical characteristics are shown in Table 5. Based on this, the mathematical characteristics of the evaluation clouds of the primary indexes can be obtained by adopting the fitting operation rules as shown in Table 2 on the weight and evaluation clouds of the secondary indexes. The calculation results are shown in Table 6.

Table 5 Weight and evaluation clouds of the secondary indexes

Index	Weight cloud	Evaluation Cloud A	Evaluation Cloud B
u11	(0.4414,0.04,0.005)	(0.2,0.04,0.005)	(0.3,0.04,0.005)
u12	(0.4321,0.04,0.005)	(0.1,0.04,0.005)	(0.3,0.04,0.005)
u13	(0.1265,0.04,0.005)	(0.2,0.04,0.005)	(0.4,0.04,0.005)
u21	(0.6425,0.04,0.005)	(0.3,0.04,0.005)	(0.3,0.04,0.005)
u22	(0.3575,0.04,0.005)	(0.2,0.04,0.005)	(0.4,0.04,0.005)
u31	(0.5782,0.04,0.005)	(0.3,0.04,0.005)	(0.3,0.04,0.005)
u32	(0.4218,0.04,0.005)	(0.2,0.04,0.005)	(0.2,0.04,0.005)
u41	(0.7822,0.04,0.005)	(0.2,0.04,0.005)	(0.3,0.04,0.005)
u42	(0.2178,0.04,0.005)	(0.3,0.04,0.005)	(0.4,0.04,0.005)

Table 6 Weight and evaluation clouds of the primary indexes

Index	Weight cloud	Evaluation Cloud A	Evaluation Cloud B
u1	(0.2240,0.04,0.005)	(0.157,0.0279,0.0035)	(0.313,0.0343,0.0043)
u2	(0.4894,0.04,0.005)	(0.264,0.0327,0.0041)	(0.336,0.0355,0.0044)
u3	(0.2865,0.04,0.005)	(0.258,0.0321,0.0040)	(0.258,0.0321,0.0040)
u4	(0.0434,0.04,0.005)	(0.222,0.0324,0.0041)	(0.322,0.0381,0.0048)

Similarly, the mathematical characteristics of the comprehensive evaluation cloud models of the intelligence levels for the two UAV swarms A and B can be obtained as C₁(0.274,0.0281,0.0035) and C₂(0.348,0.0336,0.0042), respectively, by adopting the fitting operation rules as shown in Table 2 on the weight and evaluation clouds of the primary indexes. Their comprehensive evaluation cloud chart is shown in Figure 4.

Figure 4 Comprehensive evaluation cloud chart of the intelligence levels for the two UAV swarms.

5.4. Evaluation results analysis

Considering Figure 4, the most concentrated cloud drops of UAV swarm A are located between Levels 2 and 3, whereas that of UAV swarm B are located between Levels 3 and 4. This indicates that the intelligence level of UAV swarm B is higher than that of UAV swarm A. The evaluation cloud of UAV swarm B is slightly larger than that of UAV swarm A, considering the span. This demonstrates that the evaluation conclusion of UAV swarm B has a higher fuzziness and randomness than that of UAV swarm A. The evaluation cloud of UAV swarm B is slightly larger than that of UAV swarm A, considering the dispersion degree and thickness. This indicates that the evaluation conclusion of UAV swarm B reached a better consensus than that of UAV swarm A. The evaluation exhibited some fuzziness and randomness. The intelligence of UAV swarm A meets the evaluation requirements of Level 2. Nonetheless, there is still a gap in reaching the evaluation requirements of Level 3. Moreover, the intelligence of UAV swarm B meets the evaluation requirements of Level 3; however, there is still a gap in reaching the evaluation requirements of Level 4. Regarding the perspective of dimension distribution, there is still a certain distance between the current intelligence evaluation results and the most satisfactory intelligence evaluation results. This shows that there is still a great need for further improvement.

Considering Table 6, the four primary indexes have different effects on the overall intelligence of an unmanned swarm. Furthermore, they are the collective orientation, decision, observation, and action competencies in order of importance. This conclusion is consistent with our general understanding of intelligence. Thinking can best reflect intelligence, perception is the second, and action is the last (Liu et al., 2007; Wu & Zhou, 2021). There are some inconsistencies in the evaluation results of the four primary evaluation indexes of each evaluated UAV swarm. These inconsistencies effectively meet the actual evaluation requirements of the intelligence levels for unmanned swarms. This is because the actual unmanned swarm cannot be developed completely based on the intelligence level classification method established by the evaluators. Furthermore, it is possible that the intelligence of a certain dimension is slightly higher or lower. The intelligence level of UAV swarm A is lower than that of UAV swarm B mainly because its evaluation results on the four primary indexes are less than or equal to those of UAV swarm B. Regarding Table 5, the nine secondary indexes have different effects on their superior indexes. There are also some inconsistencies in the evaluation results of the nine secondary indexes of each evaluated UAV swarm. These inconsistencies were also allowed in the evaluation process.

Subsequently, the researchers and developers should continue to work hard in the following aspects. First, they should pay attention to the overall intelligence realisation of unmanned swarms based on the collective intelligent behavior model based on the collective OODA loop. Second, they should continuously improve the intelligence level of unmanned swarms in a progressive manner based on the given intelligence level classification. Moreover, they should regularly evaluate the intelligence level of unmanned swarms in a suitable and feasible evaluation method to prove that its intelligence has been improved.

6. Conclusions and future studies

A comprehensive evaluation of the intelligence levels for unmanned swarms is a complex matter. We propose a suitable evaluation solution. This study combines the collective OODA loop and the group extensive cloud model to achieve this goal. The study has constructed a two-layer evaluation system, covering the collective observation, orientation, decision, and action competencies. The constructed evaluation index system can suitably and comprehensively reflect the overall intelligence of unmanned swarms owing to the introduction and reference of the collective intelligent behavior model of unmanned swarms based on the collective OODA loop. The proposed group extensive AHP method replaces exact numbers with interval numbers when constructing the comparison matrix of the evaluation indexes. It also effectively reduces the influence of individual subjective preferences by synthesising the judgments of multiple experts. In addition, there is no additional consistency check because the obtained weight vector naturally satisfies consistency. Therefore, it can more flexibly, objectively, and conveniently calculate the weights of the evaluation indexes than the traditional AHP used in previous studies. The comprehensive evaluation method based on the cloud model simplified the measurement and quantification of the evaluation indexes. This can fully describe the fuzziness and randomness of the evaluation process. We enabled the transformation of a qualitative evaluation into a quantitative evaluation by rendering results more objective and accurate than the fully qualitative evaluation methods used in previous studies. The limitations of the proposed method are that it is not a fully quantitative evaluation and still needs to rely on the judgment of experts in the evaluation process.

Subsequently, this study can be improved from the following two aspects. Considering the construction of an intelligence evaluation index system for unmanned swarms, it should be designed more detailedly with the continuous opening of relevant parameters. It should also incorporate quantitative evaluation indexes. Moreover, its structure is not necessarily designed strictly based on a hierarchical structure because of the complicated correlation between evaluation indexes. Regarding the evaluation methods, the weight-determining method based on group extension AHP can further consider the authority of different expert judgments rather than a simple geometric average processing. Furthermore, the comprehensive evaluation method based on the cloud model can be further optimised by introducing the extension and group decision-making methods to further improve its flexibility and objectivity.

Acknowledgement

This study was supported in part by the National Defense Science and Technology Innovation Program of China under grant number18-163-11-ZT-003-010-01 and the National Key Research and Development Program of China under grant number 2017YFB1001900. Moreover, we would like to appreciate domain experts for their participation in the case study and the three reviewers for their valued comments on this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by The National Defense Science and Techonology Innovation Program of China: [Grant Number No.18-163-11-ZT-003-010-01]; The National Key Research and Devevlopment Program of China: [Grant Number No.2017YFB1001900].