Theoretical and empirical validation of software trustworthiness measure based on the decomposition of attributes

Abstract

From the perspective of attribute decomposition, there are a variety of software trustworthiness metric models. However, little attention has been paid to using more rigorous methods and to performing theoretical validation. Axiomatic methods formalise the empirical understanding of software attributes through defining ideal metric properties. They can offer precise terms for the software attributes' quantification. We have utilised them to assess software trustworthiness on the basis of attribute decomposition, presented four properties, constructed a software trustworthiness measure (STMBDA for short). In this paper, we extend the set of properties, introduce two new properties, namely non-negativity and proportionality, and perfect substitutability and expectability. We verify the theoretical rationality of STMBDA by demonstrating that it conforms to the new property set and the empirical validity by evaluating the trustworthiness of 23 spacecraft software. The validation results show that STMBDA is able to effectively assess the spacecraft software trustworthiness and identify weaknesses in the development process.

Keywords:

• Theoretical validation
• empirical validation
• trustworthiness measurement
• axiomatic approaches

1. Introduction

Software is increasingly ubiquitous in our everyday lives and plays an indispensable role in the function of our society. A system failure caused by the software operation (directly or indirectly) can have very severe consequences, resulting in not only monetary, time, or property losses, but also casualties (Wong et al., 2010, 2017). As a result, software trustworthiness has attracted widespread attention in recent years. Its measurement has become a hot topic among researchers (He et al., 2018; Maza & Megouas, 2021). Software trustworthiness is able to be represented through many attributes (Chen & Tao, 2019; Gupta et al., 2021; Steffen et al., 2006); in this paper, these attributes are referred to as trustworthy attributes. Trustworthy attributes are often at levels that cannot be directly measured, so they are further decomposed into sub-attributes. A lot of software trustworthiness metric models on the basis of attribute decomposition are established. However, few studies have focused on more rigorous methods for measuring software trustworthiness and theoretically validating these measures. Strictly measuring the trustworthiness of software is able to assist with the assessment and improvement of software trustworthiness. Theoretical validation is a necessary activity for defining meaningful metric models and a required step for empirical validation of them (Srinivasan & Devi, 2014). Theoretical validation methods can be divided into two categories (Srinivasan & Devi, 2014): one is on the basis of the measurement theory (Briand, Emam, et al., 1996; Zuse & Bollmann-Sdorra, 1991), and the other is in the view of axiomatic approaches (Briand, Morasca, et al., 1996). Axiomatic approaches, which formally describe the empirical understanding of software attribute by defining desired metric properties, have been used to measure internal attributes, such as size, inheritance, complexity, and so on (Meneely et al., 2012).

To make software trustworthiness measurement more stringent, we once used axiomatic approaches to assess software trustworthiness on the basis of attributes. We extend this work to utilise axiomatic approaches to evaluate software trustworthiness from the point of attribute decomposition, give the expected properties of software trustworthiness measurement in the view of attribute decomposition, including monotonicity, acceleration, sensitivity, substitutivity and expectability, and establish a software trustworthiness measure (STMBDA) (Tao et al., 2015). In this paper, we complete the property set and introduce two new properties, namely non-negativity and proportionality. The reasons for introducing these two properties are as follows The measure result of software trustworthiness cannot be negative and non-negativity is used to describe this case. Software trustworthiness is the user's subjective recognition of the objective quality of software. The quantification of software trustworthiness needs to truly reflect users' approval. A software that is trusted requires that individual attribute (sub-attribute) values should not be too low, proportionality is to characterise this situation. At the same time, the substitutivity and expectability are improved. The improved substitutability adds two constraints to better depict the substitution. The first is that substitutivity between critical and non-critical attributes should be not only harder than that between critical attributes but also more difficult than that between non-critical attributes. The second is that the substitutivity between sub-attributes should be harder than that between attributes. The improved expectability adds user expectations related to sub-attributes. We verify STMBDA's theoretical rationality by proving that it satisfies the new property set. Its empirical validity is verified by utilising it to measure the trustworthiness of 23 spacecraft software. Compared with some established software trustworthiness metric models, STMBDA is able to better evaluate the software trustworthiness.

The rest of the work is organised as follows. We present the related works in Section 2. The expected measurement properties from the point of the attribute decomposition are described in Section 3, including the properties presented in Tao et al. (2015), the newly introduced properties and the improved properties. STMBDA is introduced in Section 4 and we carry out its theoretical validation in the same section. We give STMBDA-based measurement process in Section 5. We conduct an empirical validation of STMBDA by a real case in Section 6. The comparative study is presented in Section 7. We end the paper with the conclusion and future work in the last section.

2. Related works

Typical models comprise uncertain theory (Shi et al., 2008), evidence theory (Ding et al., 2012), machine learning (Devi et al., 2019; Lian & Tang, 2022; Medeiros et al., 2020; Xu et al., 2021), system testing (Muhammad et al., 2018), crowd wisdom (H. M. Wang, 2018), social-to-software framework (Yang et al., 2018), heuristic-systematic processing model (Gene & Tyler, 5384–5393), users feedback (B. H. Wang et al., 2019), STRAM (Security, Trust, Resilience, and Agility Metrics) (Cho et al., 2019), vulnerability loss speed index (Gul & Luo, 2019), input validation coding practices behaviour (Lemes et al., 2019), trajectory matrix (Tian & Guo, 2020), trustworthy evidence of source code (Liu et al., 2021), axiomatic approaches (Tao & Chen, 2009, 2012; J. Wang et al., 2015), etc.

Next, we select some of the typical models mentioned above for a detailed introduction. Shi et al. (2008) first build a software dependability indicator system on the basis of the existing research results and then give a software dependability metric method by combining AHP and the fuzzy synthesised evaluation method. Ding et al. (2012) develop a software trustworthiness evaluation model on the basis of evidential reasoning. In this method, they also introduce two discounting factor estimation approaches to measure the reliability degree of evaluation result. Medeiros et al. (2020) utilise machine learning algorithms to obtain the knowledge related to vulnerabilities from software metrics extracted from source code of some representative software projects. Devi et al. (2019) analyse the effects of class imbalance and class overlap in traditional learning models, and their research results can be used to classify data sets of software trustworthiness assessment with class imbalance. Xu et al. propose a QoS prediction model. In this model, neural networks and matrix factorisation are combined to perform non-linear collaborative filtering on the latent feature vectors of users and services (Xu et al., 2021). Lian and Tang (2022) give an API recommendation method on the basis of neural graph collaborative filtering technique. The experimental results show that the performance of this method is better than the most advanced methods in API recommendation. Falcone and Castelfranchi (2002) study the relationship between trust and control. Muhammad et al. (2018) give the software trustworthiness rating strategy, which utilises the system test execution completion score to measure software trustworthiness. Yang et al. present a Social-to-Software software trustworthiness framework. This framework consists of a generalised index loss, the ability trustworthiness measurement solution, basic standard trustworthiness measurement solution and identity trustworthiness measurement solution (Yang et al., 2018). Wang et al. establish an updating model of software component trustworthiness. The component's trustworthy degree is calculated according to users' feedback, the updating weight is obtained based on the number of users, and the final trustworthy degree of the system is computed by the Euler distance (B. H. Wang et al., 2019). Xie et al. (2022) present an approach to cover specifications of nominal behaviour and security, in which combined validation is utilised to verify the SysML models' nominal behaviour and Fault Tree Analysis for security analysis.

3. Properties of software trustworthiness measures based on attribute decomposition

Trustworthy attributes are divided into critical and non-critical attributes. Critical attributes are the trustworthy attributes that the software must have, and others are called non-critical attributes (Tao & Chen, 2009). Using the same notations as Tao and Chen (2009), denote the value of critical attributes by $y_1,\dots ,y_m$, and the values of non-critical attributes by $y_{m+1},\dots ,y_{m+s}$. Assuming that there are n sub-attributes to form trustworthy attributes, set their values as $x_1,\dots ,x_n$, respectively. Let $y_i(1\le i\le m+s)$ be attribute measure functions about $x_1,\dots ,x_n$, and T be the software trustworthiness measure function of $y_i(1\le i\le m+s)$. The expected measure properties in the view of attribute decomposition are as follows. Among them, monotonicity, acceleration, and sensitivity are properties that have been proposed in Tao et al. (2015), non-negativity and proportionality are newly introduced properties, and substitutivity and expectability are improved properties.

(1) Non-negativity

1. $0\le T$.

Non-negativity implies that the evaluation result of software trustworthiness is non-negative.

(2) Proportionality

1. $(\mathrm{\exists }{c}_1,c_2\in {\mathrm{\Re }}^{+})c_1\le {\displaystyle \frac{y_i}{y_j}}\le c_2,1\le i,j\le m+s$,

2. $(\mathrm{\exists }{c}_3,c_4\in {\mathrm{\Re }}^{+})c_3\le {\displaystyle \frac{x_i}{x_j}}\le c_4,1\le i,j\le n$.

Proportionality refers to the assumption that there should be an appropriate proportionality between attributes (sub-attributes). For example, supposing that critical attributes of a certain type of software consist of resilience and survivability, high-confidence software of this type requires both good resilience and high survivability. Very good resilience and low survivability or very high survivability and poor resilience are not appropriate. There are similar reasons for the proportional suitability of sub-attributes.

(3) Monotonicity (Tao et al., 2015)

1. $\mathrm{\partial }T/\mathrm{\partial }{y}_i\ge 0,1\le i\le m+s$,

2. $\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_k\ge 0,1\le i\le m+s,1\le k\le n$.

That is, the increased value of the attribute does not cause a decrease in the software's trustworthy degree, and the increase in the sub-attribute value does not result in a lower attribute value.

(4) Acceleration (Tao et al., 2015)

1. ${\mathrm{\partial }}^{2}T/\mathrm{\partial }{y_i}^2\le 0,1\le i\le m+s$, 

2. ${\mathrm{\partial }}^2{y}_i/\mathrm{\partial }{x_k}^2\le 0,1\le i\le m+s,1\le k\le n$.

Acceleration is used to characterise the rate of change of a attribute (sub-attribute). When only one attribute $y_i$ is increased and the other attributes $y_j(j\ne i)$ remain unchanged, the efficiency of using attribute $y_i$ will decrease. For the sub-attributes, there are similar explanations.

(5) Sensitivity (Tao et al., 2015)

1. $0\le {\displaystyle \frac{\frac{\mathrm{\partial }T}{T}}{\frac{\mathrm{\partial }{y}_i}{y_i}}}=\frac{\mathrm{\partial }T}{\mathrm{\partial }{y}_i}\frac{y_i}{T}=f_1(y_i,w_i),1\le i\le m+s$,

2. $0\le {\displaystyle \frac{\frac{\mathrm{\partial }{y}_i}{y_i}}{\frac{\mathrm{\partial }{x}_k}{x_k}}}=\frac{\mathrm{\partial }{y}_i}{\mathrm{\partial }{x}_k}\frac{x_k}{y_i}=f_2(x_k,w_{i,k}),1\le i\le m+s,1\le k\le n$,

where $w_i$ is the ith attribute's weight, $w_{i,k}$ is the weight of kth sub-attribute that constitutes the ith attribute, $f_1$ is a function of $y_i$ and $w_i$, $f_2$ is a function of $x_k$ and $w_{i,k}$. Sensitivity represents the percentage change in software trustworthiness (attribute value) caused by the percentage change in the attribute value (sub-attribute value). They should be non-negative and associated with the attributes (sub-attributes) and their weights. Furthermore, the software trustworthiness should be more sensitive to the smallest critical attribute relative to its weight. The reason is that its improvement is able to greatly increase the software trustworthiness ; therefore, the percentage change in its value will result in a relatively larger percentage change in the software trustworthiness.

(6) Substitutivity

1. $(\mathrm{\exists }{c}_5,c_6\in {\mathrm{\Re }}^{+})c_5\le {\sigma }_{rt}\le c_6$,

where (1) ${\sigma }_{rt}=\frac{\mathrm{d}(y_r/y_t)}{\mathrm{d}(-\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_t}{\mathrm{\partial }T/\mathrm{\partial }{y}_r})}\times \frac{-\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_t}{\mathrm{\partial }T/\mathrm{\partial }{y}_r}}{y_r/y_t},1\le r,t\le m+s,r\ne t.$(1) ${\sigma }_{rt}$ are applied to indicate the difficulty of substituting between attributes, complying with $0\le {\sigma }_{rt}\le \mathrm{\infty }$. The smaller ${\sigma }_{rt}$, the harder it is to replace the rth and tth attributes.

1. $(\mathrm{\exists }{c}_7,c_8\in {\mathrm{\Re }}^{+})c_7\le {\sigma }_{ikl}\le c_8$,

where (2) ${\sigma }_{ikl}=\frac{\mathrm{d}(x_k/x_l)}{\mathrm{d}(-\frac{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_l}{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_k})}\times \frac{-\frac{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_l}{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_k}}{x_k/x_l},1\le i\le m+s,1\le k,l\le n,k\ne l.$(2) ${\sigma }_{ikl}$ are applied to represent the difficulty of replacement between the sub-attributes that constitute the ith attribute. Similarly, ${\sigma }_{ikl}$ meets $0\le {\sigma }_{ikl}\le \mathrm{\infty }$, the smaller ${\sigma }_{ikl}$, the harder it is to replace the kth and the lth sub-attribute.

Property 1 and property 2 indicate that the attributes are able to be replaced for each other to a certain extent, so can the sub-attributes.

1. ${\sigma }_{ij}\le {\sigma }_{rt}$, $1\le i\le m,m+1\le j\le m+s,1\le r,t\le m,i\ne j,r\ne t$,

   ${\sigma }_{ij}\le {\sigma }_{rt}$, $1\le i\le m,m+1\le j\le m+s,m+1\le r,t\le m+s,i\ne j,r\ne t$.

2. ${\sigma }_{ikl}\le {\sigma }_{rt}$, $1\le i,r,t\le m+s,1\le k,l\le n,r\ne t,k\ne l$.

Property 3 states that the substitutivity between critical and non-critical attributes should be not only harder than that between critical attributes but also more difficult than that between non-critical attributes. Property 4 means that the substitutivity between sub-attributes should be harder than that between attributes.

(7) Expectability

1. $(x_0\le min\{x_1,\dots ,x_n\})\Rightarrow (x_0\le y_i\le max\{x_1,\dots ,x_n\})$,

2. $(y_0\le min\{y_1,\dots ,y_{m+s}\})\Rightarrow (y_0\le T\le max\{y_1,\dots ,y_{m+s}\})$,

where $x_0$ and $y_0$ are the user's minimum expected values for sub-attributes and attributes, respectively. Expectability implies that if all sub-attributes (attributes) meet the user's expectations, then the attribute trustworthiness (software trustworthiness) should also achieve the user's expectations and be less than or equal to the maximum value of all sub-attributes (attributes).

4. A software trustworthiness measure based on the decomposition of attributes and its theoretical validation

In this section, we introduce the software trustworthiness measure on the basis of the attribute decomposition constructed in Tao et al. (2015) and theoretically validate it by demonstrating that it conforms to the properties described in Section 3.

Definition 4.1

Software trustworthiness measure based on the decomposition of attributes (STMBDA for short) (Tao et al., 2015)

STMBDA is defined as follows. $\{\begin{array}{l}T={\{\alpha {[{\left({\displaystyle \frac{\underset{1\le i\le m}{min}\{y_i\}}{10}}\right)}^{ϵ}{y}_1^{{\alpha }_1}{y}_2^{{\alpha }_2}\cdots y_m^{{\alpha }_m}]}^{-\rho }+\beta [y_{m+1}^{{\beta }_{m+1}}{y}_{m+2}^{{\beta }_{m+2}}\cdots y_{m+s}^{{\beta }_{m+s}}{]}^{-\rho }\}}^{-{\displaystyle \frac{1}{\rho }}}{y}_i={\left(\sum _{k=1}^n{w}_{i,k}{x}_k^{-{\rho }_i}\right)}^{-{\displaystyle \frac{1}{{\rho }_i}}},1\le i\le m+s.\end{array}$Where

1. m, s and n are the number of critical attributes, non-critical attributes and sub-attributes, respectively;

2. $y_i(1\le i\le m+s)$ are the values of attributes;

3. T is the software trustworthiness measure function of $y_1,\dots ,y_{m+s}$;

4. α and β represent the proportion of critical and non-critical attributes, such that $\alpha +\beta =1$ and $0\le \beta <0.5<\alpha \le 1$.

5. ${\alpha }_i(1\le i\le m)$ are the weight values of critical attributes, satisfying $\sum _{i=1}^m{\alpha }_i=1$ and $0\le {\alpha }_i\le 1$;

6. ${\beta }_j(m+1\le j\le m+s)$ express the weight values of the non-critical attributes with $\sum _{j=m+1}^{m+s}{\beta }_j=1$ and $0\le {\beta }_j\le 1$;

7. $w_{i,k}(1\le i\le m+s,1\le k\le n)$ are the weight values of sub-attributes making up the ith attribute with $\sum _{k=1}^n{w}_{i,k}=1$ and $0\le w_{i,k}\le 1$, if the kth sub-attribute is not one of the sub-attributes forming the ith attribute, set $w_{i,k}=0$;

8. ϵ is utilised to control the effect of the smallest critical attribute on the software trustworthiness with $0\le ϵ\le min\{1-{\alpha }_{{min}^{\prime }},\frac{\ln y_0-\ln y_{{min}^{\prime }}}{\ln y_{{min}^{\prime }}-\ln 10}\}$, among them, $y_0$ is a value that all attributes must achieve and ${min}^{\prime }$ is the i with $\underset{1\le i\le m}{min}\{y_i\}$;

9. ρ is a parameter associated with the substitutivity between critical and non-critical attributes with $0<\rho$;

10. ${\rho }_i(1\le i\le m+s)$ are parameters related to the substitutivity between the sub-attributes that make up the ith attribute satisfying $0<\rho \le {\rho }_i$;

11. $x_k(1\le k\le n)$ are the values of sub-attributes with $1\le max\{x_0,y_0\}\le x_k\le 10$, among them, $x_0$ is a value that all sub-attributes must reach.

For convenience, we denote the i with $\underset{1\le i\le m}{min}\{y_i\}$ by ${min}^{\prime }$, the i with $\underset{1\le i\le m}{max}\{y_i\}$ by ${max}^{\prime }$, the i with $\underset{m+1\le i\le m+s}{min}\{y_i\}$ by ${min}^{″}$, the i with $\underset{m+1\le i\le m+s}{max}\{y_i\}$ by ${max}^{″}$, and let $\begin{array}{rl}{y}_{{min}^{\prime }}& =\underset{1\le i\le m}{min}\{y_i\},y_{{max}^{\prime }}=\underset{1\le i\le m}{max}\{y_i\},y_{{min}^{″}}=\underset{m+1\le i\le m+s}{min}\{y_i\},y_{{max}^{″}}=\underset{m+1\le i\le m+s}{max}\{y_i\},\\ a_1& ={[{\left(\frac{y_{{min}^{\prime }}}{10}\right)}^{ϵ}{y}_1^{{\alpha }_1}{y}_2^{{\alpha }_2}\cdots y_m^{{\alpha }_m}]}^{-\rho },b_1=[y_{m+1}^{{\beta }_{m+1}}{y}_{m+2}^{{\beta }_{m+2}}\cdots y_{m+s}^{{\beta }_{m+s}}{]}^{-\rho }.\end{array}$The proof of Proposition 4.2(2) has been given in the Claim 1 in Tao et al. (2015). Here we only give the proof of Proposition 4.2(1).

Proposition 4.2

1. $1\le x_0\le y_i\le \underset{1\le k\le n}{max}\{x_k\}\le 10,1\le i\le m+s$.

2. T conforms to non-negativity and $1\le T\le 10$.

Proof.

(1) Because $\{\begin{array}{ll}0<{\rho }_i,& 1\le i\le m+s,\\ 0\le w_{i,k}\le 1,& 1\le i\le m+s,1\le k\le n,\\ 1\le x_k\le 10,& 1\le k\le n,\end{array}$then $\sum _{k=1}^n{w}_{i,k}{\left\{\underset{1\le k\le n}{max}\{x_k\}\right\}}^{-{\rho }_i}\le \sum _{k=1}^n{w}_{i,k}{x}_k^{-{\rho }_i}\le \sum _{k=1}^n{w}_{i,k}{\left\{\underset{1\le k\le n}{min}\{x_k\}\right\}}^{-{\rho }_i}.$Substituting $\sum _{k=1}^n{w}_{i,k}=1$ in the above inequality, we can get ${\left\{\underset{1\le k\le n}{max}\{x_k\}\right\}}^{-{\rho }_i}\le \sum _{k=1}^n{w}_{i,k}{x}_k^{-{\rho }_i}\le {\left\{\underset{1\le k\le n}{min}\{x_k\}\right\}}^{-{\rho }_i}.$Raising to $-{\frac{1}{\rho }}_i$ power in the above inequality, it follows that $\underset{1\le k\le n}{min}\{x_k\}\le y_i={\left\{\sum _{k=1}^n{w}_{i,k}{x}_k^{-{\rho }_i}\right\}}^{-{\frac{1}{\rho }}_i}\le \underset{1\le k\le n}{max}\{x_k\}.$From the definition of STMBDA, we know that $1\le max\{x_0,y_0\}\le x_i\le 10,1\le k\le n,$then $\{\begin{array}{l}{\displaystyle 1\le x_0\le \underset{1\le k\le n}{min}\{x_k\}\le y_i\le \underset{1\le k\le n}{max}\{x_k\}\le 10,1\le i\le m+s,}\\ {\displaystyle 1\le y_0\le \underset{1\le k\le n}{min}\{x_k\}\le y_i\le \underset{1\le k\le n}{max}\{x_k\}\le 10,1\le i\le m+s.}\end{array}$

Proposition 4.3

Proportionality holds for T.

Proof.

Since $1\le x_i\le 10,1\le i\le n,$then for $1\le i$, $j\le n$ $\frac{x_i}{10}\le \frac{x_i}{x_j}\le \frac{x_i}{1}\Rightarrow \frac{1}{10}\le \frac{x_i}{x_j}\le \frac{10}{1}.$According to Proposition 4.2(1), we can obtain that $1\le y_i\le 10,1\le i\le m+s,$it follows that for $1\le i,j\le m+s$, $\frac{y_i}{10}\le \frac{y_i}{y_j}\le \frac{y_i}{1}\Rightarrow \frac{1}{10}\le \frac{y_i}{y_j}\le \frac{10}{1}.$The proposition follows immediately from what we have proved.

Proposition 4.4

Monotonicity is satisfied by T (Tao et al., 2015).

Proposition 4.5

T complies with acceleration (Tao et al., 2015).

Proposition 4.6

Sensitivity holds for T (Tao et al., 2015).

The proof process of Proposition 4.4, 4.5, and 4.6 can be found in Tao et al. (2015)

Proposition 4.7

T meets substitutivity.

Proof.

By Equation (2(2) ${\sigma }_{ikl}=\frac{\mathrm{d}(x_k/x_l)}{\mathrm{d}(-\frac{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_l}{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_k})}\times \frac{-\frac{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_l}{\mathrm{\partial }{y}_i/\mathrm{\partial }{x}_k}}{x_k/x_l},1\le i\le m+s,1\le k,l\le n,k\ne l.$(2) ), the substitutivity between the sub-attributes making up the ith $(1\le i\le m+s)$ attribute is able to be determined as follows: (3) ${\sigma }_{ikl}=\frac{1}{{\rho }_i+1},1\le i\le m+s,1\le k,l\le n,k\ne l.$(3) Similarly, through Equation (1(1) ${\sigma }_{rt}=\frac{\mathrm{d}(y_r/y_t)}{\mathrm{d}(-\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_t}{\mathrm{\partial }T/\mathrm{\partial }{y}_r})}\times \frac{-\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_t}{\mathrm{\partial }T/\mathrm{\partial }{y}_r}}{y_r/y_t},1\le r,t\le m+s,r\ne t.$(1) ), for the substitutivity between critical attributes, it follows that (4) ${\sigma }_{ij}=1,1\le i,j\le m,i\ne j,$(4) for the substitutivity between non-critical attributes, we can deserve (5) ${\sigma }_{ij}=1,m+1\le i,j\le m+s,i\ne j,$(5) and the substitutivity between critical and non-critical attributes can be obtained as ${\sigma }_{ij}=\{\begin{array}{lll}{\displaystyle \frac{1+\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}}{(1+\rho {\beta }_i)+(1+\rho {\alpha }_j)\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}}},& j\ne mi{n}^{\prime },1\le j\le m,m+1\le i\le m+s,{\displaystyle \frac{1+\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}}{(1+\rho {\beta }_i)+(1+\rho ϵ+\rho {\alpha }_j)\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}}},& j={min}^{\prime },m+1\le i\le m+s.\end{array}$Since for $j\ne {min}^{\prime },1\le j\le m,m+1\le i\le m+s$, $0\le {\alpha }_j,{\beta }_i\le 1,$and $0<\rho ,$then we have $\begin{array}{rl}(1+\rho min\{{\alpha }_j,{\beta }_i\})(1+\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j})& \le (1+\rho {\beta }_i)+(1+\rho {\alpha }_j)\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}\\ & \le (1+\rho max\{{\alpha }_j,{\beta }_i\})(1+\frac{y_i}{y_j}\frac{\mathrm{\partial }T/\mathrm{\partial }{y}_i}{\mathrm{\partial }T/\mathrm{\partial }{y}_j}),\end{array}$therefore, $\frac{1}{1+\rho max\{{\alpha }_j,{\beta }_i\}}\le {\sigma }_{ij}\le \frac{1}{1+\rho min\{{\alpha }_j,{\beta }_i\}}.$Observe that for $j\ne mi{n}^{\prime },1\le j\le m,m+1\le i\le m+s$, $\{\begin{array}{l}{\displaystyle \frac{1}{1+\rho }}\le {\displaystyle \frac{1}{1+\rho max\{{\alpha }_j,{\beta }_i\}}},\\ {\displaystyle \frac{1}{1+\rho min\{{\alpha }_j,{\beta }_i\}}}\le 1,\end{array}$and $0<\rho \le {\rho }_i,1\le i\le m+s,$it follows (6) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j\ne mi{n}^{\prime },1\le j\le m,m+1\le i\le m+s.$(6) Similarly, for $j={min}^{\prime }$, $m+1\le i\le m+s$, $\frac{1}{1+\rho max\{ϵ+{\alpha }_j,{\beta }_i\}}\le {\sigma }_{ij}\le \frac{1}{1+\rho min\{ϵ+{\alpha }_j,{\beta }_i\}}.$Note that for $j=mi{n}^{\prime },m+1\le i\le m+s$, $\{\begin{array}{l}\frac{1}{1+\rho }\le \frac{1}{1+\rho max\{ϵ+{\alpha }_j,{\beta }_i\}},\\ \frac{1}{1+\rho min\{ϵ+{\alpha }_j,{\beta }_i\}}\le 1,\end{array}$and $0<\rho \le {\rho }_i,1\le i\le m+s,$it follows (7) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j={min}^{\prime },=m+1\le i\le m+s.$(7) It can be seen from Equation (3(3) ${\sigma }_{ikl}=\frac{1}{{\rho }_i+1},1\le i\le m+s,1\le k,l\le n,k\ne l.$(3) ) that the sub-attributes can be replaced with each other to a certain extent. According to Equations (4(4) ${\sigma }_{ij}=1,1\le i,j\le m,i\ne j,$(4) ), (5(5) ${\sigma }_{ij}=1,m+1\le i,j\le m+s,i\ne j,$(5) ), (6(6) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j\ne mi{n}^{\prime },1\le j\le m,m+1\le i\le m+s.$(6) ) and (7(7) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j={min}^{\prime },=m+1\le i\le m+s.$(7) ), attributes can be replaced by each other to a certain extent. By Equations (6(6) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j\ne mi{n}^{\prime },1\le j\le m,m+1\le i\le m+s.$(6) ) and (7(7) $\frac{1}{1+{\rho }_i}\le {\sigma }_{ij}\le 1,j={min}^{\prime },=m+1\le i\le m+s.$(7) ), the substitutivity between critical and non-critical attributes is harder than both the substitutivity between critical attributes and that between non-critical attributes, and the substitutivity between sub-attributes is harder than that between attributes.

In summary, the proposition is proved.

Proposition 4.8

T satisfies expectability.

Proof.

By the definition of STMBDA, we can obtain that $x_0\le min\{x_1,\dots ,x_n\}.$According to Proposition 4.2(1), it follows that $x_0\le y_i\le max\{x_1,\dots ,x_n\}.$Since $\{\begin{array}{l}0\le {\alpha }_i\le 1,1\le i\le m,\\ {\displaystyle \sum _{i=1}^m{\alpha }_i=1.}\end{array}$and from the proof process of Proposition 4.2(1) we know that $1\le y_i\le 10,1\le i\le m.$Then $\frac{y_{{min}^{\prime }}^{1+ϵ}}{{10}^{ϵ}}\le \underset{1\le i\le m}{min}\left\{{\left(\frac{y_i}{10}\right)}^{ϵ}\right\}{y}_1^{{\alpha }_1}{y}_2^{{\alpha }_2}\cdots y_m^{{\alpha }_m}\le y_{{max}^{\prime }}.$Likewise, we can prove $y_{{min}^{″}}\le y_{m+1}^{{\beta }_{m+1}}{y}_{m+2}^{{\beta }_{m+2}}\cdots y_{m+s}^{{\beta }_{m+s}}\le y_{{max}^{″}}.$Because of $0<\rho$ and $0\le \alpha ,\beta \le 1,$ it follows that $\alpha y_{{max}^{\prime }}^{-\rho }+\beta y_{{max}^{″}}^{-\rho }\le \alpha a_1+\beta b_1\le \alpha {\left(\frac{y_{{min}^{\prime }}^{1+ϵ}}{{10}^{ϵ}}\right)}^{-\rho }+\beta y_{{min}^{″}}^{-\rho }.$Due to $\alpha +\beta =1$, we obtain that $min\{y_{{max}^{\prime }}^{-\rho },y_{{max}^{″}}^{-\rho }\}\le \alpha a_1+\beta b_1\le max\{{\left(\frac{y_{{min}^{\prime }}^{1+ϵ}}{{10}^{ϵ}}\right)}^{-\rho },y_{{min}^{″}}^{-\rho }\}.$So $min\{\frac{y_{{min}^{\prime }}^{1+ϵ}}{{10}^{ϵ}},y_{{min}^{″}}\}\le T\le max\{y_{{max}^{\prime }},y_{{max}^{″}}\}=max\{y_1,\dots ,y_{m+s}\}.$Recall that we have proved that $1\le y_0\le y_i\le 10,1\le i\le m+s,$then $y_0\le min\{y_1,\dots ,y_{m+s}\}.$Since $0\le ϵ\le min\{1-{\alpha }_{{min}^{\prime }},\frac{\ln y_0-\ln y_{{min}^{\prime }}}{\ln y_{{min}^{\prime }}-\ln 10}\},$it follows that $y_0\le min\{\frac{y_{{min}^{\prime }}^{1+ϵ}}{{10}^{ϵ}},y_{{min}^{″}}\},$and we can deserve that $y_0\le T\le max\{y_1,\dots ,y_{m+s}\}.$

From the conclusions of Proposition 4.2, 4.3, 4.4, 4.5, 4.6, 4.7 and Proposition4.8, the following theorem can be obtained.

Theorem 4.9

STMBDA conforms to all the seven properties introduced in Section 3.

5. Measurement process based on STMBDA

The STMBDA-based measurement process is given in Figure 1. For a given software, Step 1 is used to determine the sub-attribute values $x_1,\dots ,x_n$, the sub-attribute weight values $w_{i,k}(1\le i\le m+s,1\le k\le n)$ and the parameter values ${\rho }_1,\dots ,{\rho }_{m+s}$. Then the attribute values $y_i(1\le i\le m+s)$ are calculated by STMBDA in Step 2. Step 3 computes critical attribute weight values ${\alpha }_1,\dots ,{\alpha }_m$, the non-critical attribute weight values ${\beta }_{m+1},\dots ,{\beta }_{m+s}$, and the parameter values ρ and ϵ. The above weight values can be calculated using the method proposed in Tao and Chen (2012). In Step 4, STMBDA is utilised to get the degree of software trustworthiness based on the results of Step 2 and Step 3.

Figure 1. Measurement procedure based on STMBDA.

6. Empirical validation

The empirical validation contains case study, survey and experiment (Fenton & Bieman, 2015). An empirical validation of STMBDA is conducted by a real case in this section. The trustworthiness of spacecraft software is one of the crucial factors to guarantee the success of space mission. However, their current evaluation is only qualitative. To tighten the measurement of trustworthiness of the spacecraft software, with the assistance of STMBDA, their trustworthiness is assessed by utilising axiomatic approaches. The trustworthy attributes of spacecraft software contain 9 attributes, subdivided into 28 sub-attributes (J. Wang et al., 2015). The attributes, sub-attributes and the corresponding weight values are described in Table 1 (J. Wang et al., 2015).

Table 1. Trustworthy attributes, sub-attributes of spacecraft software and their weight values.

Attribute	Attribute weight value	Sub-attribute	Sub-attribute weight value
1. Test verification	0.20	(1) Test plan and implementation	$w_{1,1}=0.16$
		(2) Code walk through and static analysis	$w_{1,2}=0.17$
		(3) Test content comprehensiveness	$w_{1,3}=0.17$
		(4) Special testing situation	$w_{1,4}=0.17$
		(5) Test coverage	$w_{1,5}=0.17$
		(6) Test environment, methods and tool usage	$w_{1,6}=0.16$
2. Analysis and design	0.17	(7) Input file controlled condition	$w_{2,7}=0.33$
		(8) Demand analysis	$w_{2,8}=0.33$
		(9) Software design	$w_{2,9}=0.34$
3. Reliability and safety	0.15	(10) Reliability and safety analysis	$w_{3,10}=0.33$
		(11) Reliability and safety design	$w_{3,11}=0.34$
		(12) Reliability and safety verification	$w_{3,12}=0.33$
4. Configuration management	0.11	(13) Configuration management organisation, requirements, and tools	$w_{4,13}=0.33$
		(14) Change control situation	$w_{4,14}=0.34$
		(15) Configuration audit, documentary	$w_{4,15}=0.33$
5. Software technology status change	0.09	(16) Basis, demonstration and approval of technical status change	$w_{5,16}=0.34$
		(17) Tests and verifications after changes	$w_{5,17}=0.33$
		(18) Implementations after changes	$w_{5,18}=0.33$
6. Quality problem close loop	0.09	(19) Quality problem zero completion	$w_{6,19}=0.50$
		(20) Implementation of one example against three tasks	$w_{6,20}=0.50$
7. Third party evaluation situation	0.09	(21) Evaluation input and evaluation plan situation	$w_{7,21}=0.33$
		(22) Evaluation of implementation	$w_{7,22}=0.33$
		(23) Problem solving situation	$w_{7,23}=0.34$
8. Overall planning and implementation	0.05	(24) Development planning and execution	$w_{8,24}=0.31$
		(25) Functionality and performance compliance with specification	$w_{8,25}=0.36$
		(26) Integrity of software development documentation	$w_{8,26}=0.33$
9. Software development environment	0.05	(27) Development supporting software	$w_{9,27}=0.50$
		(28) Development supporting hardware	$w_{9,28}=0.50$

Following the weight values of the nine attributes, the first four attributes are regarded as critical attributes and the last five attributes as non-critical attributes. The following parameter values for STMBDA are then deserved by Table 1: m = 4, s = 5, n = 28, $\alpha =0.20+0.17+0.15+0.11=0.63$, $\beta =0.09+0.09+0.09+0.05+0.05=0.37$, and $\begin{array}{rl}({\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4)& =(\frac{0.20}{0.63},\frac{0.17}{0.63},\frac{0.15}{0.63},\frac{0.11}{0.63})=(0.32,0.27,0.24,0.17),\\ ({\beta }_5,{\beta }_6,{\beta }_7,{\beta }_8,{\beta }_9)& =(\frac{0.09}{0.37},\frac{0.09}{0.37},\frac{0.09}{0.37},\frac{0.05}{0.37},\frac{0.05}{0.37})=(0.24,0.24,0.24,0.14,0.14).\end{array}$If the kth sub-attribute is not in the set of the sub-attributes constituting the ith attribute, then $w_{i,k}=0$, which are not given in this table.

Trustworthiness of each sub-attribute is classified into four levels: A, B, C and D. To calculate the attribute values based on STMBDA, levels of sub-attributes are converted to specific values. Level A is converted to 10, Level B to 9, Level C to 7, and Level D to 2. A panel of 10 experts was invited to classify the 28 sub-attributes of the 23 spacecraft software (J. Wang et al., 2015). The scoring method is that each expert divides them according to the four grades of A, B, C, and D in the form of a combination of subjective and objective principles. For the specific classification standards, please refer to Section 7.2 of the reference (Chen & Tao, 2019). We chose 11 representative software as subjects, which are numbered 2, 4, 6, 7, 9, 18, 19, 20, 21, 22 and 23. Figure 2 shows the distributions of sub-attribute values of these 11 software. The horizontal axis of each subgraph of Figure 2 displays the number of sub-attribute, and the vertical axis shows the sub-attribute values.

Figure 2. Distributions of sub-attribute values of 11 representative software.

Given that the difficulty of the substitution between the sub-attributes constituting critical attributes should be greater than that between the sub-attributes making up non-critical attributes, let ${\rho }_j<{\rho }_i(1\le i\le 4,5\le j\le 9)$. For simplicity, set ${\rho }_1={\rho }_2={\rho }_3={\rho }_4,{\rho }_5={\rho }_6={\rho }_7={\rho }_8={\rho }_9$, their specific values used in this case are given in the Figure 3. The distributions of attribute values computed through STMBDA of these 11 software are presented in Figure 3. In each subgraph of Figure 3, the vertical axis represents the number of attribute and the horizontal axis displays the attribute value. In order to compare with the model given in J. Wang et al. (2015) (referred to as PBSTE3), PBSTE3 is also used to measure the attribute trustworthiness of these 11 representative software. In PBSTE3, both the trustworthiness measurement model and the attribute measurement model are in the form of the product of power functions. According to the sub-attribute weight values given in Table 1, the attribute value distributions calculated by the attribute measurement model in PBSTE3 are shown as the yellow line in Figure 3.

Figure 3. Distributions of attribute values of 11 representative software.

It can be seen from Figures 2 and 3 that the measurement results of attribute trustworthiness obtained by STMBDA can reflect the actual development of spacecraft software. For instance, since such software is developed by GJB5000A standard, the software change control for these 11 software is generally good. Meanwhile, the weaknesses in the software development process are able to be easily identified. For example, these 11 software generally lack special testing, the reason is that the testing verification technology is not advanced because of the design of dynamic timing, space, data use, and control behaviour, etc. Furthermore, the attribute metric model presented herein is more universal when compared to the attribute metric model built in J. Wang et al. (2015). We can adjust software attribute trustworthiness through the parameters ${\rho }_i(1\le i\le 9)$ in STMBDA. If we have higher trustworthy requirements for attribute, then we raise the values of ${\rho }_i(1\le i\le 9)$, and vice versa. However, the distributions of sub-attribute values in Figure 2 demonstrate that some grading criteria are too high to be achieved and some are too low to be easily implemented, thus the grading standards of sub-attribute trustworthiness need to be improved in future applications.

For the given parameter values $({\rho }_1,{\rho }_2,{\rho }_3,{\rho }_4,{\rho }_5,{\rho }_6,{\rho }_7,{\rho }_8,{\rho }_9)=(6,6,6,6,3,3,3,3,3),$the distributions of trustworthy degrees of these 11 software calculated through STMBDA are given in Figure 4. The yellow line in Figure 4 is obtained by the software trustworthiness metric model presented in J. Wang et al. (2015) according to the weight values presented in Table 1. As can be seen from Figure 4, STMBDA does not greatly raise the trustworthy degree of the software due to the high value of individual sub-attribute, but the sub-attributes with lower values will reduce the degree of the software. That is, for the software to be trusted, each attribute must be trusted to a certain value. Meanwhile, the software trustworthiness measurement model built in J. Wang et al. (2015) cannot reflect this situation well. In a similar way, we can adjust the software trustworthiness through the parameter ρ as required. If we have a higher trustworthy requirement for software, then we raise the value of ρ, and vice versa. At the same time, we can regulate the influence of the minimum critical attribute on software trustworthiness by modifying the parameter ϵ

Figure 4. Distributions of trustworthy degrees of 11 representative software.

This case demonstrates that STMBDA is suited to the measurement of the spacecraft software trustworthiness and can effectively assess their trustworthiness and accurately discover the vulnerability in the development process, which is very important for improving the development level of this type of software.

7. Comparative study

In the remainder of this section, we compare STMBDA with PBSTE1 (Tao & Chen, 2009), PBSTE2 (Tao & Chen, 2012), PBSTE3 (J. Wang et al., 2015), evidence theory-based software trustworthiness measure (ERBSTM) (Ding et al., 2012) and fuzzy theory-based software trustworthiness measure (FTBSTM) (Shi et al., 2008) by the properties described in Section 3. The comparison results are given in Table 2, among them, × indicates the measure does not meet the corresponding property and √ represents the measure conforms to the corresponding property.

Table 2. Comparative study by the properties introduced in Section 3.

Property	STMBDA	PBSTM1	PBSTM2	PBSTM3	FTBSTM	ERBSTM
Non-negativity	√	√	√	√	√	√
Proportionality	√	×	×	×	×	×
Monotonicity	√	√	√	√	√	√
Acceleration	√	√	√	√	×	×
Sensitivity	√	√	×	√	√	√
Substitutivity	√	√	√	√	×	×
Expectability	√	×	×	√	√	√

It is easy to prove that all the metric models comply with non-negativity. PBSTM1, PBSTM2, PBSTM3, ERBSTM and FTBSTM do not take into account the issue associated with proportionality, so none of them meet proportionality. Ding et al. (2012) and Shi et al. (2008) demonstrate both ERBSTM and FTBSTM satisfy expectability. J. Wang et al. (2015) show that PBSTM3 conforms to monotonicity, acceleration, sensitivity, substitutivity and expectability.

Next, we give a counter-example to show that neither PBSTM1 nor PBSTM2 satisfies the expectability. For a given software, suppose the number of its critical attributes m is 3 and the number of its non-critical attributes s is 2. The weight vector of critical attributes is $({\alpha }_1,{\alpha }_2,{\alpha }_3)=(0.5278,0.3325,0.1396)$, and that of non-critical attributes is $({\beta }_4,{\beta }_5)=(0.6667,0.3333)$. Let $ϵ=0.01$ and $(y_1,y_2,y_3,y_4,y_5)=(8,8,7,8,8)$. Then the trustworthy degree of this software is 6.69 calculated by PBSTM1 and 6.54 computed by PBSTM2, both of which are less than minimum value of all the trustworthy attributes. Thus, neither PBSTM1 nor PBSTM2 complies with the expectability.

The rest of the comparison results can be obtained from the comparative study of Tao and Zhao (2018).

Finally, it can be concluded from Table 2 that STMBDA is superior to all five methods from the perspective of the properties introduced in Section 3.

8. Conclusion and future work

In this paper, we complete the set of expected properties of software trustworthiness measurement based on attribute decomposition, give two new properties, namely non-negativity and proportionality, improve substitutability and expectability, and introduce STMBDA given in Tao et al. (2015). We verify the theoretical rationality of STMBDA through showing that it complies with the new property set, and the empirical validity by measuring the trustworthiness of 23 spacecraft software. We also present a comparative study, which shows that STMBDA outperforms the other five models in terms of the new property set. It should be noted that we only give the expected property set from the perspective of experience, which is reasonable but not complete. On the other hand, in the empirical validation, we specify the parameter values ϵ, ρ and ${\rho }_i$ in advance and do not give them the specific solving algorithms.

Several issues are worthy for further investigation. Firstly, we believe that the software trustworthiness measure properties given in this paper are necessary, but not sufficient. We will be interested in extending and perfecting this set of properties. Secondly, attribute decomposition-based software trustworthiness measures that do not meet the properties described in Section 3 cannot be taken as legitimate measures. However, metrics satisfying these properties should be used only as candidate metrics, and they still need to be better checked. We have utilised STMBDA to spacecraft software trustworthiness measurement, and we will use STMBDA for trustworthiness measurement of other types of software to conduct a comprehensive empirical validation in the future. Thirdly, we do not give a way to calculate the parameter values ϵ, ρ and ${\rho }_i$ in STMBDA, and how to determine these parameter values is also important for future work.

Acknowledgments

A preliminary version of this work was presented at 20th IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) [Decomposition of Attributes Oriented Software Trustworthiness Measure Based on Axiomatic Approaches (Tao et al., 2020)].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was financially supported by the Doctoral Research Fund of Zhengzhou University of Light Industry [grant number 2016BSJJ037]; and the Science and Technology Project of Henan Province [grant number 202102210351], [grant number 212102210076].