Detecting unknown vulnerabilities in smart contracts using opcode sequences

Abstract

Unknown vulnerabilities, also known as zero-day vulnerabilities, are vulnerabilities in software, systems, or networks that have not yet been publicly disclosed or fixed. If these vulnerabilities are ever discovered by hackers, intentionally or unintentionally, they pose a major threat to network security. This is particularly true in the blockchain field, as smart contracts hold a lot of money, and if they are discovered and exploited by hackers, the financial losses to users will be even greater. However, the current research on smart contract vulnerabilities mainly focuses on known vulnerabilities, and the research on unknown vulnerabilities has been limited. Based on this, we introduce a machine learning-based method for detecting unknown vulnerabilities in smart contracts. First, the method obtains the opcode sequences executed by smart contract transactions in the EVM by instrumenting Geth and replaying the Ethereum transactions. Next, we employ an n-gram model and a vector weight penalty mechanism to extract the opcode sequence features. We then use machine learning algorithms to detect unknown vulnerabilities based on the similarity principle. Finally, we test the effectiveness of our method with four machine learning models: the K-Nearest Neighbor algorithm (KNN), Support Vector Machine (SVM), Logistic Regression (LR), and Decision Tree (DT). The SVM model performs best at detecting unknown vulnerabilities, with an accuracy of 96%, a precision of 91%, a recall of 100%, and an F1-score of 95%. We also discuss the benefits of the method: timely detection of attacks due to unknown vulnerabilities, thus reducing user losses.

Keywords:

• Blockchain
• smart contracts
• unknown vulnerabilities
• N-gram
• opcode sequences

1. Introduction

In recent years, the emergence and use of blockchain technology, particularly with the advent of smart contracts, has led to a wider application of blockchain (Kushwaha et al., 2022) in areas such as financial services (Wan et al., 2018), infrastructure (Y. Zhang et al., 2017), IoT (Christidis & Devetsikiotis, 2016; T. Wang et al., 2021), healthcare (Al Omar et al., 2019; Ekblaw et al., 2016), and others (Rahman et al., 2020; T. Wang et al., 2023). By June 2023, the cryptocurrency market for Ethereum had reached approximately $500 billion in funding. On the one hand, blockchain technology has tamper-proof features (He et al., 2023; Zou et al., 2021) and allows people to complete transactions securely in a trustless network environment, especially in the Bitcoin (Nakamoto & Bitcoin, 2008) and Ethereum (Wood, 2014) ecosystems, which have operated stably for years without any systemic failures. On the other hand, because blockchain involves numerous financial transactions that are not under the control of a single party and everything is governed by the code that forms the basis of the blockchain, it is difficult to be regulated by the appropriate authorities, which attracts the attention of many cybercriminals.

Compared to the data, network, and consensus layers of a blockchain system, the smart contract layer is relatively easy to attack. This is because the former is determined by the underlying code of the blockchain system, which prevents information from being tampered with through techniques such as cryptography and the way everyone maintains a common ledger. It is difficult for attackers to attack the system itself, so they tend to steal information through network attacks or by controlling the hosts of the nodes, but with the continued advancement of trusted computing devices (L. Zhou et al., 2021), the security of the nodes continues to improve, making the underlying blockchain system more difficult to attack. Noted that the latter includes a large number of blockchain contract applications, which are written by users themselves and have many security vulnerabilities (Chen et al., 2020; Kushwaha et al., 2022), such as re-entrancy and incorrect check for authorisation. A large amount of virtual currency is stored in these smart contract accounts, which quickly attracts the attention of a large number of hackers and causes significant losses to blockchain users. For example, 3.6 million ethers were stolen in 2016 by exploiting a re-entrancy vulnerability in a Decentralised Autonomous Organisation (DAO) smart contract (Siegel, 2018). Furthermore, due to a multi-signature wallet contract vulnerability, hundreds of millions of USD were frozen from parity wallets in 2017 (Qureshi, 2017). And due to an integer overflow vulnerability, the $10 billion BeautyChain (BEC) project instantly went to zero in 2018 (Etherscan, 2018).

To address the above issues, previous research has mainly included two types of methods: static analysis methods and dynamic analysis methods. The former is the static analysis of smart contract source codes or bytecodes before contract deployment, which generally helps to detect smart contract vulnerabilities using some static analysis tools (Feist et al., 2019; Tikhomirov et al., 2018) or some symbolic execution methods (Luu et al., 2016; Mossberg et al., 2019; Torres et al., 2018). The latter is to perform vulnerability analysis in a real-running environment after contract deployment, such as with ContractFuzzer (Jiang et al., 2018), which verifies the vulnerability of smart contracts by randomly generating a large number of inputs. These methods can reduce the vulnerabilities of smart contracts to some extent, but such methods often require professional assistance to complete. The detection efficiency is also very low.

Based on the above problems, researchers have proposed some automatic smart contract vulnerability detection methods based on machine learning (Sürücü et al., 2022), aiming to improve the detection efficiency while increasing the detection accuracy. However, the existing machine learning-based contract vulnerability detection methods mainly focus on known vulnerabilities. Unknown vulnerabilities in smart contracts are also rarely investigated. In this paper, we propose a machine learning-based scheme for detecting unknown vulnerabilities in smart contracts based on the similarity principle of vulnerability features. The main contributions to this work are as follows:

• We create a dataset of seven known vulnerabilities through instrumenting Geth and replaying the Ethereum transactions.

• We propose a new data feature extraction method. It combines an n-gram model and a vector weight penalty mechanism to improve the detection of unknown vulnerabilities.

• Based on the similarity principle of vulnerability behavioural features, we propose a machine learning-based unknown vulnerability detection scheme and validate its effectiveness through four machine learning model experiments.

The rest of this paper is structured as follows: In Section 2, the related works are classified and summarised. In Section 3, we then have a short introduction to some concepts around unknown vulnerabilities, similarity principle, and so on. Next, a detailed description of the system scheme for unknown vulnerability detection is given in Section 4. The validation and analysis of the experiments are performed in Section 5. The discussion is given in Section 6. Finally, we conclude the paper and look forward to the future work in Section 7.

2. Related work

There are many static analysis tools available to detect vulnerabilities in smart contracts, such as Slither (Feist et al., 2019) and Smartcheck (Tikhomirov et al., 2018). These tools mainly detect pattern matches in the code that may violate coding guidelines or contain vulnerabilities. The detection rate of vulnerabilities is not high. Symbolic execution (Mossberg et al., 2019) is another important static analysis method that replaces the contract program code with symbols for execution and finds the corresponding vulnerability path in the execution path of the symbols. However, such methods rely excessively on manual assistance. Although there are some intelligent pattern-matching schemes, these need to be manually defined well in advance. In addition, symbolic execution is too prone to problems such as path explosion. Compared to the former, the dynamic analysis method offers better accuracy. However, it requires the right inputs to generate the actual execution, and finding the right inputs is not easy. Commonly used tools include ContractFuzzer (Jiang et al., 2018), Echidna (Grieco et al., 2020), ReGuard (C. Liu et al., 2018), etc. They execute smart contracts by generating a large number of random inputs and then detecting them based on vulnerabilities in the execution log. However, both static and dynamic analysis methods require knowledge and insight into all possible execution paths. Their computational complexity increases dramatically with path depth.

Compared to statistical methods of static or dynamic analysis, the AI model-based automated detection method reduces the reliance on human assistance and makes vulnerability detection more stable. It is more accurate than static analysis methods. It is more efficient than dynamic analysis methods. Xu et al. (2021) introduced a smart contract-based analysis model to support smart contract vulnerability detection by constructing an abstract syntax tree of smart contracts. Wang et al. (2020) extracted the dual format features from the simplified opcodes of smart contracts and used machine learning to identify this contract vulnerabilities. Eshghie et al. (2021) proposed a machine learning-based dynamic vulnerability detection method for smart contracts that supports data collection by building a transaction metadata monitoring tool. Zhou et al. (2022) proposed a tree-based machine learning vulnerability detection method to perform vulnerability analysis of smart contracts, which captures multidimensional features by constructing abstract syntax trees. Xue et al. (2022) proposed a machine learning-guided cross-contract fuzzy testing method that focuses on detecting vulnerabilities generated by cross-contract functions. Shakya et al. (2022) proposed a machine learning-based hybrid model detection method for smart contracts that uses both high-level syntax features and bytecode features to construct a hybrid feature space. Zhang et al. (2022) proposed a smart contract vulnerability detection method based on a hybrid model that strategically combines different word embeddings with different deep learning methods. To enhance robustness, Zhang et al. (2022) again proposed a serial-parallel convolution suitable for the hybrid model.

The above studies on smart contract vulnerability detection are for known vulnerabilities. Unknown vulnerabilities are mainly used in traditional application vulnerability detection. Singh et al. (2019) detected and prioritised zero-day attacks by composing a hybrid method that follows a probabilistic approach to identify zero-day attack paths to detect unknown vulnerabilities that exist but have not occurred in the network. In the social network scenario, Tang et al. (2021) used a neural network algorithm for secondary judgment to check for unknown malicious users based on the periodic activity characteristics of active malicious accounts. Choi et al. (2017) mined the target system for anomalous attacks by extracting the normal or anomalous behaviour of the attacks on the target system. Periyasamy and Arirangan (2019) used vulnerability syntax trees and various predictive models together to accomplish unknown vulnerability detection in applications. Hwang et al. (2020) proposed a semi-supervised learning-based technique that uses event logs to detect unknown attacks without a priori knowledge. Al-Zewairi et al. (2020) used shallow and deep ANN classifiers to detect unknown security attacks. Zhang et al. (2021) proposed a deep learning-based scheme for detecting unknown threats to information systems. Li et al. (2021) used a deep learning approach to detect vulnerabilities using the principle of code similarity to known vulnerabilities and found a total of seven unknown vulnerabilities.

Up to now, there has been little research into the detection of unknown vulnerabilities in smart contracts. In the preliminary version, we propose a machine learning-based scheme for detecting unknown vulnerabilities based on the similarity principle of vulnerability features (2022). In this extended version, we have made additional technical contributions by reconstructing the framework graph of the scheme, increasing the number of datasets for model training, removing the false alarm data in the training set and the known vulnerability data in the test set, re-optimising the definition of the vector weight penalty mechanism, and adding a new trigram model to extract the features to improve the model's detection performance.

3. Preliminaries

In this section, we give a brief description of the unknown vulnerabilities, similarity principle, the blockchain, smart contracts, EVM, opcode, and opcode sequence, as well as the security vulnerabilities involved in our scheme.

3.1. Unknown vulnerabilities

We consider unknown vulnerabilities to be those that already exist but cannot be discovered using existing vulnerability detection methods. Such vulnerabilities are objectively present, even if they have not been discovered. They are hidden threats to network security. If these vulnerabilities are discovered by hackers, intentionally or unintentionally, one day, they will pose a significant threat to network security. There may be some unknown vulnerabilities that are not subject to any constraints and may have no similarities to known vulnerabilities. There may also be some unknown vulnerabilities that have some similarities to one or more known vulnerabilities but cannot be detected using existing vulnerability detection methods. We study here the latter, i.e. unknown vulnerabilities that have similarities with known vulnerabilities.

3.2. Similarity principle

The similarity principle refers to the fact that there are certain similarities in the characterisation of unknown and known vulnerabilities. An opcode sequence is a single execution of a transaction involving a smart contract. If the transaction triggers a vulnerability in the smart contract, then the transaction may be an attack. The opcode sequences that we obtain for smart contract vulnerabilities can be understood as containing smart contract vulnerability features as well as attack features that exploit the vulnerability. Therefore, for the opcode sequences, the similarity of the behaviour of unknown and known vulnerabilities can be analysed from two aspects: From the vulnerability aspect, some vulnerabilities (e.g. timestamp dependency and block number dependency) will use the opcodes to obtain some key information about a block, such as TIMESTAMP and NUMBER, and use it as a comparison, such as EQ, LT, and GT, which will determine the action of the next jump operation, such as JUMP and JUMPI. So there is some similarity in the key opcodes they execute. From an attack aspect, the attacks all cause losses to the user and cause abnormal transfers of a large number of ethers or tokens in a short period of time. As a result, there is some similarity in their attacking behavioural characteristics. Therefore, we can use machine learning techniques to mine the characteristics of known vulnerabilities and then detect unknown vulnerabilities that have similarities to known vulnerabilities.

3.3. Blockchain

Blockchain is a decentralised, distributed database technology that is maintained by multiple nodes, each of which holds a complete copy of the data. Blockchain records form a tamper-proof chain structure, with each block containing the hash of the previous block, creating a record that can never be reversed. Ethereum is a smart contract platform based on blockchain technology that allows developers to build decentralised applications (DApps) and execute smart contracts on them. Ethereum has smart contract programming capabilities, so smart contracts can be written to implement various functions such as token issuance, digital identity verification, etc.

3.4. Smart contract

A smart contract is a computer program designed to automatically enforce the terms of a contract and ensure that the terms are carried out exactly as intended. Smart contracts can be used to replace traditional legal contracts. Their execution is based on blockchain technology, providing decentralisation, security, and transparency. Smart contracts are an integral part of the Ethereum platform, enabling it to support a wide range of decentralised applications.

3.5. Ethereum virtual machine

The Ethereum Virtual Machine (EVM) is one of the core components of Ethereum, a virtual machine that executes smart contracts on the Ethereum network. It uses a stack-based computer architecture with high flexibility and programmability and supports multiple programming languages, including Solidity (Solidity, 2019), Vyper, etc., making Ethereum a powerful platform for smart contracts.

3.6. Source code, bytecode, opcode, and opcode sequence

On EVM, a contract is deployed in three steps. First, the source codes are written by developers in a high-level language (e.g. Solidity). Second, the source codes are compiled into bytecodes using a compiler. Bytecodes are an array of bytes encoded by hexadecimal numbers. Bytecodes can also be converted into opcodes again by EVM. Finally, developers can deploy bytecodes to the Ethereum mainnet by sending transactions. We know that transactions involving smart contracts are sent from external accounts, which in turn trigger the invocation and execution of smart contracts. The smart contracts, in turn, may trigger one or more internal transactions. Thus, a complete path is formed from the time a transaction is issued by an external account to the time when all internal transactions are executed. This complete path is actually the execution trace of all the smart contracts involved in the transaction. Thus, we can represent the path of the transaction in terms of all the opcodes executed sequentially on the trace. These sequentially executed opcodes then form the opcode sequence for the transaction. As shown in the top right of Figure 1, A developer can deploy bytecode to the Ethereum mainnet by sending a transaction that invokes the smart contract's constructor method, which in turn generates an opcode sequence for the constructor method. As shown in the bottom right of Figure 1, the developer can also invoke the bytecode of the store method of the smart contract by sending a transaction, which in turn results in an opcode sequence for the store method. In this paper, the dataset we will use is the dataset of transaction opcode sequences, which are well categorised based on the type of vulnerability.

Figure 1. The relationship among source code, bytecode, opcode, and opcode sequence.

3.7. Security vulnerabilities in smart contracts

(1) Re-entrancy Vulnerability: The smart contract re-entrancy vulnerability is a common security flaw that allows an attacker to repeatedly execute code by calling a contract function multiple times, each time accessing and modifying the state of the contract. Such attacks typically result in the losses of funds for the contract and other unwanted consequences. The nature of the smart contract re-entrancy vulnerability is that contracts do not properly handle updates to state variables when calling external contracts. When a contract calls another contract, it passes the state of the current contract to the called contract. If the called contract is malicious and repeatedly calls a function of the current contract, the state of the current contract could be modified multiple times, leading to unexpected results.

(2) Unexpected Function Invocation: The bytecode sequence of each smart contract contains a scheduler that reads the function ID from the input data of the transaction, matches that function ID against the sequence of function IDs encoded in the scheduler, and determines the function to be called. If no matching function is found, the fallback function is called (Ethereum, 2018). This situation is called an unexpected function invocation (Chen et al., 2020). Unexpected function invocations can have serious consequences.

(3) No Check after Contract Invocation: In Ethereum, there are many ways for a smart contract to call another smart contract, such as sending ethers to another contract using transfer() and send() or calling another contract's function directly. When calling another smart contract within a smart contract, execution will continue even if the called contract throws an exception. If the call fails unexpectedly or if an attacker forces the call to fail, this may result in an exception in the logic of the subsequent program. If the exception is not handled correctly, the contract may behave unpredictably, resulting in lost funds and other unwanted consequences.

(4) Missing the Transfer Event: The lack of standard event vulnerability in smart contracts is due to contracts not properly implementing or using standard events. In Solidity, events are a mechanism for recording and processing changes in the state of a contract. By defining events in the contract, changes to critical data can be reported to external applications and users. ERC-20 is currently the most popular token standard, which states that when one of the standard functions transfer() or transferFrom() is called, the standard event transfer must be fired for notification (Ethereum, 2015). The transfer event should be fired even if the number of tokens transferred is 0. An attacker can easily hide his malicious behaviour by exploiting such a vulnerability to cheat the trading market.

(5) Strict Check for Balance: If the contract logic is only concerned with a specific ether balance, its behaviour may be incorrect. For example, a contract may use the this.balance statement to get the current ether balance of that contract. Some developers will compare the contract balance to a specified amount. If the contract balance is equal to the specified amount, they can perform some sensitive operations (such as sending ethers). But it is not safe to perform strict checks on the contract balance because the contract balance can be manipulated by others. For example, if a contract self-destructs, all its ethers can be sent to the specified account address (Ethereum, 2019). An attacker could make their own contract self-destruct and send the contract balance to the vulnerable contract to manipulate the result of the balance comparison. The vulnerability could result in a loss of ethers. In the worst case, this could cause a denial of service condition, rendering the contract unusable.

(6) Timestamp & Block Number Dependency: Smart contracts often require the use of timestamps or block numbers to perform certain operations, such as determining whether or not a time has elapsed, calculating the hash value of a block, and so on. However, timestamps or block numbers can be easily forged or manipulated by an attacker, thereby invalidating or rendering unreliable the terms or parameters in the contract. An attacker could exploit this vulnerability to perform unauthorised operations such as modifying data, transferring funds, etc.

(7) Incorrect Check for Authorisation: The tx.origin is a global variable in Solidity that returns the address of the account that sent the transaction. Using this variable for authorisation may make the contract vulnerable to attack if the authorised account invokes a malicious contract. If the contract developer uses tx.origin for user identity verification, it may expose the contract to the risk of phishing attacks (Solidity, 2019). A malicious call can bypass the authorisation check because tx.origin returns the original sender of the transaction, in which case the original sender of the transaction is the authorised account but not the malicious contract.

4. System framework for unknown vulnerability detection

In this section, we give a detailed description of the design and implementation of our scheme. Our scheme has the ability to detect and analyse unknown vulnerabilities in smart contracts from historical Ethereum transaction data. Figure 2 provides an overview of our scheme. The scheme consists of four different phases:

1. Data Collection: The main task of the data collection phase is to extract the information of well-classified transaction opcode sequences by replaying Ethereum transactions using TxSpector (M. Zhang et al., 2020) and SODA (Chen et al., 2020) and store them in the database.

2. Data Preprocession: The main task of the data preprocession phase is to first simplify the opcodes by grouping functionally similar opcodes into one category, thus reducing the feature dimensionality. The second task is to vectorise the opcode sequences for use in machine learning.

3. Model Training: Here, we use the known vulnerability opcode sequence vectors and the normal opcode sequence vectors from step 2 to train the model using DT, SVM, KNN, and LR machine learning algorithms.

4. Unknown Vulnerability Detection: We take a known vulnerability as an unknown vulnerability and feed it into the trained model for testing. This known vulnerability does not appear in the training set in step 3. The model's accuracy, precision, recall, F1-score, and ROC curve indicate whether it is good or bad at detecting unknown vulnerabilities.

Figure 2. System framework for unknown vulnerability detection.

4.1. Data collection

In the data collection phase, the data collection is divided into two parts: the first part is to collect information such as opcode sequences during contract transaction execution, which is based on the TxSpector. The second part is to collect transaction tag information, which is based on the vulnerability detection plugin of SODA.

For the first part, we first modify the Ethereum client source code (M. Zhang et al., 2020), deploy the modified client source code, and connect it to the Ethereum mainnet to become a full node. Then we synchronise all the information (G. Wang et al., 2022), such as transactions and accounts. Since all contract transactions are executed during the synchronisation process, the instrumentation Geth will collect the opcode sequences and other data during contract execution. Finally, this data is collected and stored in our database.

For the second part, we first deployed a plugin to collect transaction tagging information in the Ethereum client, which is based on the SODA plugin. The plugin will then monitor smart contract deployment information, invocation information, stack, memory, and storage information, ether transfer information, etc. in real time to determine what type of vulnerability this transaction belongs to and label it. Next, the plugin outputs label data for each contract transaction. Finally, the tag data is associated with the data collected in the first part, one by one, and stored in our database.

In short, we have completed the collection and classification of transaction opcode sequences. To date, our categorised transaction opcode sequence dataset is arguably the first transaction-based vulnerability dataset. The good or bad collection and categorisation of transaction opcode sequences play an important role in the effectiveness of later model training and detection. If the data is misclassified, the model will learn the wrong features, resulting in false positives or negatives in the test results. Therefore, we use two top papers as the basis for the collection and classification. For the classified opcode sequence data, we again performed a manual audit to ensure correctness.

4.2. Data preprocession

In the data collection phase, we stored the data in our database. In the data preprocession phase, the data is transformed into vector data that the machine learning model can learn. Our scheme uses a bigram or trigram model and a vector weighting penalty mechanism.

(1) Simplify Opcodes: We replace several opcodes with similar functions one by one to reduce the variety of opcodes and to avoid dimensional disasters when converting opcode sequences into vectors. The contents of the replacement string are shown in Table 1. The original opcodes are the opcodes to be replaced, and the new opcodes are the opcodes used to replace the original opcodes. The replacement rules are: replace all 32 strings from PUSH1-PUSH32 with PUSH; replace all 16 strings from SWAP1-SWAP16 with SWAP; replace all 16 strings from DUP1-DUP16 with DUP; and replace all 5 strings from LOG0-LOG4 with LOG.

Table 1. Simplified rules for smart contract opcodes.

Original Opcodes	New Opcodes
PUSH1-PUSH32	PUSH
SWAP1-SWAP16	SWAP
DUP1-DUP16	DUP
LOG0-LOG4	LOG

(2) Feature Space: The n-gram processes data to better calculate the statistical relationship between opcodes, so that the characteristics of the data are preserved as much as possible when the data is converted into vectors. The n-gram model (Hassan et al., 2020) is an extension of the bag of words model (L. Liu et al., 2019). N is a number, and N = 1 is the bag of words model. When $N=1$, the model takes only a single opcode as a feature, completely ignoring the order between the opcodes. The single feature cannot reflect the connection between the opcodes. When $N>3$, the feature dimension of the opcode sequences is too large, and the classification complexity is too high. Therefore, this paper uses the n-gram (N = 2 or N = 3), i.e. the bigram model or the trigram model, to extract features. Let the number of the feature be k, and the feature vector of the ith sample (i.e. the ith opcode sequence) be $[m_{i1},m_{i2},\dots ,m_{\mathrm{ij}},\dots ,m_{\mathrm{ik}}]$, where $m_{\mathrm{ij}}$ represents the frequency of the jth feature in the ith sample, and its equation is expressed as: (1) $m_{\mathrm{ij}}=\frac{\mathrm{nu}{m}_{\mathrm{ij}}}{\mathrm{su}{m}_i}$(1) Where the $\mathrm{su}{m}_i$ is the total number of features in the ith opcode sequence. The $\mathrm{nu}{m}_{\mathrm{ij}}$ is the number of occurrences of the jth feature in the ith opcode sequence.

Table 2 shows the simplified opcode feature extraction process. We extract the bigram or trigram features from the simplified opcode sequence. Each transaction has its own feature vector. Each distinct bigram or trigram in the vector is a feature. These features are used to identify vulnerabilities. The feature vectors of all samples in turn form the feature matrix (FM), which is defined as shown in Equation (2(2) $\mathrm{FM}=\left[\begin{array}{cccc}{m}_{11}& m_{12}& \cdots & m_{1k}\\ m_{21}& m_{22}& \cdots & m_{2k}\\ ⋮& ⋮& \ddots & ⋮\\ m_{n1}& m_{n2}& \cdots & m_{\mathrm{nk}}\end{array}\right]$(2) ). (2) $\mathrm{FM}=\left[\begin{array}{cccc}{m}_{11}& m_{12}& \cdots & m_{1k}\\ m_{21}& m_{22}& \cdots & m_{2k}\\ ⋮& ⋮& \ddots & ⋮\\ m_{n1}& m_{n2}& \cdots & m_{\mathrm{nk}}\end{array}\right]$(2) However, there are many common opcodes in the opcode sequence that occur very frequently, so the weights of their corresponding feature vectors are so large that they interfere greatly with the prediction of the model.

Table 2. The features extracted from an opcode sequence.

The Opcode Sequence	PUSH,PUSH,MSTORE,CALLVALUE,DUP
The Bigram Features	PUSH,PUSH;
	PUSH,MSTORE;
	MSTORE,CALLVALUE;
	CALLVALUE,DUP;
The Trigram Features	PUSH,PUSH,MSTORE;
	PUSH,MSTORE,CALLVALUE;
	MSTORE,CALLVALUE,DUP;

(3) Weight Penalty Mechanism: In order to reduce the noise interference of common opcodes, improve the distinction between common opcodes and vulnerability feature opcodes, and better identify the vulnerability feature opcodes, a weight penalty mechanism is used in this paper. If a word appears in a large number of articles, then it may be a very common word that contributes little to the special meaning of a paragraph. One processing method in natural language processing is term frequency-inverse document frequency (TF-IDF) (Davari et al., 2017), which counts the number of texts in which the word appears in all texts. In this case, the inverse document frequency is expressed as: (3) $\mathrm{id}{f}_t=\log \frac{n}{d{f}_t}$(3) Where n is the total number of samples. The $d{f}_t$ is the sample number of all samples containing feature t.

However, since many opcodes appear multiple times in the opcode sequence of each transaction, only the proportion of each opcode in all samples is calculated. The weight penalty of many common opcodes is the same, which does not reflect the true inverse word frequency. Therefore, in order to adapt to this type of data for opcode sequences, this paper proposes a weight penalty mechanism that counts not only the number of samples containing feature t but also the number of occurrences of feature t in each sample. The detailed process is as follows:

Set $w_{\mathrm{ij}}$ as the penalty weight for the jth feature in the ith sample and $\mathrm{nu}{m}_{\mathrm{ij}}$ as the number of occurrences of the jth feature in the ith sample. The sum is the total number of features in all samples. The total number of samples is n. Then the weight penalty equation is expressed as: (4) $w_{\mathrm{ij}}=\log \left(\frac{\mathrm{sum}}{\mathrm{nu}{m}_{1j}+\mathrm{nu}{m}_{2j}++\mathrm{nu}{m}_{\mathrm{nj}}}\right)$(4) Then the corresponding weight penalty matrix FW of the feature matrix FM is constructed as follows: (5) $\mathrm{FW}=\left[\begin{array}{cccc}{w}_{11}& w_{12}& \cdots & w_{1k}\\ w_{21}& w_{22}& \cdots & w_{2k}\\ ⋮& ⋮& \ddots & ⋮\\ w_{n1}& w_{n2}& \cdots & w_{\mathrm{nk}}\end{array}\right]$(5) The final feature vector equation for the jth feature in the ith sample is expressed as: (6) $M_{\mathrm{ij}}=m_{\mathrm{ij}}\ast w_{\mathrm{ij}}$(6) From the formulas (1(1) $m_{\mathrm{ij}}=\frac{\mathrm{nu}{m}_{\mathrm{ij}}}{\mathrm{su}{m}_i}$(1) ), (4(4) $w_{\mathrm{ij}}=\log \left(\frac{\mathrm{sum}}{\mathrm{nu}{m}_{1j}+\mathrm{nu}{m}_{2j}++\mathrm{nu}{m}_{\mathrm{nj}}}\right)$(4) ) and (6(6) $M_{\mathrm{ij}}=m_{\mathrm{ij}}\ast w_{\mathrm{ij}}$(6) ), we get (7) $M_{\mathrm{ij}}=\frac{\mathrm{nu}{m}_{\mathrm{ij}}}{\mathrm{su}{m}_i}\ast \log \left(\frac{\mathrm{sum}}{\mathrm{nu}{m}_{1j}+\mathrm{nu}{m}_{2j}++\mathrm{nu}{m}_{\mathrm{nj}}}\right)$(7) Where n is the total sample quantity. The sum is the total number of features in all samples. The $\mathrm{su}{m}_i$ is the total number of features in the ith opcode sequence. The $\mathrm{nu}{m}_{\mathrm{ij}}$ is the number of occurrences of the jth feature in the ith opcode sequence. Then the feature matrix FM*, composed of all samples, is constructed as follows: (8) $F{M}^{\ast }=\mathrm{FM}\ast \mathrm{FW}$(8) Finally, using the bigram feature extraction as an example, we extracted a total of 1236 dimensional features. The feature vectors of all the samples are shown in Table 3.

Table 3. Feature vectors and vulnerability labels.

No.	Bigram(1236 Dimensions)	Lable
1	0.05869,0.03752,0.04936,…0.00689	1
2	0.03058,0.04107,0.00951,…0.03698	0
3	0.02106,0.01352,0.00036,…0.07026	0
···	···	···
n	0.05692,0.01962,0.03651,…0.06301	1

In conclusion, simplified opcodes can avoid the dimensionality disaster when converting the sequence of opcodes into vectors. The n-gram model can better mine the relationship between the front and back opcodes. The weight penalty mechanism can reduce the noise interference of common opcodes, improve the distinction between common opcodes and vulnerability feature opcodes, and better identify vulnerability feature opcodes. Thus, a good data pre-processing method is beneficial for machine learning models to better learn features and improve vulnerability detection.

4.3. Model training

Using SODA, we collected a total of seven known vulnerabilities. In the model training phase, we took six known vulnerabilities as positive samples and 70% of the normal samples as negative samples. The two parts together form the training set. Depending on the feature vectors and labels of the training set, we experimented with four machine learning models, namely KNN (Hart, 1968), SVM (Hearst et al., 1998), LR (Christodoulou et al., 2019), and DT (Kamiński et al., 2018), for detection and comparison.

(1) KNN: With KNN, the prediction process for each sample is as follows: first, each prediction sample is fed into the model, and all the training samples are sorted according to the distance of the prediction sample; then, the nearest k samples are found; and finally, according to which class of samples has the largest number, the prediction result is the sample of that class.

(2) SVM: The essence of the SVM as a classification model is to find a hyperplane such that it correctly classifies as many of the test samples as possible and is furthest away from the hyperplane.

(3) LR: LR is a classification model that is essentially transformed from a linear regression model and combined with the sigmoid function to transform it into a model outputing probability between 0 and 1, thus forming a classification model.

(4) DT: The decision tree algorithm is a classical classification algorithm. It first pre-processes the data and then generates readable rules and a decision tree with the help of an induction algorithm. The new data is then analysed with the help of the generated decision tree.

In short, different training models adapted to different scenarios have different detection effects. We need to choose them according to the characteristics of the specific problem and data set. In this paper, our detection object is the opcode sequence and high-dimensional data, which is exactly what the SVM model is good at. So through the later experiments, we can see that the SVM has the optimal detection effect.

4.4. Unknow vulnerability detection

In terms of unknown vulnerability detection, we adopt a similar idea to Li et al. (2021), i.e. we use supervised learning methods to detect unknown vulnerabilities by exploiting the similarity principle between unknown and known vulnerabilities. In our scheme, the supervised learning approach is only the first step of unknown vulnerability detection, which focuses on classifying unknown vulnerabilities from a test set into a set of vulnerabilities using the similarity principle. Then, we need to filter the set of vulnerabilities using SODA, i.e. filter the known vulnerabilities and identify the unknown vulnerabilities. To be noticed, the supervised learning approach is only used to detect partial unknown vulnerabilities, the ones that have some similarity to known vulnerabilities.

To verify the correctness of the similarity principle, We treat the known vulnerability: the incorrect check for authorisation, as an unknown vulnerability. Here, the vulnerability of an incorrect check for authorisation does not appear in the training set. We take the incorrect check for authorisation as a positive sample. The remaining 30% of normal samples are used as negative samples. The two parts together form the test set. This test set is first fed into the trained vulnerability detection model, which then detects each sample and determines whether it is vulnerable. As shown in Figure 2, the vulnerability detection results have the following conditions: If the detection score is 0, it means that the similarity between the test samples and the other six vulnerabilities is less than 0.5, and the test samples are marked as normal samples; if the detection score is 1, it means that the similarity between the test samples and the other six vulnerabilities is greater than or equal to 0.5, and the test samples are marked as the samples with a vulnerability. We then use SODA to filter out the known vulnerabilities, leaving us with the unknown ones.

In the following, we will experimentally verify how good the model is at detecting unknown vulnerabilities using metrics such as accuracy, precision, recall, F1-score, and ROC curve.

5. Experimental

5.1. Experimental environment

Due to the relatively large amount of data and the dimensionality of each vector for the experiments we conducted, there are certain CPU, memory, and storage requirements. Our specific server information is shown in Table 4.

Table 4. Experimental environment.

Software and hardware	Configurations
Server Model	HP ProDesk 680 G3
Operating System	Ubuntu 20.04
CPU	Intel i7-7700
Memory Size	64G
Disk Capacity	8T SSD

5.2. Experimental dataset

The experimental data in this paper was obtained by accessing the Ethereum mainnet and replaying contract transactions, as detailed in Section 4.1. This dataset contains a total of 8 types, including a normal sample denoted S0 and seven vulnerability samples, namely re-entrancy, unexpected function invocation, no check after contract invocation, missing the transfer event, strict check for balance, timestamp & block number dependency, and incorrect check for authorisation, denoted S1, S2, S3, S4, S5, S6, and S7, respectively. For the vulnerability samples, we use S7 as a simulated unknown vulnerability, i.e. the samples have never appeared in the training set.

Since many transactions executed on the same path will generate many duplicate opcode sequences, after removing the duplicate or invalid opcode sequences, we select a total of 12,246 samples, of which 2,992 are normal samples S0, 8,462 are known vulnerability samples S1–S6, and 792 are simulated unknown vulnerability samples S7. We use 70% normal samples S0 and 8,462 known vulnerability samples S1–S6 as the training set; we use 30% normal samples S0 and 792 known vulnerability samples S7 as the testing set. The number of each type is shown in Table 5.

Table 5. Data set of opcode sequences.

Dataset	Type	Name	Number	Sum
Training set	S0	Normal	2094	10556
	S1	Re-entrancy	146
	S2	Unexpected Function Invocation	2750
	S3	No Check after Contract Invocation	726
	S4	Missing the Transfer Event	1768
	S5	Strict Check for Balance	686
	S6	Timestamp & Block Number Dependency	2386
Testing set	S0	Normal	898	1690
	S7	Incorrect Check for Authorisation	792

5.3. Evaluation metrics

The target of this paper is the unknown vulnerability of the smart contract, so the simulated samples of the unknown vulnerability are positive samples of the experiment. The normal samples are negative. For binary classification problems, the statistics are a combination of results and predictions, which give the confusion matrix, as shown in Table 6.

Table 6. Confusion matrix.

	Predicted
Real	Positive Examples	Negative Examples
Positive Examples	True Positive(TP)	False Negative(FN)
Negative Examples	False Positive(FP)	True Negative(TN)

Where TP means that the positive example is predicted as a positive example, FN means that the positive example is predicted as a negative example, FP means that the negative sample is predicted as a positive sample, and TN means that the negative sample is predicted as a negative sample.

In order to demonstrate the effectiveness of the smart contract unknown vulnerability detection scheme in this paper, we used accuracy, precision, recall, F1-score, and ROC to evaluate the model. Their calculation formula is as follows:

1. Accuracy: The accuracy shows the classifier's ability to judge the entire dataset. (9) $\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$(9)

2. Precision: The precision represents how many of the predicted positive samples are actually positive. (10) $\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$(10)

3. Recall: The recall is the ratio of the number of samples retrieved to the total number of samples retrieved, which measures the completeness of the retrieval system. (11) $\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$(11)

4. F1-score: The F1-score indicates the weighted average of the precision and recall rates. (12) $F1-\mathrm{score}=\frac{2\ast \mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$(12)

5. ROC: The ROC is a composite indicator reflecting the continuous variables of sensitivity and specificity, with the false positive rate (FPR) and the true positive rate (TPR). (13) $\begin{array}{rl}\mathrm{FPR}& =\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}\end{array}$(13) (14) $\begin{array}{rl}\mathrm{TPR}& =\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\end{array}$(14)

5.4. Results and analysis

Our scheme extracts features by combining the bigram model or trigram model with a weight penalty mechanism and uses four machine learning models (KNN, SVM, LR, and DT) to train and detect samples of unknown vulnerabilities, respectively.

(1) Effect of the Weight Penalty Mechanism: To evaluate the effect of our proposed weight penalty mechanism, we analyse the performance of the bigram model with and without the weight penalty mechanism. The experimental results are shown in Figure 3. We evaluate the effectiveness of unknown vulnerability detection from four perspectives: accuracy, precision, recall, and F1-score.

Figure 3. Experimental results with and without the weight penalty mechanism. (a) Accuracy. (b) Precision. (c) Recall and (d) F1-score.

As a whole, the detection model is effective with or without the weight penalty mechanism. From the experimental results, we can see that the SVM model with the bigram and weight penalty mechanism is the best among the four unknown vulnerability detection models. The accuracy, precision, recall, and F1-score of the SVM model are 93%, 90%, 91%, and 91%, respectively. We can also see that the detection of unknown vulnerabilities achieves good detection results even without using the weight penalty mechanism. From the experimental data, DT and SVM have the best detection effect, and the F1-score reaches 90% and 88%, respectively. KNN and LR are also effective, although their performance is not as good as DT and SVM. This also shows that the similarity principle is effective in detecting unknown vulnerabilities.

Furthermore, we can also see that the inclusion of a weight penalty mechanism allows for better detection of each model. For example, when the weight penalty mechanism is not used, the accuracy, precision, recall, and F1-score of the SVM model are 91%, 88%, 88%, and 88%, respectively; when the weight penalty mechanism is used, the accuracy, precision, recall, and F1-score of the SVM model are 93%, 90%, 91%, and 91%, respectively. The accuracy, precision, recall, and F1-score of the SVM model are improved by 2%, 2%, 3%, and 3%, respectively. The LR model showed the most significant improvement, with 13%, 14%, 10%, and 13% improvements in accuracy, precision, recall, and F1-score, respectively.

In conclusion, it has been verified through experiments that it is effective for unknown vulnerability detection based on the similarity principle. If we add the weight penalty mechanism, the effect will be better.

(2) Comparison with the Bigram or Trigram Model: To verify whether the trigram model can better extract the vulnerability features of the opcode sequence, we experimentally compare the smart contract unknown vulnerability detection scheme when the trigram and bigram models are combined with the weight penalty mechanism to extract the opcode sequence features, respectively. The accuracy, precision, recall, and F1-score of the four machine learning models are shown in Figure 4.

Figure 4. Experimental results with the bigram or trigram model. (a) Accuracy. (b) Precision. (c) Recall and (d) F1-score.

From the experimental results in Figure 4, it can be seen that the accuracy, precision, recall, and F1-score of the SVM model are 96%, 91%, 100%, and 95%, respectively. Compared to the bigram model, the accuracy, precision, recall, and F1-score are improved by 3%, 1%, 9%, and 4%, respectively. It can be seen that using the trigram model improves the effectiveness of unknown vulnerability detection. However, it also results in an increase in overhead.

(3) The ROC Curves: We use receiver operating characteristic (ROC) curves to measure the performance of our scheme (a trigram model combined with a weight penalty mechanism) by measuring the relative costs of TPR and FPR. Obviously, the ideal point is (0, 1), meaning that all positive and negative samples are correctly classified. Therefore, the closer to the top left point, the better the classification result. Figure 5 shows the ROC curve of our scheme. We can see that the SVM model has the best performance, achieving a high TPR even with a low FPR and the largest area under the curve (AUC). This also demonstrates the feasibility of using the similarity principle to detect unknown vulnerabilities.

Figure 5. ROC with different machine learning models.

6. Discussion

In this paper, we propose a method to detect unknown vulnerabilities in smart contracts by replaying Ethereum transactions, and we also verify the effectiveness of our method through experiments. Our method is innovative in the following ways:

1. By reviewing the literature in recent years, it can be seen that vulnerability detection methods for smart contracts basically detect known vulnerabilities, but our method is based on the similarity principle to detect unknown vulnerabilities. Our method can be said to provide some new ideas for detecting unknown vulnerabilities.

2. The opcode sequences we collect are transaction opcode sequences. A transaction opcode sequence represents the execution of a smart contract and is a dynamic feature of a smart contract. It can be used to detect not only unknown vulnerabilities but also unknown attacks. If the instrumented Geth is deployed industrially and becomes a full node, we can detect the transaction opcode sequences to find the attacks on Ethereum, find out the cause of the attacks, fix them, and reduce the user losses.

However, our method is based on the similarity principle to detect unknown vulnerabilities. It also has some limitations, which can lead to false positive and false negative detection results. The specific limitations are as follows:

1. We have limited information about the characteristics of known vulnerabilities.

2. The vulnerability data we use is obtained from SODA. There may be cases where the type of vulnerability is mislabelled in the vulnerability data.

3. There may be some unknown vulnerabilities that are not subject to any constraints and may have no similarities to known vulnerabilities.

7. Conclusion

To date, researchers have proposed many very effective detection methods for known vulnerabilities in smart contracts. However, very little research has been done on unknown vulnerabilities. We propose a method to detect unknown vulnerabilities based on the similarity principle using the opcode sequences of transactions. To get the transaction opcode sequences, we modify Geth and replay the Ethereum transactions. To improve the detection effect, we also propose a weight penalty mechanism. Finally, the experiments also verified the validity of our method.

In future work, to improve the performance of our scheme, in addition to SODA, we will incorporate more and better detection methods to improve the diversity and accuracy of known vulnerability classifications. We will also explore more effective features to characterise the opcode sequences of transactions. In addition, our method is currently only applicable to Ethereum. We will extend this method to other blockchain domains, such as Hyperledger. As described in Section 6, we will also be able to detect attacks through transactions and then find the cause, fix them, and reduce the user losses.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported in part by the National Key Research and Development Program of China (2020YFB1005804), and in part by the National Natural Science Foundation of China under Grant 62372121.