A novel sequence-based prediction method for ATP-binding sites using fusion of SMOTE algorithm and random forests classifier

Abstract

Correctly identifying the protein-ATP binding site is valuable for both protein function annotation and new drug discovery. However, the number of non-ATP-binding residues is much more than the number of ATP-binding residues, which makes the prediction a classical imbalanced learning problem. Previous studies often apply the under-sampling technique to construct a relatively balanced dataset, but some information is inevitably lost during the sample process. In this work, we utilize the SMOTE algorithm, which generates the balanced dataset by generating ATP-binding sites with the idea of interpolation. The Random Forest is selected as classifier to ensure the acceptable training speed. With the combination of complementary template-based method, the prediction performance of the proposed method is further improved. After comparing with other sequence-based predictors, our proposed method achieves satisfying performance and proved to be efficient for ATP-binding sites prediction.

Keywords:

• ATP-binding
• imbalanced-learning
• protein sequences
• SMOTE
• random forest

Introduction

Adenosine-5′-triphosphate (ATP) is an important nucleotide which is involved in various biological processes in cells such as energy metabolism [1], membrane transportation [2], signaling [3] and replication of DNA [4]. As of February 2019, 6885 structures in the Protein Data Bank (PDB) [5] are annotated as ATP-binding, which accounts for about 4.63% among all records. The ATP-binding proteins contain ATP-binding sites which allow the interaction between ATP and proteins. Therefore, accurately identifying the ATP-binding sites in proteins is of great importance for protein function detection [6, 7] and new drug development of antibacterial and anti-cancer chemotherapy[8–10].

Previous efforts to identify the ATP-binding sites in proteins have been made using wet-lab techniques. However, the traditional experimental methods are often cost-intensive and time-consuming, which makes them insufficient for wide application. Based on this background, computational methods which combine the machine learning technique and biological information has been developed to predict the location of various binding sites [11–15]. Two types of biological information are often involved in the prediction of ATP-binding sites: sequence information and structural information. However, until February 2019, the number of known protein sequences collected in Swiss-Prot database [16] which are manually annotated was about 559077, whereas the number of known structures stored in PDB database was merely about 148827. The huge gap between the number of accessions corresponding to the two types of biological information has revealed the wider application prospect of predicting the ATP-binding sites based on sequence information. Therefore, many pioneering works have been proposed in recent years. In 2009, Chauhan et al. [17] introduced ATPint, which is the first custom-designed predictor for ATP-binding sites in protein sequences based on a dataset consisting of 168 non-redundant ATP-binding proteins. The position specific scoring matrix (PSSM) is used to represent the evolutionary feature and the support vector machine (SVMs) is used to obtain the prediction result. In 2011 and 2012, Chen et al. [18, 19] developed two stronger predictors called ATPsite and NsitePred, respectively, based on a larger-scale dataset. Besides the PSSM, a number of new features are applied in these methods such as the residue conservation scores, the predicted secondary structure and collocation of amino acid pairs, which makes them obtain better prediction accuracies. Yu et al. [20] developed the TargetATPsite in 2013 with the idea of classifier ensemble, and successfully improved the prediction performance. In 2015, Andrews et al. [21] developed TSC_ATP, which is a two-layer structure algorithm for the prediction of ATP-binding sites from protein sequences.

Although significant progress has been made by these previous studies, there is still place for performance improvement. The prediction of ATP-binding sites in protein sequences is a typical imbalanced classification problem, since the number of non-ATP-binding residues is much more than the number of ATP-binding residues in protein sequences. For example, in the dataset of ATP227, which was applied in Chen’s [18] work, the number of ATP-binding residues is 3393, whereas the number of non-ATP-binding residues is 80409. Therefore, directly using the traditional machine learning technique may lead to a biased result and poor performance. Previous works often utilized the under-sampling technique, which randomly selects the same number of samples from a majority class as the number in the minority class to construct a relatively balanced data subset that can solve the imbalanced problem [17–21]. However, some valuable properties can be hidden in the non-ATP-binding residues; simply abandoning them will leave the training process incomplete and will have a negative effect on the prediction performance. In 2002, Chawla et al. [22] introduced the algorithm of synthetic minority over-sampling technique (SMOTE) to solve the imbalanced learning problem from another aspect. The SMOTE algorithm does not abandon any sample in the dataset as it creates new samples with the idea of interpolation for the minority class. Compared with the under-sampling technique, the SMOTE algorithm keeps the complete training set without losing any valuable information, which helps the classifier receive more training information and keep more stable performance during the testing process [23]. Compared with simply over-sampling, which just duplicates the minority samples, the SMOTE algorithm generates new minority samples based on interpolation, which efficiently increases the generalization capability and avoids the over-fitting problem for the classifier [24].

In this study, we apply the SMOTE algorithm in combination with the Random Forest classification algorithm to predict the ATP-binding sites based on the information extracted from protein sequences. First, a balanced dataset is constructed by SMOTE; then, various sequence features are extracted and the random forest classifier is built based on these features; finally, in order to further improve the prediction performance, a complementary template-based method is applied, which is integrated with the Random Forest classifier to obtain the final decision. The five-fold cross-validation on two benchmark datasets has shown that our proposed method efficiently improves the prediction performance and provides insights into the mechanism of protein–ATP interaction.

Materials and methods

Datasets

Two benchmark datasets are applied to verify the effectiveness of our proposed method. The two datasets are constructed by previous studies and have been commonly used in other prediction methods.

ATP168: The dataset was originally constructed in the study of ATPint [17]. Briefly, 360 ATP binding proteins are extracted from SuperSite encyclopedia [25]. Then the CD-HIT [26] is applied to remove the redundant protein sequences and reduce the sequence identity to 40%. Finally, the dataset is formed with 168 non-redundant ATP-binding proteins which contains 3,104 ATP-binding residues and 59,225 non-ATP-binding residues.

ATP227: The dataset was firstly curated in the study of ATPsite [18] by Kurgan. The maximum pairwise sequence identity of proteins is reduced to 40% by CD-HIT. A residue is defined as ATP-binding residue if any of its non-hydrogen atoms is less than 3.9 Å away from any non-hydrogen atom in the ATP molecule. Finally, the dataset is constructed with 227 non-redundant ATP binding proteins and contains 3,393 ATP-binding residues and 80,409 non-ATP-binding residues.

Primary sequential feature-based predictor

The problem of imbalanced learning

The prediction of ATP-binding sites is a classical imbalanced learning problem, since the number of non-binding residues is far more than the number of binding residues. The imbalance between the two classes often leads to biased prediction performance as the classifier is not sensitive enough for binding residues. In order to avoid this situation, previous studies tend to apply the under-sampling technique which randomly extracts a similar number of non-binding residues and combines them with binding residues to construct a relatively balanced dataset. However, the under-sampling technique will inevitably lose some non-binding residues which are equally important for the classifier because some valuable binding properties are hidden in these non-binding residues. In this study, the SMOTE algorithm is utilized to solve the imbalanced learning problem which will not lose any information in the dataset. For each binding residue, the SMOTE algorithm first randomly selects a neighboring binding residue according to the distance from the feature matrix, for example the Euclidean distance. Then a straight line is drawn to connect the binding residue and its neighboring binding residue. Finally, a new binding residue sample will be generated by selecting a point in the straight line based on the idea of interpolation. Compared with traditional over-sampling which increases the number of minority class by simply duplicating samples, SMOTE algorithm has the advantage that the classifier can be more general and efficiently avoid being over-fitted. The formula of generating new binding residue sample for SMOTE algorithm is shown as follows: $$X_{\mathit{new}}=X_{\mathit{old}}+\mathit{rand}\left(0,1\right)\times \left(X_{\mathit{neighbor}}-X_{\mathit{old}}\right)\mathrm{ }\left(1\right)$$

Where $X_{\mathit{new}}$ stands for the new-generated binding residue sample, $X_{\mathit{old}}$ stands for the binding residue in the training set and $X_{\mathit{neighbor}}$ stands for the neighboring binding residues of $X_{\mathit{old}}.$

Feature representation

In this study, we utilize a sliding window of size 15 including the centered residue and 7 adjacent residues on the left and right side to represent the feature pattern. The size of the sliding window is determined by experimental attempts. We tried different sizes for the sliding window from 1 to 19, the Matthews Correlation Coefficient (MCC) values corresponding to different sizes of sliding window on two benchmark datasets are shown in Figure 1.

Figure 1. MCC values for different sizes of sliding window on ATP-168 and ATP-227.

After determining the size of the sliding window, these features are considered to construct the feature matrix including:

1. Position Specific Scoring Matrix (PSSM)

   Position Specific Scoring Matrix (PSSM) well encodes evolutionary information of a protein sequence. Previous studies have shown its significance in ligand binding sites prediction and structure prediction, such as protein-RNA binding site prediction and protein secondary structure prediction [20, 27].

   For a query sequence, the PSSM is generated by running PSI-BLAST [28] to search against the Swiss-Port database with three iterations and E-value of 0.001. Then, we normalize the original values in the PSSM to the range of 0 and 1 with logistic function as follows: $$S_{i,r}^{\left(N\right)}=\mathrm{ }\frac{1}{1+e^{-S_{i,r}}}\mathrm{ }\left(2\right)$$

   Where the $e^{-S_{i,r}}$ represents the original value in the PSSM and the $S_{i,r}^{(N)}$ represents the normalized value. With higher value in the PSSM, the residue is more conserved during the evolutionary process as it is probably a residue with specific functions. With the sliding window of size 15, the dimension of the PSSM feature is 20*15 = 300.

2. Predicted secondary structure

   Previous studies have shown that secondary structures are relevant to ATP-binding residues; different secondary structure composition is noticed in ATP-binding residues and non-ATP-binding residues. In this study, the PSIPRED [27] software is applied to provide the secondary structure information which predicts the probabilities belonging to three types of secondary structure including coil (C), helix (H) and strand (E) for each residue based on protein sequences. Therefore, for a sliding window of size 15, the dimension of predicted secondary structure feature is 3*15 = 45.

3. Predicted solvent accessibility

   The value of solvent accessibility represents the surface area of a given residue that is accessible to the solvent. A residue with higher solvent accessibility often means that it has more potential to interact with molecules in an outer environment. In this study, the ASAquick [29] is utilized to predict the solvent accessibility with sequence information. Therefore, the dimension of solvent accessibility feature for the sliding window is 1*15 = 15.

4. Physicochemical property

   According to the dipoles and volumes of side chains of amino acids, 20 types of amino acids can be divided into 7 classes as follows: class a = {Ala, Gly, Val}, class b = {Ile, Leu, The, Pro}, class c = {His, Asn, Gln, Trp}, class d = {Tyr, Met, Thr, Ser}, class e = {Arg, Lys}, class f = {Asp,Glu}, class g = {Cys}. Therefore, a 7-dimensional one-hot binary key is used to encode the physicochemical property for each residue. For the sliding window of size 15, the dimension of physicochemical property is 7*15 = 105.

   In summary, for each residue in a protein sequence, four types of biological information are extracted including PSSM (300), predicted secondary structure (45), predicted solvent accessibility (15) and physicochemical property (105). The dimension of the feature matrix is 300 + 45 + 15 + 105 = 465.

Random Forest classifier

The Random Forest (RF) algorithm has been successfully applied in many fields for data classification and achieves satisfying performance. RF uses an ensemble of unpruned decision trees, each of which is built from a bootstrap sample of the training data using a randomly selected subset of variables. It is worth mentioning that after utilizing the SMOTE algorithm, the scale of the dataset has increased, which means that the training process will become more time consuming. Based on this background, the RF classification algorithm is selected as our classifier. The reason is that for large-scale of high-throughput training data, the RF classifier shows better training speed than other classifiers, for example the support vector machines (SVM). After the SMOTE process, the training speed of the RF classifier for ATP-168 dataset is about five minutes, whereas the training speed of SVM is more than ten hours. Therefore, in order to solve the imbalanced learning problem and make our method more practicable, the combination of SMOTE algorithm and RF classifier is used to build the primary feature-based ATP-binding sites predictor.

Complementary template-based method

Inspired by previous methods for RNA-binding site prediction [30, 31], we also develop a template-based prediction method as complement to improve the prediction performance. The template-based method detects the sequence homology between the query sequence and sequences in the training set. The small homologous segments in the query sequence probably have similar functions to the segments in the training data, which are called the templates. In this study, the templates are detected using the bit-score provided by the PSI-BLAST program. It is worth mentioning that even though the sequence pairwise identity is less than 40% among the dataset, small similar segments still exist which can be used for template detection. If the bit-score between the query sequence and one sequence in the training set is larger than a threshold, we will check the template in the training set to figure out whether it contains an ATP-binding site or not. If the binding site is detected, the corresponding residue in the similar segment from the query sequence will be labeled as ATP-binding. Otherwise, if the ATP-binding site is not included in the template or there is no template in the training set, the complementary template-based method will not be applied. The bit-score threshold in the proposed method is set to 50.

Architecture of the proposed method

The architecture of the proposed method is shown in Figure 2. The training process of the primary sequential feature-based predictor is shown in green lines; the testing process of the primary sequential feature-based predictor is shown in blue lines and the process of complementary template-based predictor is shown in orange lines. For a query protein sequence, PSI-BLAST is used to generate the PSSM profile, PSIPRED is applied to generate the predicted secondary structure and ASAquick predicts the solvent accessibility. The physicochemical feature is directly calculated according to the composition of protein sequence and is combined with these inputs to construct the feature matrix. Then the feature matrix is sent into the RF classifier which is trained with a relatively balanced dataset constructed by the SMOTE algorithm to generate the probability of ATP-binding. Finally, the complementary template-based method is applied to detect the ATP-binding site from segment homology aspect. The final prediction result is obtained by combining the output of the RF classifier and template-based method. The corresponding framework of the prediction procedure is shown in Figure 3.

Figure 2. Architecture of the proposed method.

Figure 3. Framework of the proposed prediction procedure.

Results and discussions

Performance evaluation indices

In this study, five-fold cross-validation is used to assess the prediction performance of the proposed method. First, all the residues in the dataset are randomly divided into five parts. Then four parts are used to train the classifier and the remaining part is used to examine the prediction performance. Finally, this procedure is repeated five times to make sure that each part has been applied for testing. The final prediction performance will be obtained by averaging the prediction performance of five parts.

Four routinely used evaluation indices are selected to show the effectiveness of our proposed method including the overall accuracy (ACC), sensitivity (Sen), specificity (Spe) and the Matthews Correlation Coefficient (MCC), which is one of the most potent evaluation criteria for the binary classifier. These evaluation indices can be defined as follows: $$\mathrm{ACC}=\mathrm{ }\frac{TP+TN}{TP+TN+FP+FN}\mathrm{ }\left(3\right)$$ $$\mathrm{Sensitivity}=\mathrm{ }\frac{TP}{TP+FN}\mathrm{ }\left(4\right)$$ $$\mathrm{Specificity}=\mathrm{ }\frac{TN}{TN+FP}\mathrm{ }\left(5\right)$$ $$\mathrm{MCC}=\mathrm{ }\frac{TP\mathrm{*}TN-FP\mathrm{*}FN}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}}\mathrm{ }\left(6\right)$$ where the TP, TN, FP and FN represent the number of true positive, true negative, false positive and false negative samples, respectively. However, these four indices are threshold-dependent, which means they are influenced by the threshold of prediction. The reason is that for a soft-type predictor which outputs a continuous numeric value to represent the probability of a sample belonging to a predicted class, applying different prediction thresholds will produce different prediction confusion matrices, which leads to different values for the evaluation indices. How to objectively report these indices is an important problem. For the situation of imbalanced learning, overly pursuing the high overall accuracy is not appropriate and may lead to deceiving prediction performance as the number of the majority class greatly exceeds the number of the minority class. Thus, in order to objectively show the prediction performance, the evaluation index of AUC is introduced, which is the area under the Receiver Operating Characteristic (ROC) curves. The introduction of AUC, which is threshold-independent, increases in direct proportion to the prediction performance.

For threshold-dependent evaluation indices, inspired by the work of ATPsite and NsitePred, the threshold that maximizes the prediction’s MCC value is selected to report the prediction performance.

SMOTE algorithm helps to improve the prediction performance

As severe imbalance exists between the ATP-binding residues and non-binding residues, in this study, we introduce the SMOTE algorithm to the problem of ATP-binding prediction instead of under-sampling that has been commonly used in previous studies. Compared with under-sampling, the SMOTE algorithm will not lose any training information and constructs the balanced dataset using manually generated samples instead of simply duplicating the minority samples, which efficiently avoids over-fitting for the classifier. In order to show the effectives of the SMOTE algorithm in ATP-binding prediction, for two benchmark datasets, we utilize under-sampling, original dataset without under-sampling and the SMOTE algorithm, respectively, to construct the RF predictors. The prediction performance of the RF classifier over five-fold cross-validation is shown in Table 1.

Table 1. Prediction performance based on the under-sampled dataset, original dataset and SMOTE dataset over five-fold cross-validation.

Dataset	Method	ACC	Sensitivity	Specificity	MCC	AUC
ATP-168	Under-sampling	0.949	0.349	0.981	0.386	0.841
	Original data	0.959	0.255	0.997	0.434	0.847
	SMOTE	0.959	0.383	0.989	0.480	0.873
ATP-227	Under-sampling	0.958	0.397	0.983	0.412	0.848
	Original data	0.967	0.308	0.996	0.476	0.856
	SMOTE	0.969	0.439	0.992	0.535	0.896

According to Table 1, the classifier constructed on datasets applied with the SMOTE algorithm achieves the best performance among three predictors, with the highest ACC of 0.959 and MCC of 0.480 on ATP-168, highest ACC of 0.969 and MCC of 0.535 on ATP-227 respectively. By comparing the performance of the under-sampling technique and SMOTE algorithm, we can find that the sensitivity and specificity are improved on both datasets, which means the classifier has better distinguishing capacity on both ATP-binding residues and non-binding residues. Since the datasets after SMOTE and under-sampling are relatively balanced, the improvements can be explained by: first, the dataset of under-sampling does not contain all the non-binding residues which are equally valuable for prediction performance; second, the SMOTE algorithm generates the new minority samples based on random linear interpolation, which makes the newly generated samples as significant as the samples in the original dataset for prediction. When comparing the performance of the SMOTE algorithm and the original data, the specificity of the dataset after the SMOTE algorithm is slightly lower than that of the original dataset. This can be explained by the severe imbalance of the original dataset: it contains much more non-binding residues than ATP-binding residues. Therefore, the classifier is biased toward non-binding residues. However, for ATP-binding residues, the sensitivity of the dataset after the SMOTE algorithm greatly surpasses the sensitivity of the original dataset, which is 0.128 and 0.131 higher on ATP-168 and ATP-227, respectively. The improvements have shown that the classifier based on datasets applied with the SMOTE algorithm has better ability to identify the ATP-binding residues in protein sequences.

However, these four indices are threshold-dependent, in order to further demonstrate the effectiveness of the SMOTE algorithm on ATP-binding site prediction, we draw the ROC curves to compare the prediction performance of classifiers constructed on datasets applied with SMOTE algorithm, original datasets without under-sampling and datasets with under-sampling. Figure 4a, b shows the prediction ROC curves on ATP-168 and ATP-227.

Figure 4. Prediction ROC curves of datasets with SMOTE algorithm, original datasets without under-sampling and datasets with under-sampling on ATP-168 (a) and ATP-227 (b).

From Figure 4, it can be found that the classifier on SMOTE datasets outperforms the classifiers on the under-sampled dataset and the original dataset on both ATP-168 and ATP-227. The AUC value, which is threshold-independent, is calculated on both datasets. For ATP-168, the AUC for SMOTE, original dataset and under-sampling are 0.873, 0.847 and 0.841, respectively. For ATP-227, the AUC for SMOTE, original dataset and under-sampling are 0.896, 0.856 and 0.848, respectively. The best AUC values for both datasets are obtained by the classifier with the SMOTE algorithm, which is coincident with the result of threshold-dependent indices.

Based on the performance comparison on both threshold-dependent and threshold-independent evaluation indices, the SMOTE algorithm that we applied to train our prediction classifier can efficiently restrain the negative effect from data-imbalance on ATP-binding site prediction. Besides, it can further improve the prediction performance as compared with the under-sampling technique.

Effectiveness of the complementary template-based method

Inspired by Yang’s work in predicting the RNA-binding sites, we develop a complementary template-based method for ATP-binding site prediction. The method is described in the Materials and methods section. For five-fold cross-validation, four parts are used for training and the remaining part is used for testing, the template will be searched from four training parts. In order to evaluate the usefulness of the complementary template-based method, we compare the prediction performance with and without the template-based method on both ATP-168 and ATP-227 datasets. The prediction performance over five-fold cross-validation is shown in Table 2.

Table 2. Comparison of prediction performance with and without the template-based method on ATP-168 and ATP-227 over five-fold cross-validation.

Dataset	Method	ACC	Sensitivity	Specificity	MCC	AUC
ATP-168	With template	0.959	0.383	0.989	0.480	0.873
	Without template	0.958	0.341	0.990	0.451	0.865
ATP-227	With template	0.969	0.439	0.992	0.535	0.896
	Without template	0.968	0.395	0.993	0.511	0.892

It can be observed that the MCC value improves by 0.029 and 0.024 on ATP-168 and ATP-227, respectively, after appending the complementary template-based method. Besides the MCC value, the sensitivity also improves by 0.042 and 0.044 for both datasets, which demonstrates that the complementary template-based method indeed helps to identify the ATP-binding residues in the protein sequence. Similar to the validation process for the SMOTE algorithm, we also examine the effectiveness of our complementary template-based method with threshold-independent indices. Figure 5 shows the ROC curves of the prediction result with and without the template-based method on both ATP-168 and ATP-227 datasets.

Figure 5. ROC curves of prediction result with and without template-based method on ATP-168 (a) and ATP-227 (b).

We can find that the prediction performance with the complementary template-based method is better than the performance without the template-based method. On ATP-168, the AUC of the prediction with the template-based method is 0.873, whereas the AUC without the template-based method is 0.865. On ATP-227, the AUC values with and without the template-based method are 0.896 and 0.892. The improvements on both threshold-dependent and threshold-independent evaluation indices have shown that the combination of the primary feature-based method and the complementary template-based method achieves better performance compared to the single prediction method. This indicates that the combination strategy is efficient for ATP binding-site prediction.

Performance comparison with previous works

Comparison with ATPint

ATPint is the first prediction method which was specifically designed for protein-ATP binding sites proposed by Chauhan et al. [17]. ATPint utilizes evolutionary features extracted from protein sequential information and SVM as classifier for prediction. Chauhan et al. [17] tried a number of thresholds to reveal the prediction performance under different thresholds and finally the threshold whose sensitivity and specificity was nearly equal was selected to report the performance. In order to fairly compare the prediction performance with ATPint, we also adjusted our threshold to balance the sensitivity and specificity similar to the process of ATPint. The ATP-168 dataset was created in the work of ATPint; therefore, we performed five-fold cross-validation on ATP-168 to compare with ATPint. The comparison result is shown in Table 3.

Table 3. Performance comparison of our proposed method with ATPint on ATP168 over five-fold cross-validation.

Method	ACC	Sensitivity	Specificity	MCC	AUC
ATPint	0.751	0.744	0.758	0.249	0.823
Proposed method	0.790	0.777	0.785	0.291	0.873

From Table 3, it is easy to observe that all of four threshold-dependent evaluation indices are improved. The ACC improves from 0.751 of ATPint to 0.790 of proposed method and the MCC improves from 0.249 to 0.291. For AUC, which is threshold-independent, our proposed method still obtains a better performance of 0.873 over ATPint (0.823). The higher value in MCC demonstrates that our proposed method has better performance than ATPint under a suitable threshold, whereas the higher value in AUC indicates that our method obtains overall better performance than ATPint on both ATP-binding residues and non-ATP-binding residues regardless of which threshold is selected.

Comparison with other sequence-based methods

Chen et al. [18] created the dataset of ATP-227 in their work of ATPsite. Since then, ATP-227 dataset is widely used as benchmark dataset for ATP-binding site prediction. Chen et al. [19] also developed another prediction method called NsitePred, which utilizes the sequential information combined with predicted structural information. Yu et al. [32] developed the TargetATP, which combines Adaboost and SVM to predict the ATP-binding sites. Then they developed another prediction method named TargetATPsite [20], which utilizes the image sparse representation technique to encode the evolutionary information for prediction. Andrews and Hu [21] developed a two-stage prediction method called TSC_ATP with the structure of two-layered K-Nearest Neighbors (K-NN) classifier. These mentioned methods all utilize the under-sampling technique to solve the imbalanced data problem. In order to fairly compare with them, we also made prediction on ATP-227 over five-fold cross-validation. The prediction results comparison is shown in Table 4.

Table 4. Prediction results comparison of our proposed method with other sequence-based methods on ATP-227 over five-fold cross-validation.

Method	ACC	Sensitivity	Specificity	MCC	AUC
ATPint	0.648	0.539	0.651	0.078	0.627
ATPsite	0.962	0.361	0.988	0.433	0.854
NsitePred	0.960	0.444	0.982	0.460	0.861
TargetATP	0.966	0.412	0.990	0.501	0.895
TargetATPsite	0.966	0.445	0.989	0.520	0.881
TCS_ATP	0.962	0.419	0.985	0.460	0.860
Proposed method	0.969	0.439	0.992	0.535	0.896

As shown in Table 4, for threshold-dependent evaluation indices, our proposed method achieves the highest ACC and MCC of 0.969 and 0.535, which outperforms other sequence-based prediction methods in the table. At the same time, for threshold-independent evaluation indices, the AUC of our proposed method is 0.896, which is also better than those of the other methods. The result of the performance comparison demonstrates that our proposed method keeps outstanding and stable performance among other sequence-based predictors. The improvement can be explained from two aspects: first, the training set keeps the full information and solves the imbalanced problem by the SMOTE algorithm, whereas other sequence-based methods sacrifice some valuable information to keep the dataset balanced. Second, besides the classifier trained by the SMOTE algorithm, a template-based method is developed to further improve the prediction performance from homologous segments in the sequence.

In order to further prove the generalization ability of our proposed method, an independent testing set constructed by Chen et al. [19] is applied. The independent testing set consists of 17 ATP-binding protein chains which share less than 40% sequence identity with protein chains in ATP-227. This process keeps the testing set strictly independent and can be correctly used to reveal the prediction performance. Based on this background, we train our predictor on the ATP-227 dataset and the independent testing set is used to evaluate the generalization ability of our proposed method. Table 5 lists the performance comparison of our proposed method and other sequence-based methods on the independent testing set.

Table 5. Performance comparison of our proposed method with other sequence-based methods on an independent testing set.

Method	ACC	Sensitivity	Specificity	MCC	AUC
ATPint	0.655	0.512	0.660	0.066	0.606
ATPsite	0.969	0.367	0.991	0.451	0.868
NsitePred	0.967	0.460	0.985	0.476	0.875
TargetATP	0.969	0.489	0.989	0.542	0.912
TargetATPsite	0.972	0.458	0.991	0.530	0.882
Proposed method	0.973	0.504	0.990	0.553	0.912

It can be found from Table 5 that our proposed method obtains the AUC of 0.912, which is equal to TargetATP but much better than the other predictors. However, the MCC and ACC of our proposed method are 0.553 and 0.973, respectively, which are higher than those of TargetATP (equal to 0.542 and 0.969, respectively). The MCC and ACC of our proposed method are also much better than those of the other sequence-based prediction methods. The results indicate that our proposed method has good generalization ability for ATP-binding sites prediction.

Case study

Inspired by previous works [18, 19, 32], in order to show the prediction performance more visually, we display the prediction result of the protein sequence whose PDB id is 1O93_A in Figure 6. The 1O93_A shares less than 40% sequence identity with the sequences in ATP-227; therefore, the ATP-227 dataset is used as the training set. Besides our proposed method, predictors constructed on ATP-227 with the under-sampling technique and the original ATP-227 dataset without under-sampling are utilized. At the same time, predictor TargetATP, which shows second best performance in the performance comparison section, is also involved. The result of TargetATP is obtained through its online-service (http://www.csbio.sjtu.edu.cn:8080/TargetATPsite/).

Figure 6. Binding sites predicted by the proposed method, predictor constructed on under-sampled dataset, predictor constructed on original dataset and TargetATP for protein 1O93_A.

Note: Black dots indicate the true binding sites in the protein sequence; blue dots indicate the binding residues predicted by the proposed method; red, orange and green dots represent the result of TargetATP, predictor constructed on under-sampled dataset and predictor constructed on original dataset, respectively. The probability curves of involved predictors are drawn in corresponding colors.

It can be observed that the predictor based on under-sampling correctly predicts 8 binding residues among 11 true binding residues, but with 61 false positive samples at the same time (Figure 6). The predictor based on original dataset correctly predicts 7 binding residues with 16 false positive samples. TargetATP correctly predicts 5 binding residues with 5 false positive samples. However, our proposed method correctly predicts 7 of 11 true binding residues with no false positive samples. The MCC of predictors based on the under-sampled dataset, the original dataset, TargetATP and the proposed method on 1O93_A is 0.246, 0.418, 0.462 and 0.793, respectively. The prediction result indicates that our proposed method has better identifying ability of ATP-binding residues and efficiently reduces the number of false positive samples, which is important for further biological experiments.

Conclusions

In this study, we developed a sequence-based prediction method for ATP-binding sites in protein sequences. Compared with previous methods, our proposed method utilizes the SMOTE algorithm, which does not lose any information in the dataset to solve the imbalanced learning problem. With the combination of Random Forest classifier, which provides satisfying training speed and classification performance, our method obtains better performance than predictors trained with the under-sampled dataset or the original dataset. In order to further improve the prediction performance, a complementary template-based method which detects homology of small segments in the query sequence is also developed. The results of the two prediction methods are merged to generate the final prediction. Compared with other sequence-based prediction methods, our proposed method obtains higher MCC and AUC values over five-fold cross-validation on benchmark datasets. On the independent dataset that is used to reveal the generalization ability, our proposed method keeps the stable performance, which demonstrates that our method can reliably predict the ATP-binding sites in protein sequences and provide assistance for experimental researchers.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are openly available in BMC Bioinformatics at https://doi.org/10.1186/1471-2105-10-434, reference [17], and in Proteome Science at https://doi.org/10.1186/1477-5956-9-s1-s4 [18].

Additional information

Funding

This work was supported by The National Nature Science Foundation of China (Grant Nos. 61662057, 61772226 and 61862056), The Natural Science Foundation of Jilin Province (Grant number No. 20200201159JC), Key Laboratory for Symbol Computation and Knowledge Engineering of the National Education Ministry of China, Jilin University.