Machine learning-based prediction of proteins’ architecture using sequences of amino acids and structural alphabets

Abstract

In addition to the growth of protein structures generated through wet laboratory experiments and deposited in the PDB repository, AlphaFold predictions have significantly contributed to the creation of a much larger database of protein structures. Annotating such a vast number of structures has become an increasingly challenging task. CATH is widely recognized as one the most common platforms for addressing this challenge, as it classifies proteins based on their structural and evolutionary relationships, offering the scientific community an invaluable resource for uncovering various properties, including functional annotations. While CATH annotation involves – to some extent – human intervention, keeping up with the classification of the rapidly expanding repositories of protein structures has become exceedingly difficult. Therefore, there is a pressing need for a fully automated approach. On the other hand, the abundance of protein sequences stemming from next generation sequencing technologies, lacking structural annotations, presents an additional challenge to the scientific community. Consequently, 'pre-annotating’ protein sequences with structural features, ensuring a high level of precision, could prove highly advantageous. In this paper, after a thorough investigation, we introduce a novel machine-learning model capable of classifying any protein domain, whether it has a known structure or not, into one of the 40 main CATH Architectures. We achieve an F1 Score of 0.92 using only the amino acid sequence and a score of 0.94 using both the sequence of amino acids and the sequence of structural alphabets.

Communicated by Ramaswamy H. Sarma

Keywords:

• CATH
• protein’s structure
• protein’s architecture
• SCOP
• machine learning
• structural alphabets
• protein blocks
• k-mer

Introduction

The extensive utilization of both databases CATH (Class, Architecture, Topology, Homology) (Sillitoe et al., 2021) and SCOP (Structural Classification of Proteins) (Andreeva et al., 2020) databases in the field of proteomics has been pervasive (Fox et al., 2015). Their significance stems from their repository of high-quality data, encompassing a vast wealth of knowledge and encyclopaedic expertise pertaining to proteins and their interrelationships (Dawson et al., 2016). Understanding sequence-structure-function triage is considered one of the main purposes such databases were established for. Due to the increasing quantity of information in each component of this complex relationship (sequence-structure-function), both classification hierarchies have been gaining much attention; this three-actor relationship plays a critical role in many fields in protein biology such as understanding disease mechanisms and designing proteins and drugs for therapeutic and bioengineering applications (Bepler & Berger, 2021). To serve the purpose, the creation of each grouping within their classification systems has been the result of meticulous, systematic analyses of protein structures, coupled with a deep understanding of protein function and evolutionary dynamics (Andreeva, 2016). These resources have proven invaluable to experimental and structural biologists, aiding them in the examination and exploration of protein structures akin to those relevant to their research (Andreeva & Murzin, 2010). Furthermore, computational biologists have harnessed the extensive spectrum of protein structural and evolutionary relationships within these databases for benchmarking and assessing protein structure comparison and prediction methods (van Kempen et al., 2023). The straightforward hierarchical classification system has also fostered the development of numerous tools and algorithms, effectively employed across various applications. These databases have extended their utility to other realms of protein research, including protein structure prediction and large-scale genome analyses and annotations (Lewis et al., 2013; Oates et al., 2015) as well as in predicting protein-protein interactions, correlating protein structure with enzymatic activity, and conducting investigations aimed at comprehending the intricacies of the protein repertoire (Andreeva et al., 2020; Chothia et al., 2003).

Whilst significant efforts were being made to annotate the high number of structures deposited in the PDB via experimental means, in August 2022, the PDB decided to include, alongside the ∼200,000 experimentally determined structures, ∼1 million Computed Structure Models (CSMs) predicted mainly by AlphaFold (Tunyasuvunakool et al., 2021). This has made the task of annotating protein structures extremely challenging. Currently, besides the standard (experimentally-determined) database of protein structures for which both CATH and SCOP have already fallen short (73% and 56%, respectively—See Figure 1), all computed structure models obtained from the AlphaFold Database (Alpha Fold DB) (Varadi et al., 2022)—a collaborative effort between the European Bioinformatics Institute (EMBL-EBI) and DeepMind – lack any CATH/SCOP official annotations in the PDB. The size of the overall AlphaFold database currently exceeds 214 million, and only a tiny fraction (∼1 M) has been added to the PDB (the latest set was added in February 2023) after passing the predicted local distance difference test (pLDDT) quality/confidence threshold; further information on pLDDT and the process AlphaFold adopts to evaluate their models can found in the literature (Tunyasuvunakool et al., 2021). In general, two main approaches have been employed for CATH/SCOP classification/annotation purposes and can be simply described as (1) sequence-based and (2) structure-based. Whilst the ultimate goal for both approaches is detecting homologous proteins, the former relies on sequence similarity search, the latter uses structural alignment methodologies (superpositions). However, challenges have arisen; accuracy in detecting distant evolutionary relationships and the inefficiency in handling the vast number of structures, respectively (van Kempen et al., 2023).

Figure 1. Schematic comparison between SCOP and CATH.

One way to help annotate those structures has been to employ machine learning (ML) and/or deep learning (DL) algorithms (Bileschi et al., 2022; Nallapareddy et al., 2023). Such algorithms have proven effective in addressing various challenges within not only proteomics but also genomics and transcriptomics domains. An important step is required when utilizing machine learning algorithms with biological sequence data that involves converting a sequence into a numeric input vector. Literature has documented numerous feature extraction techniques that can capture numerical insights from biological sequences (Bonidia et al., 2022; Liu et al., 2015; Liu et al., 2019; Chen et al., 2020; Bonidia et al., 2021). Additionally, some of them are freely available as packages such as tools and web servers, for a detailed list, see (Bonidia et al., 2022). A simple and common feature extraction methodology is the k-mer representation (overlapping sub-sequences of length k) paired with a vectorisation technique. When applied to protein sequences, instead of treating entire amino acid sequences as single entities, they can be deconstructed into k-mers to serve the purpose of capturing localized patterns and associations within sequences.

Using k-mers beyond machine learning is considered one of the oldest and most popular techniques for various alignment-free sequence analyses (Lander et al., 2001; Gribskov et al., 1987; Déraspe et al., 2022). The list of possible applications of k-mers in DNA, RNA and protein sequences is endless as they have been widely used in many bioinformatics applications such as genome assembly (Antipov et al., 2020; Luo et al., 2012; Pevzner et al., 2001), sequence error correction (Kelley et al., 2010), taxonomic classification (Lu et al., 2022), genomic and metagenomic analysis and classification (Dubinkina et al., 2016; Ondov et al., 2016), quality control of sequencing data (Lo & Chain, 2014), database searches (Harris & Medvedev, 2020), sequence clustering (Steinegger & Söding, 2018), genotyping variants (Iqbal et al., 2012), chromosome sequence classification (Rangavittal et al., 2019), structural variations (Khorsand & Hormozdiari, 2021), identification of biomarkers (Wang et al., 2018), and comparative genomics (Ghiurcuta & Moret, 2014). Vectorization techniques that can be used in conjunction with k-mer representation could include frequency Encoding, TF-IDF (Term Frequency-Inverse Document Frequency), One-Hot Encoding, Word2Vec and Embeddings, k-mer Profiles, and N-grams & Skip-grams. K-mer frequency encoding, or simply k-mer counting (Technically, it is often referred to as ‘Count Vectorization’) has dominated other techniques, despite its simplicity (Marçais & Kingsford, 2011). It was initially employed in natural language processing (NLP) and text analysis (also known as ‘bag of words’ or ‘BoW’ for brevity). Its main purpose is to convert text into numerical feature vectors by counting word frequencies, creating a numerical representation. In simpler terms, it transforms text into a matrix of word counts, where rows represent documents, columns are distinct words, and cell values show word frequencies in specific documents. In the context of protein sequences, a similar matrix is created, with rows for different sequences, columns for various k-mers, and values representing k-mer frequencies within each sequence. This resulting matrix is called the feature matrix, summarizing k-mer frequency information across all sequences.

In this article, we present a comprehensive study that employs k-mers along with three main machine learning algorithms to predict a protein domain’s architecture—out of the largest official 40 CATH Architectures. We provide two separate approaches: (1) a model that can predict the architecture from the sequence of amino acids alone, suitable for domains with unknown structures, and (2) a model that employs both the sequence of amino acids and structural features as a sequence of structural alphabets; such a model can be used for domains with known structures. We have conducted a thorough investigation into the impact of the value of k (k-mer) and the role of each sequence, i.e. amino acids and structural alphabets. Support Vector Machines (SVM) stand out as the best algorithm, with an F1 Score reaching as high as 0.94 to predict a protein domain’s architecture from the sequence of amino acids and structural alphabets.

SCOP & CATH: classifications and predictions

Categorizing proteins based on structural features was first introduced by Levitt and Chothia in 1976 (Levitt & Chothia, 1976) when they classified 31 globular proteins into one of four structural classes based on the percentage of secondary structure content and their order: (1) all-alpha proteins; (2) all-beta proteins; (3) alpha + beta; and (4) alpha/beta. The distinction between the last two lies in the fact that in alpha + beta proteins, beta strands and alpha helices are typically separated, while in alpha/beta proteins, alpha helices and beta strands are more mixed. Two decades later, Chothia et al. created a manually curated online database known as the Structural Classification Of Proteins (SCOP) (Murzin et al., 1995). This database classifies protein domains with known three-dimensional structures not only based on their structural features but also on their evolutionary relationships. The classification is hierarchical and composed of 4 levels, with the original structural classes serving as the top level (Andreeva et al., 2014). The hierarchy, from top to bottom, is as follows: Structural Classes, folds, superfamilies, and families (Andreeva et al., 2020). In summary, the top two levels (Classes and Folds) categorize protein domains based on structural features, whereas the third and fourth levels (Superfamilies and Families respectively) rely on evolutionary relationships to group them—See Figure 1. As of 16 June 2022, – the latest release – SCOP encompasses 5 structural classes, 1,562 folds, 2,816 superfamilies, and 5,936 families spanning over a range of 72,544 non-redundant domains corresponding to 115,656 unique protein structures (861,631 chains) from the Protein Data Bank (PDB) (Berman, 2000; Burley et al., 2021).

Nearly two years after the initial release of SCOP in 1995, an alternative semi-automatic hierarchical classification called CATH—named after the first four levels of its hierarchy: Class, Architecture, Topology and Homology (homologous superfamily)—was established (Orengo et al., 1997). The first level – Class—is quite similar to SCOP’s. Architecture, however, classifies protein domains based on the overall shape defined by the secondary structure units. Topology relies on additional structural features, namely, connectivity between secondary structures. In general, the definition of Folds in SCOP is quite similar to CATH’s Topology which makes Architecture a distinct classification that lies between Structural classes and Folds/Topologies. The last level, the homologous superfamily (H), as its name suggests, is based on evolutionary relationships. Figure 1 depicts a comparison between CATH and SCOP.

There have been two levels of CATH: CATH-Plus (4.3 is the latest version) which is considered the ‘stable’ release and CATH-B as a daily snapshot of the latest structures. While CATH-B offers updates at a comparatively faster pace than CATH-Plus and currently encompasses an additional 37K PDB structures, it provides a limited amount of information. CATH-Plus 4.3.0—based on PDB as of July 2019 – covers 500,238 domains spanning over 150,885 unique PDB structures categorised into 5 classes, 43 architecture, 1,472 Topologies and 6,631 superfamilies. Such numbers haven’t been updated since then (Sillitoe et al., 2021). CATH team, in the past few years, has gone beyond the PDB repository to assign domains and provide superfamily annotations for protein sequences with unknown structures in databases such as UniProt (Bateman et al., 2023) and Ensembl (Martin et al., 2023). To this end, CATH team uses mainly Hidden Markov Models (HMM) to identify domains and assign them to superfamilies using a set of ‘representative’ domain structures. They have annotated more than 82 million protein sequences (151 million protein domains) via CATH’s structure-based domain superfamily assignments (Lewis et al., 2018). The main tools they rely on for annotations are the 25+ years old in-house SSAP (Orengo & Taylor, 1996) and CATHEDRAL (Redfern et al., 2007) as structure-based, and HMMER3 (Mistry et al., 2013), PRC (Madera & Bateman, 2008), HHsearch (Steinegger et al., 2019) and MMseqs2 (Steinegger & Söding, 2017) as sequence-based. Except for the first level, i.e. Structural class, all levels comprise unbalanced classification, this is true for superfamilies, where the largest 100 superfamilies represent more than 54% of the total number of the ∼150 million annotated protein sequences; amongst those 100 superfamilies, 11 are called the ‘mega’ ones.

The Architecture level in CATH is crucial, as it was the initial step towards organizing the known folds into groups that unveil a shared shape formed by the overall arrangement of secondary structure units, without considering the connections that link those units. Such a grouping was initially executed manually as a subjective grouping of folds sharing the same shape, however, regardless of the length or even the number of the secondary structure units. For instance, the aba sandwich architecture, consisting of three layers, encompasses numerous topologies (folds) that exhibit diverse sizes and proportions of alpha helices and beta sheets. While this falls into a relatively broad category, domains within this architectural pattern can be succinctly and effectively characterized: they possess a central beta-sheet flanked by layers of alpha helices on both sides (see Figure 2).

Figure 2. Two structures belong to the same Architecture (3-Layer (aba) Sandwich) with different Topologies (Left-to-right: (1) Rossmann fold, PDBID: 1A2B and (2) Ribosomal Protein L9, PDBID: 2HBA). Although the number and length of the secondary structures is quite different, they both share the same overall shape: beta-sheets surrounded by alpha helices. Images created using Mol* (Sehnal et al., 2021).

Since classifying a protein structure into CATH and/or SCOP requires some manual effort, there have been some efforts to make the process fully ‘automated’ by utilising mainly machine learning algorithms. Predicting the topic level, i.e. the structural class—which will be discussed in the next paragraph, has gained tremendous amount of research due to its relative simplicity over the remaining levels. To the best of our knowledge, there have been no attempts to target the middle levels such as Architecture, Topology and Fold. Definitely, the ultimate goal is to predict the SCOP’s family and/or CATH’s superfamily since based on both hierarchies, once the bottom level has been determined, the remaining levels will be implicit, thus a protein has been fully classified. In the paragraph after the next one, we will discuss the main two remarkable works that targeted SCOP’s family and CATH’s superfamily, called MT-LSTM (Bepler & Berger, 2021) and CATHe (Nallapareddy et al., 2023) respectively. Whereas the former achieved an excellent accuracy (>90%), the latter didn’t exceed 75%. In this work, for the first time, we target the ‘structural’ part of CATH, that is, Architecture (and consequently, the structural Class) and achieve a reliable F1 Score.

Assigning a protein structure to a specific Structural class either in CATH or SCOP was amongst the very early attempts to predict CATH/SCOP’s top annotation level due to the relative simplicity as the number of main classes at that level is very small (4 and 5 respectively) compared to the remaining levels (Chou & Zhang, 1995; Chou, 2005; Zhu et al., 2021). Providing a thorough review of such algorithms is beyond the scope of this article as it has been a heavily studied topic, however, they all took advantage of (1) machine learning-based techniques such as SVM (Zhu et al., 2019; Dehzangi et al., 2014), Artificial Neural Networks (ANN) (Jahandideh et al., 2007; Wenzheng et al., 2014), and Ensembles & Meta-Classifiers (Dehzangi et al., 2013; Hayat et al., 2012; Cai et al., 2006), and/or (2) features that reveal class-related information like physiochemical-based information (Dehzangi et al., 2013; Li et al., 2008), amino acid sequence reverse encoding (Mizianty & Kurgan, 2009), pseudo amino acid composition (Liang & Zhang, 2017; Ding et al., 2007), Position Specific Scoring Matrix (PPSM) profile (Hayat & Khan, 2012) and structural based information including secondary structure prediction (Liu & Jia, 2010; Ding et al., 2014). Some of the aforementioned methods did achieve an accuracy of up to 90% (Abbass & Nebel, 2015); a couple of years ago, using symmetrical recurrence quantification analysis to extract features, reverse encoding and random forest classifier, a team reported 100% accuracy (Abdennaji et al., 2021).

Earlier in 2023, the CATH team developed a sequence-based neural network model to predict a protein’s superfamily called CATHe where ‘e’ stands for embeddings; the term used to denote the numerical representations of protein sequences obtained from protein language models (pLMs) (Nallapareddy et al., 2023). It was an attempt to cope with the overhead of the manual curation used—along the automated workflow—to classify the large amount of protein sequences being poured into the databases and to be able to detect some ‘hard’ remote homologues missed by the HMM-based approaches. They tried 6 different models, the best one was the artificial neural networks (ANN) coupled with ProtT5 (Elnaggar et al., 2022) as a source for embeddings. They reported an F1 Score of ∼0.72% on a dataset that contains the largest 1,773 superfamilies. Another remarkable work that leverages protein language models called MT-LSTM, although targeting SCOP families, is considered close to ours in the sense that researchers used information from both sequence and structure (Bepler & Berger, 2021). Through supervised learning and multi-task (MT) language model – masked language modelling task, residue-residue contact prediction, and structural similarity prediction—extracted structural information played a crucial role in producing a more accurate model to predict function. It is worth noting that the structural information that aided the overall learning process was extracted from a very tiny amount of the overall size of the dataset (28,000 vs 76,000,000). They reported an overall accuracy of 96.19%.

Structural alphabets and protein blocks

Although its foundations were laid out much earlier (Unger et al., 1989; Camproux et al., 1999; De Brevern et al., 2000; Schuchhardt et al., 1996; Fetrow et al., 1997), over the past two decades, the notion of a structural alphabet (SA) has attracted much attention due to its ability to encode protein structures/fragments into a sequence of structural letters (SL) – similar to the amino acid sequences. SA encoding plays a key role in converting the three-dimensional (3D) protein conformations into a one-dimensional (1D) representation, thereby allowing for a simplified protein structure analysis. This approach has dramatically simplified many protein bioinformatics applications such as the comparison of 3D conformations by using well-known sequence comparison algorithms on SA sequences (Tung et al., 2007; Guyon et al., 2004).

There have been many successful attempts to create structural alphabets such as Units of Structural Alphabet (USA) (Tung & Nacher, 2013), HMM-SA (Camproux et al., 2004; Camproux & Tufféry, 2005), M32K25 (Pandini et al., 2010), SAFlex (Allam et al., 2018), Protein Blocks (PB) (De Brevern et al., 2000) and others (Ku & Hu, 2008; Sander et al., 2006). Protein Blocks (PBs) are widely recognized as the primary structural alphabet due to their extensive adoption beyond their originators’ laboratories. Since its launch more than two decades ago (De Brevern et al., 2000), PBs have been applied to numerous protein bioinformatics applications and topics such as protein superimpositions (Joseph et al., 2012; Faure et al., 2019; Gelly et al., 2011; Léonard et al., 2014), protein backbone prediction (Vetrivel et al., 2017), molecular dynamics analyses (de Brevern, 2005), protein contacts analysis (Faure et al., 2008), globular protein structures reconstruction (Dong et al., 2007), peptides design (Thomas et al., 2006), locating binding sites (Dudev & Lim, 2007), reduced amino acid alphabet (Etchebest et al., 2007; Zuo and Li, 2009), protein flexibility prediction (Narwani et al., 2019), local protein conformation predictions (Li et al., 2009; Suresh et al., 2013; Suresh & Parthasarathy, 2014; Zimmermann & Hansmann, 2008), Beta turns predictions (Nguyen et al., 2014), and protein structure prediction (Ghouzam et al., 2015; Ghouzam et al., 2016).

PBs comprise 16 distinct local prototypes, these sequence-independent fragments span five residues each and offer a reasonably accurate representation (RMSD < 0.42 Å) of various local 3D protein structures. Consequently, a PB is associated with every pentapeptide within the protein chain. These PBs were created through unsupervised training on a comprehensive, non-redundant protein structure database, using a set of dihedral angles that describe the protein backbone. This characteristic renders PBs as intriguing conformational prototypes for local protein structures with high precision. The prototypes are denoted from 'a‘ to 'p'. PBs 'm‘ and 'd‘ can be broadly categorized as prototypes for the alpha helix and central beta strand, respectively. On the other hand, 'a‘ to 'c‘ predominantly represent N-caps of beta strands, while 'e‘ and 'f‘ signify C-caps of beta strands. Specific to coil regions are PBs 'g‘ to 'j', whereas 'k‘ and 'l‘ correspond to N-caps of alpha helices, and 'n‘ to 'p‘ match with C-caps of alpha helices (de Brevern, 2005).

To assign a protein structure to a sequence of PBs, the local structure of each amino acid is compared to the 16 PB definitions. The most similar protein block is then assigned to the relevant residue, ultimately, this assignment results in the PB sequence. The tool responsible for this conversion, i.e. transforming a PDB file into an overlapped sequence of PBs, is known as PBxplore (Barnoud et al., 2017). In brief, PBxplore operates as follows: To assign a Protein Block (PB) to residue 'n', it requires information from five neighbouring residues (specifically, residues 'n − 2', 'n − 1', 'n', 'n + 1', and 'n + 2′). From the backbone conformation of these five residues, it calculates eight dihedral angles (ψ and ϕ), spanning from the ψ angle of residue 'n − 2′ to the ϕ angle of residue 'n + 2'. This set of eight dihedral angles is then compared to the reference angle sets for the 16 different PBs using the Root Mean Square Deviation Angle (RMSDA) measure, which essentially calculates the Euclidean distance between the angles. The PB with the smallest RMSDA is assigned to residue 'n'. If it’s not possible to compute all eight angles for a residue, a placeholder PB 'Z' is assigned. Consequently, the first two N-terminal and the last two C-terminal residues are consistently assigned to PB 'Z'.

Methodology and experiments

To the best of our knowledge, ‘CATHe’ is the only work that is considered part of the state of the art compared to ours (Nallapareddy et al., 2023). However, we couldn’t use the CATHe dataset or its results for comparison. While the testing and validation sets of CATHe’s dataset consist only of sequences from the PDB, the training set (which represents ∼98% of the overall dataset) contains sequences from CATH-Gene3D (Lewis et al., 2018). The training set includes all the UniProt sequences assignable to CATH superfamilies via HMM algorithm (Sillitoe et al., 2021) representing domains/proteins of unknown structures. Since the primary objective of this study is to incorporate both the sequence of amino acids and structural alphabets, predicting the domain’s architecture that includes domains of known structures, this dataset cannot be used. Regarding the results, CATHe predicts superfamilies—the first level of the CATH hierarchy—whilst our implemented model predicts the architecture, i.e. the third level.

Utilizing the most recent stable CATH version, namely CATH-Plus 4.3, we retrieved the domains classified under any of the 43 total Architectures. Since the majority of protein structures encompass identical chains, and to avoid redundancy in our dataset and consequently sequences with high similarity rates, only Chain A within each protein was taken into consideration. However, 3 architectures consist of a very limited number of domains (<10), namely, ‘1.40 Alpha Solenoid’, ‘2.105 3 Propeller’, and ‘2.180 Shell’. Since such low numbers of entries for those three classes would make the learning process not possible given the fact that the dataset will be divided into training, evaluation and testing subsets, we excluded these cases, resulting in a final count of 40 Architectures, encompassing a combined total of 182,695 domains. A further filtration has been applied by considering only domains that are gapless, i.e. they do not contain any missing amino acids, and consequently, missing atomic coordinates and protein blocks; keeping such ‘incomplete’ domains would affect the accuracy of any model. The total number of domains in the final dataset is 169,843 as depicted in Figure 3. Given that certain chains encompass more than one domain, this number of domains corresponds to a smaller quantity of proteins (Figures 4, 5, and 6).

Figure 3. Distribution of 40 CATH Architectures amongst the 169,843 domains – Dataset #1.

Figure 4. Distribution of 12 CATH Architectures amongst the 157,044 domains – Dataset #2.

Figure 5. Distribution of 28 excluded CATH Architectures amongst the 12,799 domains – This represents Dataset #1 subtracted by Dataset #2.

Figure 6. Flowchart of the Process to create both Datasets: 1 & 2 that contain in each row (domain), the Amino Acid Sequence, Protein Block Sequence and the Architecture ID.

To predict a protein/domain’s architecture, we have conducted three experiments; experiment #1—using the sequence of amino acids of a domain, experiment #2 – using the sequence of protein blocks of a domain, and experiment #3—using both a domain’s sequence of amino acids and sequence of protein blocks. The boundaries of domains are the ones determined by CATH. For each experiment, the main three classification machine algorithms have been employed, namely, Support Vector Machines (SVM), Logistic Regression and Random Forests. We have kept a stratified 20% as a holdout set to serve as an unseen testing dataset for final testing purposes. The results shown in this section correspond to a stratified 5-fold cross-validation methodology on the remaining 80% dataset. K-mers representation along with the Count Vectorisation technique has been adopted to convert each domain’s sequence of amino acids (domain’s length) and sequence of protein blocks (domain’s length-4). Additionally, we have considered different values of k—for k-mer—for each algorithm and each experiment—details on the range of values of k are explained in the next section. The Scikit-Learn library was used along with the default parameters of each algorithm; the source code, as well as the datasets, are all available freely online at https://zenodo.org/records/10203431

Results and discussion

Tables 1 and 2, along with Figure 7, present the F1 scores of three experiments (AA-based, PB-based, and Both) for Datasets #1 and #2, respectively. The choice of the value K to conclude an experiment was based on observed patterns. In Experiment #1, utilizing three algorithms, we noted a peak/plateau around k = 5, followed by a decline in F1 scores, so we concluded the experiment at k = 16. However, for Experiments #2 and #3, across all three algorithms, we extended the experiments to k = 32 to make the peak/plateau and subsequent decline more evident. The highest values are highlighted in bold and grey in Tables 1 and 2. Nevertheless, the consistent pattern observed for all algorithms is that the value of the peak K in AA-based experiments is consistently smaller than that in PB-based experiments. These results may align with the rationale behind selecting K values; since protein block sequences represent structural information, higher values of K are likely expected to effectively capture structural motifs, whereas, for amino acid sequences, shorter values of K are more likely to capture sequence motifs.

Figure 7. F1 Score of three algorithms at three experiments for both datasets. The first column is for Dataset #2 (12 classes) and the second column is for Dataset #1 (40 classes).

Table 1. Results (F1 Score) of Experiments 1 (using Sequence of Amino Acids), Experiments 2 (using Sequence of Protein Blocks) and Experiments 3 (Using Both Features) for the main Classes (Dataset 2 – 12 Architectures).

12 Classes – Experiments 1, 2 and 3
	F1 Score
	SVM			Logistic Regression			Random Forests
	AA- Based	PB- Based	Both	AA- Based	PB- Based	Both	AA- Based	PB- Based	Both
K = 2	0.54	0.43	0.61	0.57	0.48	0.66	0.91	0.88	0.92
K = 3	0.89	0.64	0.91	0.88	0.57	0.75	0.91	0.88	0.92
K = 4	0.94	0.78	0.95	0.93	0.62	0.77	0.91	0.88	0.90
K = 5	0.94	0.84	0.95	0.93	0.68	0.79	0.91	0.89	0.90
K = 6	0.94	0.87	0.95	0.92	0.72	0.82	0.91	0.90	0.90
K = 7	0.93	0.89	0.95	0.92	0.74	0.83	0.90	0.91	0.91
K = 8	0.93	0.91	0.95	0.92	0.75	0.84	0.90	0.91	0.91
K = 9	0.93	0.92	0.96	0.91	0.76	0.84	0.90	0.91	0.91
K = 10	0.92	0.92	0.96	0.91	0.77	0.85	0.90	0.91	0.91
K = 11	0.92	0.94	0.96	0.91	0.78	0.85	0.90	0.91	0.91
K = 12	0.92	0.94	0.96	0.91	0.78	0.86	0.90	0.91	0.91
K = 13	0.92	0.94	0.96	0.91	0.78	0.86	0.89	0.91	0.91
K = 14	0.91	0.94	0.96	0.91	0.79	0.87	0.89	0.91	0.91
K = 15	0.91	0.94	0.96	0.90	0.80	0.87	0.89	0.91	0.91
K = 16	0.91	0.94	0.95	0.90	0.80	0.88	0.89	0.90	0.91
K = 17		0.94	0.95		0.80	0.88		0.90	0.91
K = 18		0.92	0.95		0.81	0.89		0.89	0.91
K = 19		0.92	0.95		0.81	0.90		0.89	0.91
K = 20		0.92	0.95		0.81	0.90		0.89	0.91
K = 21		0.91	0.95		0.82	0.90		0.88	0.91
K = 22		0.91	0.94		0.81	0.91		0.88	0.90
K = 23		0.91	0.94		0.81	0.91		0.88	0.90
K = 24		0.90	0.94		0.81	0.91		0.87	0.90
K = 25		0.91	0.94		0.80	0.91		0.87	0.89
K = 26		0.90	0.93		0.79	0.91		0.86	0.89
K = 27		0.90	0.93		0.77	0.90		0.86	0.89
K = 28		0.89	0.93		0.77	0.90		0.85	0.88
K = 29		0.88	0.93		0.76	0.90		0.85	0.88
K = 30		0.88	0.92		0.75	0.90		0.84	0.88
K = 31		0.88	0.92		0.74	0.89		0.83	0.88
K = 32		0.87	0.92		0.72	0.89		0.82	0.87
AVG	0.90	0.88	0.93	0.89	0.75	0.86	0.90	0.88	0.90
MAX	0.94	0.94	0.96	0.93	0.82	0.91	0.91	0.91	0.92

Table 2. Results (F1 Score) of Experiments 1 (using Sequence of Amino Acids), Experiments 2 (using Sequence of Protein Blocks) and Experiments 3 (Using Both Features) for All Classes (Dataset 1–40 Architectures).

40 Classes – Experiments 1, 2 and 3
	F1 Score
	SVM			Logistic Regression			Random Forests
	AA- Based	PB- Based	Both	AA- Based	PB- Based	Both	AA- Based	PB- Based	Both
K = 2	0.71	0.51	0.75	0.67	0.23	0.40	0.89	0.83	0.89
K = 3	0.83	0.69	0.86	0.87	0.28	0.44	0.90	0.82	0.87
K = 4	0.92	0.77	0.93	0.90	0.33	0.47	0.90	0.83	0.85
K = 5	0.92	0.79	0.93	0.91	0.38	0.51	0.90	0.84	0.86
K = 6	0.92	0.81	0.93	0.91	0.43	0.55	0.90	0.85	0.86
K = 7	0.91	0.83	0.93	0.91	0.46	0.57	0.90	0.86	0.86
K = 8	0.91	0.86	0.93	0.91	0.47	0.59	0.89	0.86	0.87
K = 9	0.91	0.87	0.94	0.90	0.49	0.60	0.89	0.86	0.86
K = 10	0.90	0.88	0.94	0.90	0.51	0.61	0.89	0.86	0.87
K = 11	0.90	0.89	0.94	0.90	0.52	0.63	0.89	0.86	0.87
K = 12	0.90	0.90	0.94	0.90	0.53	0.66	0.89	0.86	0.87
K = 13	0.90	0.90	0.94	0.90	0.55	0.68	0.88	0.86	0.87
K = 14	0.90	0.90	0.94	0.90	0.58	0.70	0.88	0.86	0.87
K = 15	0.90	0.90	0.94	0.89	0.60	0.72	0.88	0.86	0.87
K = 16	0.89	0.90	0.93	0.89	0.62	0.75	0.88	0.86	0.87
K = 17		0.90	0.93		0.63	0.77		0.85	0.87
K = 18		0.90	0.93		0.65	0.79		0.85	0.87
K = 19		0.90	0.93		0.66	0.80		0.85	0.87
K = 20		0.90	0.93		0.67	0.81		0.85	0.87
K = 21		0.89	0.93		0.67	0.81		0.85	0.87
K = 22		0.89	0.92		0.66	0.82		0.84	0.88
K = 23		0.89	0.92		0.65	0.82		0.84	0.87
K = 24		0.88	0.92		0.65	0.82		0.82	0.87
K = 25		0.88	0.92		0.64	0.82		0.83	0.86
K = 26		0.87	0.92		0.63	0.82		0.82	0.87
K = 27		0.87	0.92		0.62	0.81		0.81	0.86
K = 28		0.86	0.91		0.61	0.81		0.81	0.85
K = 29		0.86	0.91		0.60	0.80		0.80	0.84
K = 30		0.85	0.91		0.59	0.79		0.79	0.84
K = 31		0.84	0.91		0.58	0.79		0.78	0.83
K = 32		0.84	0.91		0.57	0.78		0.77	0.83
AVG	0.89	0.85	0.92	0.88	0.55	0.70	0.89	0.84	0.86
MAX	0.92	0.90	0.94	0.91	0.67	0.82	0.90	0.86	0.89

SVM stands out as the top performer with F1 scores of 0.96 and 0.94 for datasets 2 and 1, respectively, when considering sequences of amino acids and protein blocks as features. Conversely, Logistic Regression performs the worst, particularly for the large dataset with 40 classes, showing a significant decline compared to the results of Dataset #1 as expected due to the algorithm’s sensitivity to imbalanced datasets. The asymptotic behaviour of SVM and logistic regression is almost the same across both datasets, while Random Forest appears to be independent of the value K. Additionally, except for Logistic Regression, the scores in experiment 3 are unsurprisingly always better than those in experiments 1 and 2. This is because combining insights from both sequences and structures, in theory, would improve accuracy compared to relying solely on one resource (sequence or structure). As mentioned earlier, these results align with the remarkable work by Bepler and Berger (Bepler & Berger, 2021), where a small amount of structural information contributed to enhancing the learning process for detecting the SCOP family.

As mentioned earlier, Table 2 exhibits a similar 'behaviour’ for dataset #1 (40 classes), but unsurprisingly shows a decrease in F1 Score, as seen in Table 3. Considering that dataset #1 contains 28 additional classes of varying sizes, including some very small ones, the results show only a relatively slight loss. For instance, in experiment #1 across all algorithms, there is a minimal decrease in F1 score. Once again, SVM outperforms other algorithms, with minimal differences in scores between the two datasets.

Table 3. Comparison between averages of Table 1 with Table 2.

		SVM			Logistic Regression			Random Forests
		AA-Based	PB-Based	Both	AA-Based	PB-Based	Both	AA-Based	PB-Based	Both
Average of F1 Score	12 Classes	0.90	0.88	0.93	0.89	0.75	0.86	0.90	0.88	0.90
	40 Classes	0.89	0.85	0.92	0.88	0.55	0.70	0.89	0.84	0.86
	Difference	0.01	0.03	0.01	0.01	0.20	0.16	0.01	0.04	0.04
Highest F1 Score	12 Classes	0.94	0.94	0.96	0.93	0.82	0.91	0.91	0.91	0.92
	40 Classes	0.92	0.90	0.94	0.91	0.67	0.82	0.90	0.86	0.89
	Difference	0.02	0.04	0.02	0.02	0.15	0.09	0.01	0.05	0.03

Additional experiments #3

Further to the aforementioned set of experiments 3 where both the sequence of amino acids and protein blocks were used as features, we have added a few additional ones that took advantage of Tables 1 and 2 to combine ‘successful’ k values for each experiment. To this end, we have conducted 6 additional experiments shown in Table 4. Different from the ones shown in Tables 2 and 3 where the same value of k is used for each feature, we have chosen—for each algorithm—the best value of k for AA-based and the best value of k for PB-based (using the third digit after the decimal point). Again, SVM shows results that one can consider in line with the rationale behind these additional experiments. We have now two new records, for, the highest F1 scores are now 0.97 and 0.95 for Dataset #2 and Dataset #1 respectively, compared to 0.96 and 0.94 attained by ‘standard’ experiment #3.

Table 4. Results (F1 Score) of Extended Experiments 3 (using both features, Sequence of Amino Acids and Sequence of Protein Blocks) for both the main and 40 Classes (Datasets 1 & 2).

12 Classes – Extended Experiments 3
	F1 Score
	SVM	Logistic Regression	Random Forests
k= 5 (AA) K = 10 (PB)			0.92
k= 5 (AA) K = 13 (PB)	0.97
k= 5 (AA) K = 21 (PB)		0.91
40 Classes – Extended Experiments 3
	SVM	Logistic Regression	Random Forests
k= 5 (AA) K = 11 (PB)			0.87
k= 5 (AA) K = 13 (PB)	0.95
k= 5 (AA) K = 25 (PB)		0.83

Detailed F1 scores for each class for both SVM 'final’ experiments, i.e. 12 classes – k = 5 (AA) K = 13 (PB) – and 40 classes – k = 5 (AA) K = 13 (PB), are shown in Tables 5 and 6, respectively. While in Table 5, one can notice that overall, the F1 score of all classes is almost the same, some classes in Table 6 show relatively low scores, namely, 6.10—Helix non-globular, 3.15—Super Roll, and 6.20 Other non-globular (we have highlighted in red all F1 scores that are less than 0.90). For 6.10 and 6.20, one can understand why such low scores may occur; as their architectural code reveals, they belong to the structural Class 6 dedicated to non-globular proteins—called 'Special’ (the main structural classes are 1- Mainly Alpha, 2- Mainly Beta, 3- Alpha Beta, 4- Few Secondary Structures). Thus, Structural Class 6 is considered an irregular group as it contains structures that do not have any regular patterns of secondary structures, as seen in Figure 8. As for 3.15—Super Roll, while it belongs to a 'standard’ Structural Class and Architecture, the size of the class (32) might have played a role in preventing the ML algorithm from accurately computing its complex architecture.

Figure 8. From Left to Right, structures of three proteins that belong to architectures 6.10 and 6.20: 2LH0 (6.10), 4GIP (6.10), 2CON (6.20). Images created using Mol* (Sehnal et al., 2021).

Table 5. Detailed F1 Score for SVM – 12 Classes – k = 5 (AA) K = 13 (PB).

Architecture	Size	F1	Recall	Precision
1.1	29,458	0.97	0.99	0.96
1.2	9,880	0.96	0.95	0.97
1.25	1,828	0.95	0.91	1
2.3	3,684	0.95	0.91	0.99
2.4	12,283	0.98	0.97	0.99
2.6	18,101	0.98	0.99	0.98
3.1	7,182	0.96	0.94	0.99
3.2	7,625	0.98	0.97	1
3.3	29,638	0.97	0.98	0.96
3.4	38,245	0.98	0.99	0.98
3.5	1,870	0.98	0.96	1
3.9	10,871	0.97	0.95	1
Average	14,222	0.97	0.96	0.99
Accuracy	0.97

Table 6. Detailed F1 Score for SVM – 12 Classes – k = 5 (AA) K = 13 (PB).

Architecture	Size	F1	Recall	Precision
1.1	29,458	0.97	0.99	0.95
1.2	9,880	0.95	0.95	0.95
1.25	1,828	0.95	0.92	0.99
1.5	498	0.97	0.94	1
2.1	1,597	0.96	0.93	0.99
2.101	130	0.95	0.91	1
2.102	117	0.97	0.95	1
2.11	31	0.93	0.87	1
2.115	205	0.98	0.96	0.99
2.12	571	0.98	0.96	1
2.13	667	0.98	0.97	1
2.14	176	0.99	0.98	1
2.15	30	0.95	0.93	0.97
2.16	510	0.97	0.94	1
2.17	941	0.97	0.95	1
2.2	646	0.89	0.82	0.97
2.3	3,684	0.95	0.92	0.98
2.4	12,283	0.98	0.97	0.99
2.5	39	0.87	0.77	1
2.6	18,101	0.98	0.99	0.97
2.7	1,522	0.97	0.95	1
2.8	704	0.98	0.96	1
2.9	53	0.99	0.98	1
3.1	7,182	0.96	0.94	0.99
3.101	19	0.97	0.95	1
3.15	32	0.81	0.69	1
3.2	7,625	0.98	0.97	1
3.3	29,638	0.96	0.97	0.95
3.4	38,245	0.98	0.99	0.97
3.5	1,870	0.99	0.97	1
3.55	149	0.92	0.86	1
3.6	1,806	0.97	0.95	1
3.65	204	0.99	0.99	1
3.7	106	0.97	0.95	0.99
3.75	93	0.94	0.88	1
3.8	358	0.97	0.94	1
3.9	10,871	0.97	0.95	1
4.1	1,264	0.92	0.86	0.99
6.1	1,012	0.79	0.67	0.98
6.2	197	0.83	0.75	0.94
Average	4,608	0.95	0.92	0.99
Accuracy	0.97

For the sake of further investigation regarding the performance of the experiments on the 40 classes, we have chosen the SVM experiments – Table 2—to plot the F1 Scores of the 80 experiments (16(AA) +32(PB) +32(AAPB)) since SVM algorithms performed the best, see Figure 9. The results are in line with the ones shown in Table 5.

Figure 9. F1 Scores of 40-Classes SVM 80 experiments for each Architecture.

Final experiments – Holdout set results

Although the preceding results are derived from stratified 5-fold cross-validation, 20% of the dataset has been reserved as a holdout set, also referred to as an unseen dataset, for conclusive testing. To save time and for brevity, we selected the experiment within each set that achieved the highest F1 Score for final testing. Adhering to standard practice, the model underwent training on the entire 80% training dataset and was subsequently tested on the remaining 20%. Table 7 below presents the outcomes of these experiments, corresponding to Table 4 and a subset of Tables 1 and 2. The numbers in black (11 out of 24) mirror the cross-validation results. The numbers in green (11 out of 24) and red (2 out of 24) are marginally higher (+0.01) and slightly lower (–0.01), respectively, with only two results exhibiting a greater deviation of 0.02 (highlighted in green and underlined). Given the negligible nature of these differences, one can infer that the results obtained through cross-validation align closely with those of the holdout set. The slightly higher F1 Scores may be attributed to the larger training dataset, providing a reasonable explanation for the observed variations.

Table 7. Holdout dataset’s results.

12 Classes – Experiments 1, 2 and 3
	F1 Score
	SVM			Logistic Regression			Random Forests
	AA-Based	PB-Based	Both	AA-Based	PB-Based	Both	AA-Based	PB-Based	Both
K = 2									0.93
K = 5	0.95			0.94			0.93
K = 10								0.91
K = 13		0.94	0.97
K = 21					0.83
K = 25						0.92
40 Classes – Experiments 1, 2 and 3
	F1 Score
	SVM			Logistic Regression			Random Forests
	AA-Based	PB-Based	Both	AA-Based	PB-Based	Both	AA-Based	PB-Based	Both
K = 2									0.89
K = 5	0.92			0.91			0.91
K = 10								0.85
K = 13		0.90	0.94
K = 21					0.69
K = 25						0.83
12 Classes – Extended Experiments 3
	F1 Score
	SVM	Logistic Regression	Random Forests
k= 5 (AA) K = 10 (PB)			0.92
k= 5 (AA) K = 13 (PB)	0.97
k= 5 (AA) K = 21 (PB)		0.91
40 Classes – Extended Experiments 3
	SVM	Logistic Regression	Random Forests
k= 5 (AA) K = 11 (PB)			0.88
k= 5 (AA) K = 13 (PB)	0.94
k= 5 (AA) K = 25 (PB)		0.83

As a result of this extensive study, we suggest the following 4 algorithms to predict a protein/domain’s architecture out of the largest 12 and 40 CATH architectures; all of them are based on Support Vector Machines, as shown in Table 8. These four algorithms’ complete source codes and datasets are all freely available online at https://zenodo.org/records/10203431.

Table 8. Final conclusion that includes the best 4 algorithms out of the study.

	Proteins/Domains of Known Structures		Proteins/Domains of Unknown Structures
	12 Architectures	40 Architectures	12 Architectures	40 Architectures
Solution	Algorithm: SVM Features: Sequences of AA and PB (K = 13)		Algorithm: SVM Features: Sequences of AA (K = 5)
Expected F1 Score	0.97	0.94	0.95	0.92

Given the reliable F1 Scores of the 4 algorithms shown above, one can claim that half of the CATH annotation (Architecture and Class levels out of 4) which represents the ‘structural’ part of the CATH hierarchy can now be accurately annotated mainly for proteins with known structures. This includes not only the experimentally determined structures that still lack CATH annotation but also the PDB’s ∼1 M CSM predicted by AlphaFold.

Conclusion

In conclusion, our study addresses the critical challenge of annotating a rapidly expanding database of protein structures, especially with the inclusion of AlphaFold predictions in the PDB repository. While CATH has traditionally been a valuable resource for classifying proteins based on structural and evolutionary relationships, the manual intervention required for annotation struggles to keep pace with the growing number of structures.

To overcome these challenges, we proposed a novel machine-learning model capable of classifying protein domains into the 40 main CATH Architectures. Through a comprehensive investigation, we achieved promising results with an F1 Score of 0.92 using only amino acid sequences and a score of 0.94 when incorporating both amino acid sequences and structural alphabets. The utilization of k-mers and three main machine-learning algorithms, with Support Vector Machines (SVM) standing out as the best performer, showcased the effectiveness of our approach.

Our findings contribute to the field by providing a fully automated method for annotating protein domains, offering an efficient solution to the challenges posed by the increasing volume of experimental and predicted structures. The developed machine-learning models, particularly the SVM-based algorithms, present a reliable means to predict the architecture of protein domains, bridging the gap between sequence and structure information.

Moreover, our study extends the applicability of machine learning beyond traditional sequence-based methods, demonstrating the efficacy of combining insights from both sequences and structures. This approach aligns with recent advancements in the field, emphasizing the importance of integrating multiple resources for improved accuracy.

The availability of our code and datasets facilitates further exploration and utilization of our models within the scientific community. We believe that our work marks a significant step toward the automation of protein domain annotation and classification, addressing the evolving landscape of structural biology and protein research.

Abbreviations
AA	=	amino acid
CATH	=	Class, Architecture, Topology, Homology
CSM	=	Computed Structure Models
DL	=	deep learning
HMM	=	Hidden Markov Models
ML	=	machine learning
PB	=	protein block
PDB	=	Protein Data Bank
pLDDT	=	predicted local distance difference test
pLMs	=	protein language models
SA	=	structural alphabet
SCOP	=	Structural Classification of Proteins
SVM	=	Support Vector Machines

Disclosure statement

No potential competing interest was reported by the author(s)

Data availability statement

The code and datasets developed in this study can be accessed at https://zenodo.org/records/10203431.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.