---
title: "PLUG: A Leakage-Free, Multifunctional Benchmark for Protein Function Prediction"
authors:
  - "A. Clay Richard"
  - "Brian Cheung"
  - "Stephanie A Wankowicz"
doi: "10.82153/daxg-y552"
license: "https://creativecommons.org/licenses/by/4.0/"
date: "2026-11-17"
version: 1
canonical_url: "https://thestacks.org/publications/plug-benchmark"
---

# PLUG: A Leakage-Free, Multifunctional Benchmark for Protein Function Prediction

_A modular, leakage-free benchmark that evaluates protein function prediction across seven functional axes under one consistent homology standard._

## Abstract

Protein function is inherently multidimensional. Proteins perform an extraordinary range of biological functions, from catalysis to molecular recognition. These functions depend on local features like active-site chemistry, on long-range effects like allostery, and on interactions with other molecules. If a model has learned protein function in general, its predictions should transfer across functional properties. However, current benchmarks usually score a single or pooled functional property, obscuring which aspects of function models actually capture. Further, these benchmarks employ various data splitting strategies, making it difficult to evaluate across benchmarks without incurring data leakage. Here, we introduce Protein Leakage-free evaluation of Unbiased Generators of function (PLUG), a modular and extensible framework for leakage-free evaluation of prediction models across multiple functional properties. Taking multiple different benchmarking datasets, PLUG constructs test sets that are controlled for homology across datasets. A user can fix the test sets and generate an unlabeled training set or partition the data into homology-aware splits for supervised and transfer learning. PLUG is also designed to easily incorporate new functional datasets, and allow users to design their own splitting strategies across existing and proprietary datasets. We show a substantial performance gap between splits controlled for homology and random splits over the same data in PLUG demonstrating that leakage-free evaluation is essential for measuring whether models capture general principles of protein function. To facilitate community engagement we developed a Hugging Face portal to track evaluations of multifunctional protein prediction models. Predictions can be submitted to a running leaderboard and model performance is evaluated on each dataset separately as well as the benchmark as a whole.

::::::figure{#graphicalabstract align="center" type="image" unnumbered="true"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_fbfa6503_bac75959c4b1"}
:::::

::::::

# Introduction

Predicting protein functions and how they change under perturbation is a central goal of biology. [](https://doi.org/10.1038/nrm2281) [](https://doi.org/10.64898/2026.04.27.716980) [](https://doi.org/10.1093/nar/gkh036) Protein function is inherently multidimensional. A single protein can perform myriad biological functions such as catalyzing reactions, binding ligands, and signaling allosterically. [](https://doi.org/10.1016/s0022-2836(65)80285-6) [](https://doi.org/10.1038/228726a0) [](https://doi.org/10.1038/206757a0) [](https://doi.org/10.1042/ebc20200108) [](https://www.ncbi.nlm.nih.gov/books/NBK26911/) The impact of each function can change with small perturbations, such as a macromolecular interaction, a mutation, or a post-translational modification. Together, this concert of functions is essential for all biological processes. The space of possible sequences for a single protein is vast ($2{0}^{N}$), with perturbations on top of this, making exhaustive experimental characterization impractical. Accurate prediction of protein function would enable interpretation of the millions of unlabeled protein sequences, understanding the impact of mutations, and designing for function.

To date, over 300 million protein sequences have been cataloged; [](https://doi.org/10.1093/nar/gkae1010) [](https://doi.org/10.1093/nar/gkac1080) however, fewer than 1% are functionally characterized. [](https://doi.org/10.1101/2023.03.14.532539) [](https://doi.org/10.1093/database/baac062) One challenge of mapping sequence to function is that protein function spans many levels of granularity. At the broadest level, Gene Ontology (GO) terms classify whole proteins by molecular function, biological process, or subcellular localization and have been instrumental in organizing the deluge of sequencing data by identifying key pathways. [](https://doi.org/10.1093/nar/gkh036) [](https://doi.org/10.1016/j.bpj.2015.04.020) Similarly, Enzyme Commission (EC) numbers assign a sequence indicating its relationship to known enzymatic activities. GO terms and EC numbers are often assigned by guilt-by-association or sequence similarity, an approach now being extended through learned representations rather than alignment alone. [](https://doi.org/10.1101/2024.07.10.602933) These methods are scalable but only coarsely describe function and can be agnostic to finer mechanistic details. Further, sequences with unrelated folds but convergent function remain a persistent blind spot and still require biochemical or cellular characterization for accurate annotation. [](https://doi.org/10.1093/g3journal/jkaf169)

Consider thrombin. At the GO level, it is a serine-type endopeptidase and a key member of the coagulation cascade. A His-Asp-Ser catalytic triad places it in the chymotrypsin-like S1 family, alongside coagulation factor Xa and the digestive enzymes trypsin and chymotrypsin. [](https://doi.org/10.1021/cr000033x) These proteins share the same fold, the same catalytic mechanism, and broadly conserved activity. Yet this shared annotation obscures finer mechanistic detail [](https://doi.org/10.1002/prot.26773) [](https://doi.org/10.7554/elife.74114). A single mutation may leave thrombin's proteolytic activity intact while rewiring allosteric communication, driving aberrant downstream signaling. Other mutations may change the proteolytic specificity of thrombin while having minimal impact on the catalytic activity. [](https://doi.org/10.1038/378413a0) Untangling these effects is required to predict mutational consequences, design better therapeutics, and engineer proteins with novel properties.

::::::figure{#figure1 align="center" type="image" label="Figure 1"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_38455e98_61f21b7642ee"}
:::::

:::::figcaption
**Figure 1.** **Protein function spans many distinct properties, and single-task evaluations capture only part of this space.** Schematic of functional properties relevant to sequence-to-function prediction with benchmarks. Properties include Gene Ontology (GO) term annotations, ligand binding, thermodynamic stability, allostery, and protein-protein interactions.
:::::

::::::

High-throughput experimental methods now enable the collection of data on biophysical or function data at scale. Deep mutational scanning (DMS), binding assays, HT-MEK, and limited proteolysis assays, among others, report quantitative properties across thousands of variants. [](https://doi.org/10.1073/pnas.2219074120) [](https://doi.org/10.1126/science.abf8761) [](https://doi.org/10.1038/s41586-023-06328-6) [](https://doi.org/10.1093/bib/bbae076) However, each assay often measures a different molecular phenotype under a single engineered selection, such as fitness or survivability, stability, binding, or enzymatic turnover. A variant can be neutral in one assay's readout yet functionally disruptive through a different, unprobed mechanism. These assays are beginning to produce the scale of data machine learning needs; however, each captures only a subset of information needed to predict protein function. Other metrics of ‘function’, such as clinical pathogenicity, report on disease association rather than protein function. [](https://doi.org/10.1016/j.ajhg.2022.10.013) [](https://doi.org/10.1038/gim.2015.30)

Machine learning has become the dominant tool for learning sequence-to-function mappings. Its capacity to learn complex, non-linear patterns across vast sequence spaces offers a path toward capturing this complex mapping. [](https://doi.org/10.1126/science.ads0018) [](https://doi.org/10.1038/s41467-025-67971-3) [](https://doi.org/10.1016/j.cell.2026.06.029) However, determining how well these algorithms predict function, and at what granularity, remains essential to building more powerful machine learning predictors and uncovering new biology. Existing benchmarks use varying functional datasets, reading out different types of functions. They also vary in size, sequence diversity, organization, and splitting strategy. [](https://doi.org/10.64898/2026.04.27.716980) [](https://doi.org/10.1101/2023.12.07.570727) [](https://doi.org/10.64898/2026.02.23.707496) As a result, performances across benchmarks are difficult to compare.

Rigorous evaluation of sequence-to-function models should contain three requirements: a clearly specified functional target, control of data leakage, and protection against contamination over time. A clearly specified target is a concrete task: “catalytic activity of a given enzyme” qualifies, while “protein function” does not. [](https://doi.org/10.1186/1471-2105-14-s3-s15) [](https://doi.org/10.1371/journal.pcbi.1002386) Second, the benchmark must control data leakage. Leakage occurs when test information reaches training, inflating apparent generalization. [](https://doi.org/10.1145/2382577.2382579) Leakage can arise from poor splits, from features that encode the target, and from group leakage, where related measurements land on both sides of a split. [](https://doi.org/10.1145/2382577.2382579) [](https://doi.org/10.1038/s41592-024-02362-y) [](https://proceedings.mlr.press/v261/hermann24a.html) Random splits assume independent observations, but proteins are related by evolution. When relatives fall on both sides, a model can recover the label from similarity alone. Shared ligands, assay formats, conditions, and mutation classes create the same shortcut. [](https://doi.org/10.64898/2026.03.08.710389) Avoiding leakage and covering enough of protein space to train on are competing goals, and this tension is the central obstacle to generalizable benchmarks. [](https://doi.org/10.1038/s41592-024-02362-y) Individual datasets have addressed leakage, [](https://doi.org/10.1093/bib/bbae076) [](https://doi.org/10.1038/s41467-025-58606-8) [](https://doi.org/10.1038/s42256-025-01124-5) [](https://doi.org/10.1021/acs.jpcb.5c08598) but no consistent definition has been applied across datasets. Finally, static benchmarks become contaminated as sequences and their homologs enter training corpora, so benchmarks must be updated and resplit continually.

Existing benchmarks meet these requirements only in part. Early functional benchmarks relied on small, idiosyncratic assay collections, [](https://doi.org/10.1038/s41592-018-0138-4) which made head-to-head comparison unreliable. The Critical Assessment of Functional Annotation (CAFA) crowdsources predictions of ontology terms such as GO annotations, but its temporal holdout does not explicitly exclude homology leakage. [](https://doi.org/10.64898/2026.04.27.716980) The two main variant-effect benchmarks each cover one of the two axes a multi-property model needs, breadth across proteins and resolution across properties, but not both. ProteinGym is broad, assembling roughly 2.5 million missense variants across 187 wild-type proteins, but most measurements fall into general “fitness” or “activity” buckets. [](https://doi.org/10.1101/2023.12.07.570727) These composite readouts entangle orthogonal effects, since a loss-of-function variant may reflect a defect in expression, stability, or activity, and a mutation that improves one property can impair another, obscuring how variants actually map to function. FLIP2 resolves specific properties, with separate benchmarks for enzymatic activity, folding stability, and binding affinity, but only goes deep on a few proteins rather than broad across many. [](https://doi.org/10.64898/2026.02.23.707496) While both ProteinGym and FLIP2 split their data to reduce data leakage, neither dataset is broad in protein sequence while also reporting specific functional or biophysical properties. No existing benchmark fixes the test corpus across functional axes while controlling leakage against all of them simultaneously.

To characterize how well models can predict multimodal protein function, we introduce Protein Leakage-free evaluation for Unbiased Generators of function (PLUG). PLUG is a modular and extensible framework for leakage-free evaluation of prediction models across multiple functional measurements. It combines several diverse functional datasets and applies consistent splitting strategies so that performance is comparable across benchmarks. It generates homology-aware training reservoirs for foundation models, and homology-aware splits for supervised fine tuning (SFT) and transfer learning. The splitting strategy and similarity thresholds are tunable to enable future studies characterizing performance as a function of training similarity. Importantly, PLUG can accept new functional datasets and rebuild the corresponding training reservoirs as further experimental data are published, so evaluation adapts as models and datasets change. This modularity also allows users to incorporate in-house datasets, so that proprietary data can be evaluated alongside existing benchmarks with consistent homology control. We demonstrate PLUG's utility by analyzing the impacts of data leakage and characterizing orthogonal functional predictions from ESM3, showing that probe performance decreases by an average of 26% across eleven functional datasets. We propose adopting PLUG as a framework for continual evaluation of protein function predictors, with careful homology detection to assess generalization and performance as a function of training similarity.

# Results

## PLUG integrates diverse functional datasets

We developed PLUG to evaluate machine learning models across functional axes while consistently controlling for data leakage. This allows prediction algorithms to be assessed across distinct functional properties, establishing which functional axes an algorithm excels or fails on. It also enables transfer learning, where a model trained on one function (e.g., stability) can be evaluated on another (e.g., allostery) under the same split criteria. Together, these comparisons allow protein function prediction to progress. Code for PLUG is open source and can be found at <https://github.com/prism-science/plug>.

To develop the first version of PLUG, referred to as the PLUG benchmark in this manuscript, we took eleven datasets representing seven unique functional axes: GO annotations (from CAFA5 via BioReason-Pro), [](https://doi.org/10.64898/2026.04.27.716980) [](https://doi.org/10.64898/2026.03.19.712954) [](https://huggingface.co/datasets/wanglab/bioreason-pro-test-data) fitness labels (based on DMS; from ProteinGym), [](https://doi.org/10.1101/2023.12.07.570727) small molecule binding affinity (from BindingDB and LP-PDBBind), [](https://doi.org/10.1021/acs.jpcb.5c08598) [](https://doi.org/10.1093/nar/gkae1075) allostery (from AlloBench and PASSerRank), [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1002/jcc.27193) [](https://doi.org/10.1093/nar/gkq1022) thermostability (from Megascale and FireProt via ThermoMPNN), [](https://doi.org/10.1038/s41586-023-06328-6) [](https://doi.org/10.1371/journal.pcbi.1004556) [](https://doi.org/10.1073/pnas.2314853121) protein–protein interactions (PPIs; from the gold-standard PPI dataset and STRING’s SHS27K), [](https://doi.org/10.1093/bib/bbae076) and molecular dynamics (MD) root mean squared fluctuations (RMSF) (from ATLAS). [](https://doi.org/10.1038/75556) [](https://doi.org/10.1038/225563a0) [](https://doi.org/10.1021/bi00514a017) [](https://doi.org/10.1038/35015694) [](https://doi.org/10.1038/256705a0) [](https://doi.org/10.1038/277578a0) [Table 1](#table1) provides details on the initial set of included datasets. In total, the PLUG benchmark covers 36,816 unique wild-type proteins and contains over 3.1 million unique measurements. Further details on the datasets can be found in the Supplementary Information.

::::::figure{#table1 type="table" label="Table 1"}
| **Dataset**                                                                                                                                                                | **Function/Property**       | **Number of Unique Wild Type Sequences** | **Number of Measurements** |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -------------------------- |
| ATLAS [](https://doi.org/10.1093/nar/gkad1084)                                                                                                                            | MD RMSF                     | 1,693                                    | 1,938                      |
| AlloBench [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1093/nar/gkq1022)                                                                           | Allostery                   | 425                                      | 2,257                      |
| PASSerRank [](https://doi.org/10.1002/jcc.27193) [](https://doi.org/10.1093/nar/gkq1022)                                                                                 | Allostery                   | 331                                      | 1,885                      |
| STRING SHS27K [](https://doi.org/10.1093/nar/gkae1113) [](https://doi.org/10.1093/bioinformatics/btz328)                                                                 | Protein–Protein Interaction | 1,690                                    | 7,624                      |
| Gold Standard PPI [](https://doi.org/10.1093/bib/bbae076)                                                                                                                 | Protein–Protein Interaction | 11,019                                   | 274,500                    |
| ProteinGym [](https://doi.org/10.1101/2023.12.07.570727)                                                                                                                  | Binding                     | 12                                       | 751,063                    |
| ProteinGym [](https://doi.org/10.1101/2023.12.07.570727)                                                                                                                  | Expression                  | 18                                       | 97,869                     |
| ProteinGym [](https://doi.org/10.1101/2023.12.07.570727)                                                                                                                  | Organismal Fitness          | 65                                       | 1,034,722                  |
| ProteinGym [](https://doi.org/10.1101/2023.12.07.570727)                                                                                                                  | Activity                    | 40                                       | 447,319                    |
| ProteinGym [](https://doi.org/10.1101/2023.12.07.570727)                                                                                                                  | Stability                   | 66                                       | 134,794                    |
| Megascale [](https://doi.org/10.1038/s41586-023-06328-6) [](https://doi.org/10.1073/pnas.2314853121)                                                                     | Stability                   | 298                                      | 271,231                    |
| FireProt [](https://doi.org/10.1371/journal.pcbi.1004556) [](https://doi.org/10.1073/pnas.2314853121)                                                                    | Stability                   | 100                                      | 3,404                      |
| CAFA5 [](https://doi.org/10.64898/2026.04.27.716980) [](https://doi.org/10.64898/2026.03.19.712954) [](https://huggingface.co/datasets/wanglab/bioreason-pro-test-data) | Gene Ontology               | 8,528                                    | 8,630                      |
| BindingDB [](https://doi.org/10.1093/nar/gkae1075)                                                                                                                        | Small Molecule Binding      | 2,156                                    | 76,866                     |
| LP-PDBBind [](https://doi.org/10.1021/acs.jpcb.5c08598)                                                                                                                   | Small Molecule Binding      | 12,718                                   | 19,443                     |

:::::figcaption
**Table 1.** Included Functional Datasets. The dataset is listed along with the function it covers, the number of unique wild-type protein sequences it contains, and the total number of measurements it contains.
:::::

::::::

## Controlling for Leakage Among PLUG Datasets

Entries in biological function datasets often share sequence, structure, ligand composition, and binding partners. These similarities can result in data leakage, inflating confidence in a model’s generalization. As each dataset typically defines their own train/validation/test splits, it is difficult to evaluate SFT or transfer learning across these datasets. PLUG applies the same splitting scheme across all functional datasets. The PLUG framework is amenable to any definition of leakage that can be quantified with a similarity measure. Further, most leakage parameters are user-tunable, which allows construction of a dataset suited to a given application.

The default values in the PLUG benchmark consider proteins to be similar if they have a sequence ID greater than 30%, a TM-score greater than 0.7, and Tanimoto score greater than 0.5 using the Morgan Fingerprint with radius of two ([Figure 2](#figure2)A). [](https://doi.org/10.1038/nbt.3988) [](https://doi.org/10.1093/bioinformatics/btq066) [](https://doi.org/10.1093/nar/gki524) [](https://doi.org/10.1186/s13321-015-0069-3) [](https://doi.org/10.1021/acs.jpcb.4c01875) For sequence similarities, we consider three types of coverage ([Figure 2](#figure2)B). Mode 0 requires the alignment to cover 80% of both the target and query sequences, so it flags only pairs of similar length. Mode 1 requires 80% coverage of the target sequence. This identifies matches where most of the target sequence is similar to a region in the query sequence, capturing the case where a shorter target sequence fits within a longer query sequence. Mode 2 requires 80% coverage of the query sequence. It catches the reverse case: a short query protein that is a subset of a longer target protein. For example, consider a 600-residue query protein that contains a domain identical to a 200-residue target protein. The alignment covers 100% of the target sequence but only 33% of the query sequence. Mode 1 measures coverage of the target sequence, sees 100%, and labels it a match. Mode 2 catches the same pair when the roles are reversed, with the 600-residue protein in the target set.

PLUG combines modes 1 and 2. Two sequences are considered similar if they share more than 30% identity with more than 80% coverage in either direction (COV_MODE = 1 ∪ 2). We implement this bidirectional union protocol, which searches for homologous sequences first from query (e.g., UniRef90 sequences) to target (e.g., PLUG benchmark) and then from target to query using the multiple coverage modes of MMseqs2 and the target and query TM-scores from Foldseek. [](https://doi.org/10.1038/nbt.3988) [](https://doi.org/10.1038/s41587-023-01773-0)

## Use cases for PLUG

PLUG supports three main use cases; foundation model pretraining, SFT, and representation and transfer learning. For foundation models, it generates homology-aware training reservoirs. For SFT and transfer settings, it generates homology-aware splits with tunable similarity thresholds.

_Pretraining._ PLUG filters unlabeled pretraining data to prevent leakage into downstream functional evaluations. The user supplies a reservoir of protein sequences or structures and a target pretraining dataset size. PLUG scans each pretraining candidate against all proteins in the 11 benchmark datasets and rejects candidates with homology above the leakage thresholds. Sampling continues until the pretraining target size is reached. The resulting dataset contains no homologs of any evaluation protein, so a foundation model trained on it can be evaluated on every functional axis without leakage. Comparing models trained with and without this filtering measures how much pretraining overlap inflates downstream performance.

_Supervised fine-tuning._ For SFT on a single functional dataset, PLUG splits that dataset into train, validation, and test sets using default or user-defined thresholds. Splitting biological data into maximally dissimilar groups is NP-hard, and many heuristics exist. [](https://doi.org/10.1093/bib/bbae076) [](https://doi.org/10.1038/s41467-025-58606-8) [](https://doi.org/10.1038/s42256-025-01124-5) [](https://doi.org/10.1021/acs.jpcb.5c08598) For single-dataset splits, PLUG uses DataSAIL, which minimizes the total sequence similarity between proteins assigned to different partitions. [](https://doi.org/10.1038/s41467-025-58606-8) DataSAIL defines leakage as an objective in an integer linear program, penalizing homologous pairs occurring in different splits. [](https://doi.org/10.1145/2382577.2382579) [](https://doi.org/10.1038/s41467-025-58606-8) [](https://doi.org/10.48550/arXiv.2102.01818) However, homologs can still land in different splits if the penalty is too weak. Each entity can be compared under only one similarity definition, unless the user builds a custom score that combines several. And pooling all datasets produces one large connected component that cannot be broken into smaller stratified splits. DataSAIL splits therefore control for leakage only within a single dataset.

_Transfer learning._ PLUG also supports transfer learning. Transfer learning evaluates how well a model trained to predict one function/property can predict a different function/property. Here, homology must be controlled across datasets at once. For example, a model can be fine-tuned to predict residue root mean square fluctuation (RMSF) on ATLAS and then used to predict allosteric effects in AlloBench. [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1093/nar/gkad1084) If homologous proteins appear in both datasets, the model can succeed by recognizing proteins it has already seen, deflating its performance. To solve the issue of transfer between datasets and to generate a stricter leakage guarantee, we use the method described in the Lo-Hi benchmark, constraining leakage while maximizing the retention of usable, leakage-free data in the splits. [](https://doi.org/10.48550/arXiv.2310.06399) [](https://doi.org/10.1109/codit.2014.6996889) By solving the balanced vertex minimum k-cut problem, we provide a strict homology guarantee across splits, but at the cost of removing datapoints. [](https://doi.org/10.1007/s10107-005-0574-7)

To formulate this optimization problem, PLUG builds a graph in which each node is a unique protein or ligand. An edge connects two nodes if they exceed any similarity threshold (sequence, structure, or ligand) as described above ([Figure 2](#figure2)A/B). Connected components are assigned to train, validation, and test splits, with measurements from each dataset stratified across the splits ([Figure 2](#figure2)C). Transitive connections can link many nodes into one component that would be too large to fit any split. In this case, we remove central nodes, thereby cutting the graph into smaller components that satisfy the constraints. Under these assumptions, the split assignment (train, validation, test, or cut) then becomes a decision variable resulting in multiple valid assignments with varying numbers of nodes dropped from occupying multiple splits. The optimizer chooses the distinct set of cuts that retain the most data, subject to three constraints: partition sizes, one assignment per node, and pair relationships. Ligands are split by a second k-cut on the ligand graph, weighted by the number of data points whose protein was assigned to each split. Finally, any protein–protein or protein–ligand pair whose members fall in different splits is dropped. This formulation guarantees that no two homologs nor binding partners can appear in different splits.

We used this algorithm to generate 80/10/10 train, validation, and test splits across all datasets, with a 5% tolerance on partition size. Split fractions apply to proteins in single-protein datasets and to rows in paired datasets. A pair is kept only if both partners fall in the same split, so strict leakage control discards some data. Across all datasets, the k-cut solution retains 78.4% of data points and 65.4% of unique proteins. The dropped proteins are highly connected members of overrepresented families (**Fig S4**). The number of proteins dropped varied by dataset. For example, the final splits retain 184 of 298 Megascale wild-type proteins, with 153 (83.2%) in train, 16 (8.7%) in validation, and 15 (8.2%) in test. BindingDB retains 34,375 of 76,866 protein–ligand interactions, split into 28,703 train, 2,912 validation, and 2,760 test.

To incorporate a new dataset, a user can add a simple adapter to the benchmarks directory that parses the raw data and returns a list of datapoints. The user then specifies the similarity measures that should be taken into consideration as well as any classes to stratify on. Rerunning the scripts then rebuilds the similarity graph under the configured thresholds and writes a new set of split labels. Similarly, once the adapter is registered in the codebase, rerunning the unlabeled dataset sampling extends the screen to exclude homologs to the new data.

::::::figure{#figure2 align="center" type="image" label="Figure 2"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_1e1c7605_8567b213a33d"}
:::::

:::::figcaption
**Figure 2.** **Overview of the PLUG method.**

**(A)** Datapoints can exist as single entities (i.e., just a protein) or pairs (i.e., protein–protein or protein–ligand) and homology between entities can be based on sequence, structure, or composition (ligands). Default parameters are 30% sequence identity with 80% bidirectional coverage, 0.7 TM-score, and 0.5 Tanimoto, but are tunable by the user.

**(B)** PLUG checks for similarity in three modes. Coverage mode 0 considers the full length of both the target and query sequence, coverage model 1 considers the full length of the target sequence, and coverage mode 2 considers the full length of the query sequence. PLUG checks sequences from both query to target and then target to query and a hit is reported in either direction with sufficient coverage. The same principle is applied to the TM-score, which has separate alignments on query and target. Tanimoto similarity is a symmetric measure so does not require this method. Using this scoring, proteins can be screened out of pretraining corpora for unsupervised learning or datasets can be split into homology-aware train, validation, and test sets for fine tuning and transfer learning.

**(C)** To split existing datasets, a similarity graph constructed using all proteins in the benchmark. Sequence, structure, and ligand similarity measures are built into the edges of the graph rather than optimized. Following the Lo-Hi method, connected components are assigned to data splits greedily for small components and by cutting large components that do not fit in a split. The solver finds the minimum number of graph cuts to maximize the retained data (the number of row weighted proteins) subject to one-assignment-per-datapoint, no edge joining two splits, partition size, row floor, and class stratification constraints
:::::

::::::

## PLUG Challenge on Hugging Face

To facilitate community engagement we developed a website to track evaluations of nascent models on Hugging Face (<https://huggingface.co/spaces/PrismScienceOrg/PLUG-challenge>). [](https://huggingface.co) Predictions can be submitted to a leaderboard and model performance is evaluated on each dataset separately as well as the benchmark as a whole. Users submit a link to the code or literature describing the method used for predicting functions. Because the benchmark spans both regression and classification tasks, each metric (Spearman correlation, AUROC, and ${F}_{max}$) is normalized so that 0 corresponds to random chance (Spearman = 0, AUROC = 0.5, ${F}_{max}$ = 0) and 1 to a perfect prediction. Methods are ranked by an overall PLUG score reflecting their improvement over random chance and the leaderboard can be sorted by performance on individual datasets. If a single or subset of functions are submitted, the left out datasets are scored as chance in the overall PLUG score. There are separate leaderboards for unsupervised and supervised models. Unsupervised models are evaluated on all data from each benchmark dataset, whereas supervised models are evaluated on only the held out homology test sets for each benchmark dataset. Version controlled updates of PLUG will include additional functional datasets as they continue to grow in size and scope allowing researchers to track the progress and adaptability of models.

## Substantial Overlap Among Functional Datasets and Pretraining Corpora Creates Data Leakage Risk

To demonstrate the impact of PLUG’s splitting, we first measured overlap between functional datasets, and between those datasets and pretraining corpora. For each dataset, we calculated the percentage of proteins with a homolog within datasets as well as all other datasets and all other datasets of a different function (>30% sequence identity and $\geq$80% bidirectional coverage; [Figure 3](#figure3)A). For example, 138/298 (46%) Megascale sequences have a homolog in ProteinGym. In the other direction, 73/187 (39%) ProteinGym sequences have a homolog in Megascale. Of these, 64 sequences are exact matches because some assays appear in both datasets. Among datasets measuring different functions, CAFA5 and PPI overlap the most (80%), while both show the least overlap with FireProt (~1%). Most proteins in several datasets have a homolog in a dataset measuring a different function (88% of ProteinGym, 88% of AlloBench, and 89% of BindingDB). This overlap creates a substantial risk of data leakage ([Figure 3](#figure3)B).

Because a fixed identity threshold captures only whether a homolog exists, not how similar it is, we also measured overlap as the mean best-hit sequence identity between datasets ([Figure 3](#figure3)B). For each query sequence, we took its highest MMseqs2 identity to any sequence in the target dataset and averaged these values across the query dataset. [](https://doi.org/10.1038/nbt.3988) Diagonal values exclude self-matches and therefore reflect each dataset’s internal sequence redundancy. The largest overlaps are between datasets from common sources including AlloBench and PASSerRank and between PPI and SHS27K. Megascale is the most dissimilar to other datasets (mean best-hit identity ranging from 0.06–0.38), consistent with its high proportion of engineered proteins. LP-PDBBind [](https://doi.org/10.1021/acs.jpcb.5c08598) has the highest internal identity (0.87), reflecting its makeup of well-studied proteins screened against many ligands. For the five datasets with RCSB PDB identifiers (ATLAS, AlloBench, PASSerRank, FireProt, and LP-PDBBind), we also assessed structural overlap using a TM-score threshold of 0.7. [](https://doi.org/10.1021/acs.jpcb.5c08598) [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1002/jcc.27193) [](https://doi.org/10.1371/journal.pcbi.1004556) [](https://doi.org/10.1093/nar/gkad1084) Structural comparisons detail a similar pattern of overlap between the datasets with deposited structures (**Supp. Figure 1**). AlloBench, PASSerRank, FireProt, and LP-PDBBind each have structural homologs in another dataset for >83% of their proteins, while ATLAS, which is curated to be structurally diverse, only shows 43% structural overlap. The sequence, structure, and ligand overlap across benchmark datasets makes multifunctional evaluation difficult without jointly considering splitting.

We then ran an experiment mimicking an unlabeled training set used by modern protein language models (PLMs). We uniformly sampled one million candidate sequences from UniRef90, restricting the lengths to between 20 and 1000 residues to evaluate the overlap with the PLUG datasets ([Figure 3](#figure3)C). At ≥ 20% sequence identity, 38.5% of the sampled sequences have a homolog in at least one PLUG dataset, falling to 21.5% at 30% identity and 8.0% at 50%. This highlights the leakage that can arise from pretraining models on datasets with homologs in downstream evaluation sets in as much as the pretraining and downstream objectives are aligned. [](https://proceedings.mlr.press/v261/hermann24a.html) [](https://proceedings.mlr.press/v235/li24a.html) Distinguishing this leakage from genuine functional performance, however, is difficult. In masked language modeling (MLM), PLMs learn to predict masked residues from their sequence context, so the patterns they learn reflect the statistics of the natural proteins in the pretraining corpus. [](https://doi.org/10.1126/science.ads0018) [](https://doi.org/10.1101/2021.07.09.450648) A high likelihood could therefore indicate true evolutionary fitness or simply that a sequence, or its homologs, was in the pretraining set. If pretraining is broad enough, a PLM may learn a valid evolutionary prior that fine-tuning then transfers to more specific functions. [](https://doi.org/10.1126/science.ads0018) [](https://doi.org/10.1101/2021.07.09.450648) If not, pretraining on test-set homologs creates a leakage risk, and filtering steps like those described here should be applied.

::::::figure{#figure3 align="center" type="image" label="Figure 3"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_9e490583_e3027e1893fe"}
:::::

:::::figcaption
**Figure 3.** **Sequence similarity between datasets and effects on a held out training set.**

**(A)** The percent of sequence homologs ($\geq$30% ID, coverage >80% bidirectional) in each pair of databases. The wild-type sequences in the row database are searched against those in the column database and the percent with a homolog is reported. The diagonal represents internal redundancy and is indicative of where a random split can cause leakage. The two columns on the right indicate the fractional overlap to all other datasets and all other datasets of a different functional axis, respectively.

**(B)** The mean best-hit ID of sequences compared across all datasets and the UniRef90 sample. The diagonal is the internal self similarity of the benchmark without same-sequence comparisons, which indicates the diversity of sequences in a given set.

**(C)** Screening a uniform sample of sequences from UniRef90 (n=1,000,000) against each dataset reveals sequences with a relative in the evaluation. Screening against all datasets at once finds 38.5%, 21.5%, 13.1%, and 8.0% homologs at 20% ID, 30% ID, 40% ID, and 50% ID, respectively.
:::::

::::::

## Homology-aware splits reveal leakage that inflates protein language model performance

To demonstrate baseline evaluation with PLUG, we benchmarked ESM3 embeddings with linear and nonlinear probes on each functional dataset, comparing splits generated on single datasets with DataSAIL and across datasets with Lo-Hi. [](https://doi.org/10.1126/science.ads0018) [](https://doi.org/10.1038/s41467-025-58606-8) [](https://doi.org/10.48550/arXiv.2310.06399) We compared each splitting method to a size-matched random split that ignores homology. To ensure that the test sets were the same difficulty, we sampled the matched random splits until the zero shot performance was comparable. This helps distinguish whether any drop in probe performance reflects leakage removal rather than testing on a set of intrinsically harder proteins. Here, zero shot property prediction uses the log-likelihoods (single sequence), masked marginal ratios (mutation based properties), and log-likelihood difference (multisequence) values directly from the model to map to a function. For single property predictions this pretraining result, which measures sequence likelihood in the training set, can be correlated to the downstream objective, like thermostability. For multi-property predictions, like the hierarchically structured GO labels, a single value is not expected to be predictive of the multiple terms. Nevertheless, for consistency, we provide zero shot evaluations on all datasets in PLUG to demonstrate a baseline performance. This further provides a comparison of the alignment of the pretraining and downstream objectives and test set distributions as a result of the homology splitting procedures. [](https://proceedings.mlr.press/v261/hermann24a.html) [](https://proceedings.mlr.press/v235/li24a.html) [](https://doi.org/10.1101/2021.07.09.450648)

We used ESM3 (esm3-sm-open-v1) to generate residue-level log-likelihoods and embeddings for each protein sequence in the PLUG benchmark. [](https://doi.org/10.1126/science.ads0018) For the small molecule binding datasets (BindingDB and LP-PDBBind), we used ChemBERTa (ChemBERTa-77M-MLM, ChemBERTa-77M-MTR) to encode SMILES strings. [](https://doi.org/10.48550/arXiv.2010.09885) [](https://doi.org/10.1021/ci00057a005) We used masked marginals and log-likelihoods as zero-shot predictors, and used frozen embeddings as input to a linear probe (ridge regression for regression tasks, logistic regression for classification) and to an MLP with a single hidden layer. We did not fully fine-tune the models or extensively optimize probe hyperparameters; the only tuning was a sweep of the ridge regularization parameter on each validation set.

The size-matched random splits (Random @ Datasail and Random @ Lo-Hi) outperform the DataSAIL and Lo-Hi splits on most tasks, indicating data leakage ([Figure 4](#figure4)). Both similarity-aware splitting strategies reduced performance relative to size-matched random, with Lo-Hi splitting having a larger drop in performance (**Table S2**). These averages are largely driven by the small molecule binding tasks. With an MLP probe, BindingDB has a 42% reduction in performance and LP-PDBBind has a 72% reduction in performance. On classification tasks and on CAFA5, which is the only dataset scored with ${F}_{max}$ Lo-Hi and DataSAIL produced similar drops in performance.

The size of the drop varied across functional axes, likely due to how we performed data splitting. We show that performance drop from homology aware split is highly correlated with inherent dataset diversity (**Supp. Figure 5**). The most significant drops in performance are concentrated in the tasks where the label is a property of the whole protein or protein complex that could be shared by a homolog. Tasks that describe mutational effects showed higher zero shot performance compared to allostery, RMSF, binding and GO labels, which consistently scored near random chance, consistent with prior benchmarking efforts. [](https://doi.org/10.1101/2021.07.09.450648) The mutational and local function benchmarks (ATLAS, Allobench, PaSSerRank, Megascale) show smaller but still meaningful drops from homology splitting than the small molecule and binding type classifications (LP-PDBBind, BindingDB, and SHS27K).

Additionally the gold standard PPI benchmark shows a smaller effect from the leakage control compared to SHS27K, likely due to the nature of its construction. In the PPI benchmark, the set of proteins with positive interaction labels were scrambled and recombined into a roughly equal number of negative pairs without experimental evidence of interaction. [](https://doi.org/10.1093/bib/bbae076) This method of protecting for data leakage differs from Lo-Hi and DataSAIL by acting on the labels rather than the homology of the inputs, but appears effective in reducing memorization from sequence features alone.

::::::figure{#figure4 align="center" type="image" label="Figure 4"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_4efa8b4f_58e0cdaecfd5"}
:::::

:::::figcaption
**Figure 4.** **Baseline evaluation of frozen ESM3 embeddings on functional datasets highlights the need to control for data leakage.** Each benchmark is scored zero-shot (using masked marginals from mutations or log likelihoods from sequences) and with linear and nonlinear probes trained on the two different splitting strategies and compared to size and difficulty matched random splits. Zero-shot performance is reported with $\left|{\rho }\right|$. Random splitting inflates evaluations as leaked homologous sequences, structures, and ligands are memorized by the probe. Controlling for homology with PLUG assesses the ability of the probe to generalize to unseen data. Error bars are 95% percentile intervals from 4000 bootstrap resampling of test proteins, assays, or pairs. ${F}_{max\ }$ for CAFA5 does not decompose into per protein scores so error bars are not reported.
:::::

::::::

## Internal Layers can Demonstrate Varying Functional Representation

One way to improve protein function prediction models is through understanding where or why they excel and how they improve with scale. The last layer of a protein language model is not always the best for a given task. [](https://doi.org/10.64898/2026.07.08.737319) We therefore used the multifaceted PLUG benchmark to test if different layers of ESM3 understand different aspects of function. We trained the same ridge and MLP probes on embeddings from every fourth layer using the Lo-Hi splits. Layer zero corresponds to the input embeddings before the first transformer block. We report only probe results, as the zero-shot performance requires the reconstruction head, which was only pretrained on the final layer ([Figure 5](#figure5)).

Consistent with previous findings, most of the functional benchmarks reach their peak probe performance in later layers. [](https://doi.org/10.64898/2026.07.08.737319) The majority of datasets (Megascale, FireProt, ProteinGym, SHS27K, AlloBench, PASSerRank, and ATLAS) improved steadily with depth, and the rate of improvement nearly tripled after layer 32. Notably, layer 32 is where ESM3 was shown to fuse its multimodal inputs into a unified representation. [](https://doi.org/10.64898/2026.07.08.737319) CAFA5 followed a different pattern. Its performance rose steadily up to layer 32 and then plateaued, with peaks at layers 40 and 44.

The remaining benchmarks showed no clear layer preference. We attribute this to the weak signal that simple probes extract from these tasks, not to an absence of layer-specific structure. In both small molecule datasets, the ligand embedding is fixed across layers, which further dilutes the signal from the protein representation. Finally, we show that controlling for homology does not have a significant effect on the scaling results from PLM predictions (**Figure S6**).

::::::figure{#figure5 align="center" type="image" label="Figure 5"}

:::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_4fa5912a_c0df2f3e9114"}
:::::

:::::figcaption
**Figure 5.** **Layerwise performance of ESM3 on functional dataset.** Embeddings extracted from every 4th layer of ESM3 are used to predict different functions using the splits generated with the Lo-Hi method. Later layers predict function best for the majority of datasets (Megascale, FireProt, ProteinGym, SHS27K, AlloBench, PASSerRank, and ATLAS), while others show less preference for a single layer.
:::::

::::::

# Discussion

Predicting protein function has been a long-standing challenge in biology. Assessing our progress on this task is complicated by the fact that protein function is multidimensional. Evaluating functional prediction therefore requires testing across functional axes under the same leakage controls. PLUG is a framework that enables researchers to build and test diverse protein function predictors with strict data leakage standards across datasets. Across 11 datasets spanning granularities and modalities, PLUG homology splits reduced ESM3 probe performance by 25.9% for linear probes and 26.3% for MLP probes relative to matched random splits, indicating that random splits can substantially overestimate how well current PLMs generalize to unseen proteins.

The reduction in performance was not uniform across tasks. Tasks annotated at the level of whole proteins or families were most impacted by split, with 41–72% reductions for Lo-Hi MLP probes. [](https://doi.org/10.64898/2026.11.01.763444) Allostery and gold-standard PPIs showed intermediate drops (15–21%), whereas stability and fitness dropped by 6–15%, and RMSF by only 2%. The gradient suggests that properties that depend on local sequence and structure, such as flexibility and stability, transfer better to unseen homologs, potentially through reusable motif pairs. [](https://doi.org/10.1073/pnas.2406285121)

However, there are multiple other factors that also may also contribute to this gradient. First, we did not remove sequences that overlap with PLM pretraining data from our validation and test sets. [](https://proceedings.mlr.press/v261/hermann24a.html) Greater overlap between the pretraining corpora and test set would inflate the apparent transferability of residue-level tasks. Additionally, our splits do not take into account position-based splitting which substantially reduces PLM performance on fitness prediction. [](https://doi.org/10.64898/2026.03.08.710389) Generalizing functional predictions to underrepresented proteins and mutations must consider the specific granularity of the task, the degree of similarity to training sets, the alignment of pretraining and downstream objectives, as well as functional confounders that are ubiquitous in biological assays. [](https://doi.org/10.64898/2026.03.08.710389) [](https://doi.org/10.64898/2026.06.04.730121) For example, because PLMs optimize for natural sequences, an unnatural but beneficial mutation may be scored poorly. Fine tuning the model on a dataset where non-natural mutations have positive labels reshapes the latent around the smaller context of the fine tuning dataset. This method shows some success, but requires data from known mutations before predicting unknown mutations, therefore reducing the utility of the model.

PLUG is built to evolve. The framework can accept additional datasets, which it uses to generate supervised learning sets or corpora to withhold from PLM pretraining. We will continue to update its benchmark datasets and splits as new functional data become available, including datasets such as FLIP2 and data from emerging assays such as HT-MEK. [](https://doi.org/10.1126/science.abf8761) [](https://doi.org/10.64898/2026.02.23.707496) PLUG is intentionally modular to encourage users to add their own datasets and select the splitting strategy best suited to the task. New datasets need only a class in the codebase that parses the raw data before rerunning the splitting method or unlabeled pretraining screen. As no single threshold protects against all forms of leakage, we recommend that users generate splits at multiple thresholds and interpret model performance across all of them. This analysis shows the many ways a model can memorize its training set or overfit to a single or off-target objective.

While PLUG provides a useful test bed for training and evaluating protein function models, we acknowledge that it does not control for every type of data leakage. For example, many enzymes have different sequences and overall folds, but share common binding pocket scaffolding. In this case a model could memorize that a certain binding pocket shape or sequence maps to a certain activity. Data leakage can also occur based on residue position. [](https://doi.org/10.64898/2026.03.08.710389) Future work will address these and similar limitations by continually adding datasets and similarity measures that capture relevant biological mechanisms.

A single model for general protein function will likely learn a foundational representation that is broadly predictive and/or be trained and evaluated on sufficiently diverse, sizable functional datasets. The former approach is susceptible to data leakage between pretraining data and evaluation homologs and the latter is constrained by the limited overlap and scale of high quality functional labels across the vast protein landscape. PLUG represents an effort to alleviate two critical roadblocks in the development of functional predictors: the broad definitions of protein function and pervasive data leakage. To help support community wide assessment, we have set up a Hugging Face challenge and aim to engage the community through submissions of new predictions. (<https://huggingface.co/spaces/PrismScienceOrg/PLUG-challenge>)

# Methods

## PLUG datasets

We gathered datasets from the literature for seven different functional axes. We chose these datasets to cover different definitions of function commonly used in the literature and to span a range of data modalities. Each benchmark is accessed through a public online server, and scripts are provided to download and rebuild each set. [](https://doi.org/10.64898/2026.04.27.716980) [](https://doi.org/10.1038/s41586-023-06328-6) [](https://doi.org/10.1093/bib/bbae076) [](https://doi.org/10.1101/2023.12.07.570727) [](https://doi.org/10.1021/acs.jpcb.5c08598) [](https://doi.org/10.1093/nar/gkae1075) [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1002/jcc.27193) [](https://doi.org/10.1371/journal.pcbi.1004556) [](https://doi.org/10.1073/pnas.2314853121) [](https://doi.org/10.1093/nar/gkad1084) [](https://doi.org/10.1093/nar/gkae1113) [](https://doi.org/10.1093/nar/gkl999)

## Similarity Measures

In practice, the similarity metrics take a few forms across benchmarks. The simplest case are datasets where the datapoint is represented by a single, wild-type protein sequence (Megascale, CAFA5, and ProteinGym). The main form of leakage in this case is single sequence similarity to other wild-type proteins. For this work, we chose a standard protein sequence comparison tool, MMseqs2, [](https://doi.org/10.1038/nbt.3988) which efficiently compares bulk sets of sequences with varying thresholds. Sequences are considered to be homologous if their ID is above 0.3 with greater than 0.8 coverage in either direction. This is a standard threshold which balances detection of remote homologs with the alignment artifacts that begin to arise in the so-called twilight zone of sequence IDs. [](https://doi.org/10.1093/protein/12.2.85) Wild type sequences are compared for mutation based datasets and the number of mutant measurements is used as the weight for split assignment.

If a dataset provides protein sequence and structure (ATLAS, AlloBench, PASSerRank, FireProt and LP-PDBBind), then the same item has two representations, and therefore two modes of data leakage, sequence and structure. We used TM-score from Foldseek which aligns the structures before scoring the residue distances, and is less sensitive to local differences and outliers. The traditional value distinguishing similar fold families is 0.5, but this lower threshold in combination with sequence similarity can transitively link most nodes into a large component that can not be split. At a TM-score of 0.5, 98.1% of ATLAS nodes become linked with a structure edge. In an analysis of a SCOP database, two proteins with a TM-score of 0.5 had a 13% chance of being from the same fold family, compared to 90% when the score increased to 0.7. [](https://doi.org/10.1093/bioinformatics/btq066) To compare structures, we use the TM-score with a threshold of > 0.7 to indicate homologous folds.

Finally, some datasets include ligands (LP-PDBBind, BindingDB). To identify similar ligands we used ECFP4 Tanimoto similarity threshold of 0.5 (under the Morgan fingerprint). [](https://doi.org/10.1186/s13321-015-0069-3)

## Splitting PLUG datasets

### Adapted Lo-Hi balanced vertex minimum k cut splitter

Let $D$ be the set of $N$ data points $(d)$ and $M$ be the set of modalities $(m)$ (i.e. protein, ligand) in a benchmark. Then let ${V}_{m}(d)$ be the set of entities of modality $m$ that datapoint $d$ contains. For example, ${V}_{m}(d)$ contains single protein sequences for CAFA5, two proteins in SHS27K, and a protein–ligand pair in BindingDB. The pooled undirected graph ${G}_{m}=({V}_{m},\ {E}_{m})$ is constructed for each modality (i.e., protein or ligand) where the set of edges are defined when a similarity is above a threshold, ${\tau }_{f}$, for each feature, $f$ (i.e. sequence, structure, Tanimoto):

$${E}_{u,v}=\{(u,v):\exists f\in {F}_{m},\ si{m}_{f}(u,v)>{\tau }_{f}\}$$

Components that fit within a split are greedily assigned to the split, $s\in S=\{train,\ val,\ test\}$, which is furthest from its fractional share ${\{q}_{train},\ {q}_{val},\ {q}_{test}\}$ subject to a tolerance, $\epsilon$. The nodes of components which can not be dealt whole are reduced to clusters, $c\in {C}_{k,m}$. The clusters are then assigned to a split or dropped by $\pi (c):\ c\ \rightarrow s$. A decision variable, ${x}_{c,s}$, is defined as $1$ if cluster $c$ is assigned to split $s$ and $0$ otherwise. Each cluster must be assigned to exactly one split or dropped. Retention can then be defined as the datapoint weighted sum of proteins over all splits and datapoints, where ${w}_{c,s}$ is the number of datapoints that would be partitioned if cluster $c$ was placed in split $s$. Thus the problem is formulated as:

$$max\ \sum\limits_{c\in C}^{}\sum\limits_{s\in S}^{}{w}_{c,s}{x}_{c,s}$$

subject to split fractions:

$${(q}_{s}-\epsilon ){N}_{d}\leq {N}_{d,s}\leq ({q}_{s}+\epsilon ){N}_{d}$$

cluster assignment:

$$\sum\limits_{s\in S}^{}{x}_{c,s}\leq 1$$

edge constraints:

$${x}_{i,s}+\sum\limits_{s'\ne s}^{}{x}_{j,s'}\leq 1$$

and a datapoint floor:

$${N}_{\ d,\ s}\geq \theta {N}_{\ d}$$

With $\theta =(0.6,\ 0.05,\ 0.05)$ for train, validation, and test sets. ${N}_{\ d,\ s}$ is the number of proteins in single protein datasets and the number of interactions for paired datasets.

Finally, if the data lends itself to classes that should not be unevenly dealt (such as the assay types in the ProteinGym benchmark) the stratification constraint is applied for classes, $l\in L$:

$$\left|{{N}_{s,l}-{q}_{s}{N}_{retained,\ l}}\right|<\delta {q}_{s}{N}_{retained,l}$$

where $\delta$ is a user defined stratification tolerance.

## Baseline evaluations

For the Megascale, FireProt and ProteinGym zero-shot predictions, the wild-type sequence was masked at the mutant site and passed through the model. The log-likelihood ratio from mutant to wild-type residues was used as the predicted fitness label, summed over multiple sites for mutations. For the linear and MLP tracks, the wild-type and mutant sequences were each passed through the model and embeddings were mean pooled over the residues, resulting in protein level features. The differences between the two protein embeddings (mutant - wild-type) and their Hadamard product were concatenated into a single vector before being passed to a ridge regression model or a simple MLP. [](https://doi.org/10.48550/arXiv.1412.6575) The Spearman $\rho$ was computed within each test wild-type protein with at least three variants and averaged, weighted by the number of variants.

For ProteinGym a separate model was fit per assay type, following the established procedure to avoid the model memorizing the assay label scales. [](https://doi.org/10.1101/2023.12.07.570727) Spearman $\rho$ was calculated for each assay with at least three variants and then averaged across all assays sharing a UniProt ID. Those per protein means were then averaged within each assay type (binding, activity, expression, organismal fitness, stability) and then combined into one final correlation with equal weight, again following the established method.

For the gold standard PPI and SHS27K zero-shot predictions, the absolute difference in the z-score of the two proteins’ mean log likelihoods was used as the zero-shot score for binding. For the linear and MLP tracks, both sequences were passed through the encoder and embeddings were mean pooled over the residues, resulting in protein-level features. The differences between the two entities embeddings (absolute difference for PPIs) and their Hadamard product were concatenated into a single vector before being passed to a logistic (single regression for PPI and one per interaction term for SHS27K) and a single or multi-label MLP classifier.

For LP-PDBBind and BindingDB zero-shot predictions, the absolute difference in the z-score of the log likelihoods between the sequence and ligand was used as the binding prediction. For the linear and MLP tracks the embeddings from each model are first projected into a 256 dimension space to account for shape mismatch. The original embeddings are then concatenated with the absolute differences and Hadamard products between the projected vectors before being passed to ridge regression or MLP.

For the ATLAS RMSF, AlloBench, and PASSerRank zero-shot predictions, the log-likelihood of the residue was used to predict its label. For the linear and MLP tracks, the sequence was passed through the model and embeddings were extracted at the residue level. These per-residue embeddings were used as features to the ridge or logistic and simple MLP models. Because RMSF scales vary per protein and allosteric site density correlates with length, we score predictions per protein and report the average score across all proteins.

For the CAFA5 GO zero-shot predictions, the average log-likelihood of the sequence was min-max scaled across the whole dataset and then used as the score for each GO term in the dataset. For the linear and MLP models, the embeddings were mean-pooled over the residues. Each of the ontology branches was considered separately and only GO terms with $\geq$ 20 positive labels in the dataset were scored. The sequence-level features were then passed to one logistic regressor per term and an MLP for multi-label classification over all GO terms in the dataset. The ${F}_{max}$ was calculated per GO branch and averaged. For all ridge regression, the alpha parameter was swept over 7 values (1e0 to 1e6) on the validation dataset then the model is refitted on train and validation. The MLPs were single hidden layer models with dimension 256 (ReLU, dropout 0.1), trained with Adam (lr of 1e-3) with early stopping on the validation set. Calculations are reported as Spearman’s $\rho$ throughout the regression tasks.

The two splitting strategies are compared paired with a randomly sampled control that splits the same proteins to random splits only considering exact sequence matches. Up to 4000 random re-deals are evaluated with the zero-shot scoring (fewer for the larger datasets) and the sample with the closest zero-shot score to the method is kept. The change in performance from random control splits to method splits is calculated with the following equation, with chance performance taken as 0.5 for AUROC, 0 for Spearman, and 0 for ${F}_{max}$.

$$\Delta =\frac{control-split}{control-chance}$$

## Layerwise Analysis and Scaling

To evaluate whether the performance was different for the layers of ESM3, we scored the same frozen model and tuned probe on embeddings extracted from every 4th layer. The probes were trained and predictions were evaluated exactly as previously described. The 0th layer represents the initial untrained projection from the encoder head before it is passed to the trunk. Ligand representations from SMILES embeddings were not swept over the layers before combining with the protein embeddings from each layer.

To test whether scaling was affected by the splitting strategies, we benchmarked 5 different sizes of ESM2 (8M, 35M, 150M, 650M, and 3B) on Megascale and ATLAS using a linear probe trained on frozen embeddings.

::::::div{.bordered.lead data-internal="acknowledgements"}
### Acknowledgements

This work was supported by Radial, a division of Astera Institute (<https://ror.org/050rbg919>). This material is based upon work supported by the Defense Advanced Research Projects Agency under this Agreement (SAW and BC received research support). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. We thank Ziyue Zou for insightful suggestions pertaining to the incorporation of ligand datasets and small molecule models.
::::::

::::::details

:::::summary
# Supplemental
:::::

## Benchmark Statistics and Overlap

Protein fitness comes from ProteinGym which has internal functional variance for binding (n=12 proteins, 751,063 labels), activity (n=40 proteins, 447,319 labels), expression (n=18 proteins, 97,869 labels), organismal fitness (n=65 proteins, 1,034,722 labels), and stability (n=66 proteins, 134,794 labels). [](https://doi.org/10.1101/2023.12.07.570727) Six proteins are assayed for both activity and expression, four proteins are assayed for binding and expression, three are assayed for activity and organismal fitness, and one is assayed for both expression and organismal fitness. Mutational folding stability is accessed via the Megascale and FireProt databases as described in the ThermoMPNN paper, which applied filtering for single substitution mutations and wild-type structural fidelity. [](https://doi.org/10.1038/s41586-023-06328-6) [](https://doi.org/10.1371/journal.pcbi.1004556) [](https://doi.org/10.1073/pnas.2314853121) Megascale comprises 298 wild-type proteins to which mutations are made and 271,231 stability measurements are recorded. FireProt, similarly filtered to reduce duplicates and errors in the paper, consists of 3,404 measurements of mutational stability to 100 unique wild-type proteins. For the PPI datasets, the gold standard set is a binary interaction benchmark (n proteins=11,019; n protein-pairs=274,500); positive pairs are composed of experimentally detected human interactions drawn from assays such as yeast two-hybrid and affinity-purification mass spectrometry then weighted by the supporting evidence. [](https://doi.org/10.1093/bib/bbae076) The negatives are protein pairs with no reported interaction, sampled to balance the positives. STRING’s SHS27K (n proteins=1,690 unique sequences and n protein-pairs=7,624), scores protein pairs by combining experimental data, curated databases, co-expression, genomic context, and automated text-mining of the literature. [](https://doi.org/10.1093/nar/gkae1113) [](https://doi.org/10.1093/bioinformatics/btz328) Allostery is included from the AlloBench and PASSerRank datasets. Both assign ‘allosteric site’ labels to residues based on the AlloSteric Database’s (ASD) manual literature curation via PubMed, US patent, and European Patent literature search for proteins with at least 3 pieces of experimental allosteric evidence. [](https://doi.org/10.1021/acsomega.5c01263) [](https://doi.org/10.1002/jcc.27193) [](https://doi.org/10.1093/nar/gkq1022) AlloBench is a pipeline produced by integrating the AlloSteric Database, UniProt, the Mechanism and Catalytic Site Atlas, and the PDB. [](https://doi.org/10.1093/nar/gkae1010) [](https://doi.org/10.1093/nar/gkx1012) [](https://doi.org/10.1093/nar/28.1.235) PASSerRank is a prediction model of allosteric sites, taking positives from ASD. The AlloBench and PASSerRank datasets apply their own filtering criteria and additional label collection on top of ASD. As such, the two datasets overlap substantially, yielding 479 total unique sequences and 1,738 and 1,375 unique allosteric site records, respectively. Small-molecule binding data are sourced from BindingDB and LP-PDBBind, which represent a collected set of protein-ligand binding labels spanning EC50, IC50, Kd, and Ki values. [](https://doi.org/10.1021/acs.jpcb.5c08598) [](https://doi.org/10.1093/nar/gkae1075) While these values are not identically comparable, we treat them as general measures of binding strength and thus report evaluations using ranking procedures (Spearman correlation) instead of raw linear correlation, following recent approaches. [](https://doi.org/10.1101/2025.06.14.659707) BindingDB is a decades-long organization of experimental binding data from literature and patents. [](https://doi.org/10.1093/nar/gkl999) Initially tailored towards medicinal chemistry, it now contains 2.9 million binding measurements between different biopolymers. [](https://doi.org/10.1093/nar/gkae1075) [](https://doi.org/10.1093/nar/gkl999) Here, we focus on protein–ligand interactions. LP-PDBBind (Leak-Proof PDBbind) is a re-curation of PDBbind that seeks to create leak proof splits by iteratively breaking off clusters scored by chemical Dice similarity [](https://doi.org/10.2307/1932409) and sequence similarity after Needleman-Wunsch alignment. [](https://doi.org/10.1016/0022-2836(70)90057-4) BindingDB and LP-PDBBind contain 2,156 and 12,718 unique protein sequences with 41,027 and 15,477 unique SMILES strings, respectively. BindingDB and LP-PDBBind, which contain large collections of medicinal chemistry campaigns testing many ligands against a drug target, yield 76,866 (50,571 IC50, 21,968 Ki, 2,332 Kd, and 1,995 EC50) and 19,443 (7,190 IC50, 5,004 Ki, and 7,249 Kd) labeled interactions, respectively. ATLAS is a database of triplicate 100-nanosecond explicit-solvent MD simulations of 1,938 high-resolution crystal structures from 1,693 unique sequences. [](https://doi.org/10.1093/nar/gkad1084) The RMSFs of each residue in each protein are provided to indicate the short-timescale dynamics of the proteins.

**Table S1. Percent of sequences removed after screening a 1 million sequence sample of UniRef90 for homologs.**

:::::figure{type="table"}
| Benchmark/ID | 20%   | 30%   | 40%   | 50%  |
| ------------ | ----- | ----- | ----- | ---- |
| ATLAS        | 9.03  | 4.13  | 2.15  | 1.04 |
| AlloBench    | 4.28  | 1.94  | 0.99  | 0.53 |
| BindingDB    | 7.95  | 3.97  | 2.16  | 1.13 |
| CAFA5        | 15.00 | 7.21  | 3.80  | 2.09 |
| FireProt     | 1.18  | 0.57  | 0.23  | 0.12 |
| LP-PDBBind   | 17.23 | 8.94  | 4.93  | 2.65 |
| Megascale    | 3.65  | 1.13  | 0.48  | 0.29 |
| PASSerRank   | 3.56  | 1.58  | 0.77  | 0.39 |
| PPI          | 22.55 | 12.45 | 7.75  | 4.75 |
| ProteinGym   | 2.65  | 1.10  | 0.57  | 0.30 |
| SHS27K       | 7.22  | 3.38  | 1.94  | 1.12 |
| All          | 38.50 | 21.50 | 13.12 | 7.96 |
:::::

**Table S2. Change in performance on homology splits compared to size matched random sampled splits and chance prediction.**

:::::figure{type="table"}
| Benchmark  | DataSAIL Linear | DataSAIL MLP | Lo-Hi Linear | Lo-Hi MLP |
| ---------- | --------------- | ------------ | ------------ | --------- |
| ATLAS      | -12.3%          | -7.1%        | +8.5%        | -2.0%     |
| AlloBench  | +0.1%           | +0.6%        | -8.7%        | -7.4%     |
| BindingDB  | -36.4%          | -65.4%       | -65.5%       | -41.8%    |
| CAFA5      | -37.1%          | -41.0%       | -31.0%       | -41.0%    |
| FireProt   | -3.3%           | +3.8%        | +3.7%        | -5.9%     |
| LP-PDBBind | -43.6%          | -42.7%       | -82.3%       | -72.3%    |
| Megascale  | -7.8%           | -11.9%       | -14.9%       | -15.1%    |
| PASSerRank | -1.4%           | -2.4%        | -6.3%        | -7.5%     |
| PPI        | -7.2%           | -7.9%        | -4.4%        | -4.0%     |
| ProteinGym | +42.0%          | -7.5%        | +4.3%        | -8.3%     |
| SHS27K     | -29.8%          | -27.3%       | -21.2%       | -22.3%    |
| All        | -12.4%          | -19.0%       | -19.8%       | -20.7%    |
:::::

:::::figure{#figureS1 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_66ad6a85_3fc583df72bc"}
::::

::::figcaption
**Figure S1. Structural overlap comparisons analogous to sequence overlap characterization for ATLAS, AlloBench, PASSerRank, FireProt, and LP-PDBBind.**

**(A)** Percent of the row datasets' entities with a TM-score above 0.7 to any structure in the column dataset.

**(B)** Mean best-hit TM-score between all proteins in the corresponding datasets.
::::

:::::

## Homology splitting with DataSAIL

DataSAIL is a method to minimize the leakage across splits in a dataset. DataSAIL differs from PLUG in its objective and constraints but seeks to achieve the same goal: generating homology splits to prevent data leakage in evals. DataSAIL clusters the items into a similarity graph and then collapses to a defined number of clusters using agglomerative or spectral clustering (we reduced it to 1000 clusters per dataset). The method defines a quantity called leakage, which is the total similarity between two sets of items, as the objective function to minimize. Additionally, the method sets an upper bound constraint on the deviation from the split fraction. DataSAIL is run on MMseqs2 identity only; structural similarity is not controlled in the DataSAIL splits. For protein–ligand datasets, DataSAIL’s two dimensional mode could not meet the split sizes because one ligand cluster holds 84% (BindingDB) and 69% (LP-PDBBind) of the datapoints. We therefore run the method in S1 mode on the proteins and assign each ligand to the split containing most of its rows, dropping ligands whose Tanimoto neighbors fall in another split. To make the spectral clustering step reproducible, we fix the seed of the ARPACK solver and are able to return reproducible splits. We use DataSAIL’s default parameters where possible ($\epsilon$=0.05, $\delta$=0.15, SCIP solver, 80/10/10 splits), matching the Lo-Hi constraints.

:::::figure{#figureS2 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_f4d82751_95bc63e9867a"}
::::

::::figcaption
**Figure S2. Sequence identities of each dataset’s test proteins to their nearest training protein in any of the eleven datasets using Lo-Hi, DataSAIL, and their size-matched random controls.** Curves show the fraction of test proteins whose nearest homolog is at least the identity on the x axis. The dotted line marks the 30% ID threshold used to split Lo-Hi.
::::

:::::

:::::figure{#figureS3 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_eb73e919_c6122596f607"}
::::

::::figcaption
**Figure S3. TM-score of ATLAS, AlloBench, PASSerRank, FireProt, and LP-PDBBind test sets to the nearest training structure in any of those datasets.** The dotted line illustrates the leakage cutoff used when splitting with Lo-Hi.
::::

:::::

:::::figure{#figureS4 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_4ae0f299_12b883dd3694"}
::::

::::figcaption
**Figure S4. t-SNE analysis of proteins in each benchmark and which split they land in for the Lo-Hi method.** The method drops a large fraction of kinases from clusters in LP-PDBBind and BindingDB. These make up a significant portion of proteins in the datasets, likely from vast therapeutic campaigns.
::::

:::::

:::::figure{#figureS5 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_97e833d7_5ec5e4390d0d"}
::::

::::figcaption
**Figure S5. Change in performance due to homology control vs dataset diversity for each functional dataset.**

**(A)** Decrease in performance on the DataSAIL splits vs internal mean best-hit sequence ID ($\rho$=-0.59)

**(B)** Decrease in performance on the Lo-Hi splits vs internal mean best-hit sequence ID ($\rho$=-0.74)

**(C)** Decrease in performance on the DataSAIL splits vs internal mean best-hit TM-score ($\rho$=-0.70)

**(D)** Decrease in performance on the Lo-Hi splits vs internal mean best-hit TM-score ($\rho$=-0.90)
::::

:::::

:::::figure{#figureS6 align="center" type="image" unnumbered="true" label}

::::image{width="100%" src="https://thestacks-01.s3.amazonaws.com/publications/01M7JWG8PJE3W38Q5TPJVAGM59/media_c8069f0e_689f674f7a8c"}
::::

::::figcaption
**Figure S6. Scaling remains intact with homology aware splits.** Linear probe predictions on ATLAS and Megascale using 5 different sizes of ESM2 show that the improvement from model size alone is not significantly different between using random and homology-aware Lo-Hi splits.
::::

:::::

:::::bibtex
@article{lee2007, author = {Lee, David and Redfern, Oliver and Orengo, Christine}, title = {Predicting protein function from sequence and structure}, journal = {Nature Reviews Molecular Cell Biology}, volume = {8}, number = {12}, pages = {995–1005}, publisher = {Springer Science and Business Media LLC}, year = {2007}, doi = {10.1038/nrm2281}}

@article{depaoliskaluza2026, author = {De Paolis Kaluza, M. Clara and Ramola, Rashika and Joshi, Parnal and Piovesan, Damiano and Reade, Walter and Orchard, Sandra and Martin, Maria J. and Ignatchenko, Alex and Rost, Burkhard and Orengo, Christine A. and Robinson-Rechavi, Marc and Durand, Dannie and Brenner, Steven E. and Greene, Casey S. and Mooney, Sean D. and Friedberg, Iddo and Radivojac, Predrag}, title = {Advances in Protein Function Prediction from the Fifth CAFA Challenge}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.04.27.716980}}

@article{consortium2004, author = {Gene Ontology Consortium}, title = {The Gene Ontology (GO) database and informatics resource}, journal = {Nucleic Acids Research}, volume = {32}, number = {90001}, pages = {258D–261}, publisher = {Oxford University Press (OUP)}, year = {2004}, doi = {10.1093/nar/gkh036}}

@article{monod1965, author = {Monod, Jacque and Wyman, Jeffries and Changeux, Jean-Pierre}, title = {On the nature of allosteric transitions: A plausible model}, journal = {Journal of Molecular Biology}, volume = {12}, number = {1}, pages = {88–118}, publisher = {Elsevier BV}, year = {1965}, doi = {10.1016/s0022-2836(65)80285-6}}

@article{perutz1970, author = {PERUTZ, M. F.}, title = {Stereochemistry of Cooperative Effects in Haemoglobin: Haem–Haem Interaction and the Problem of Allostery}, journal = {Nature}, volume = {228}, number = {5273}, pages = {726–734}, publisher = {Springer Science and Business Media LLC}, year = {1970}, doi = {10.1038/228726a0}}

@article{blake1965, author = {BLAKE, C. C. F. and KOENIG, D. F. and MAIR, G. A. and NORTH, A. C. T. and PHILLIPS, D. C. and SARMA, V. R.}, title = {Structure of Hen Egg-White Lysozyme: A Three-dimensional Fourier Synthesis at 2 Å Resolution}, journal = {Nature}, volume = {206}, number = {4986}, pages = {757–761}, publisher = {Springer Science and Business Media LLC}, year = {1965}, doi = {10.1038/206757a0}}

@article{morris2022, author = {Morris, Rhiannon and Black, Katrina A. and Stollar, Elliott J.}, title = {Uncovering protein function: from classification to complexes}, journal = {Essays in Biochemistry}, volume = {66}, number = {3}, pages = {255–285}, publisher = {Portland Press Ltd.}, year = {2022}, doi = {10.1042/ebc20200108}}

@incollection{alberts2002, author = {{Alberts B} and others}, title = {Protein Function}, booktitle = {Molecular Biology of the Cell. 4th edition}, publisher = {Garland Science}, year = {2002}, url = {https://www.ncbi.nlm.nih.gov/books/NBK26911/}}

@article{andbateman2024, author = {and Bateman, Alex and Martin, Maria-Jesus and Orchard, Sandra and Magrane, Michele and Adesina, Aduragbemi and Ahmad, Shadab and Bowler-Barnett, Emily H and Bye-A-Jee, Hema and Carpentier, David and Denny, Paul and Fan, Jun and Garmiri, Penelope and Gonzales, Leonardo Jose da Costa and Hussein, Abdulrahman and Ignatchenko, Alexandr and Insana, Giuseppe and Ishtiaq, Rizwan and Joshi, Vishal and Jyothi, Dushyanth and Kandasaamy, Swaathi and Lock, Antonia and Luciani, Aurelien and Luo, Jie and Lussi, Yvonne and Marin, Juan Sebastian Martinez and Raposo, Pedro and Rice, Daniel L and Santos, Rafael and Speretta, Elena and Stephenson, James and Totoo, Prabhat and Tyagi, Nidhi and Urakova, Nadya and Vasudev, Preethi and Warner, Kate and Wijerathne, Supun and Yu, Conny Wing-Heng and Zaru, Rossana and Bridge, Alan J and Aimo, Lucila and Argoud-Puy, Ghislaine and Auchincloss, Andrea H and Axelsen, Kristian B and Bansal, Parit and Baratin, Delphine and Batista Neto, Teresa M and Blatter, Marie-Claude and Bolleman, Jerven T and Boutet, Emmanuel and Breuza, Lionel and Gil, Blanca Cabrera and Casals-Casas, Cristina and Echioukh, Kamal Chikh and Coudert, Elisabeth and Cuche, Beatrice and de Castro, Edouard and Estreicher, Anne and Famiglietti, Maria L and Feuermann, Marc and Gasteiger, Elisabeth and Gaudet, Pascale and Gehant, Sebastien and Gerritsen, Vivienne and Gos, Arnaud and Gruaz, Nadine and Hulo, Chantal and Hyka-Nouspikel, Nevila and Jungo, Florence and Kerhornou, Arnaud and Mercier, Philippe Le and Lieberherr, Damien and Masson, Patrick and Morgat, Anne and Paesano, Salvo and Pedruzzi, Ivo and Pilbout, Sandrine and Pourcel, Lucille and Poux, Sylvain and Pozzato, Monica and Pruess, Manuela and Redaschi, Nicole and Rivoire, Catherine and Sigrist, Christian J A and Sonesson, Karin and Sundaram, Shyamala and Sveshnikova, Anastasia and Wu, Cathy H and Arighi, Cecilia N and Chen, Chuming and Chen, Yongxing and Huang, Hongzhan and Laiho, Kati and Lehvaslaiho, Minna and McGarvey, Peter and Natale, Darren A and Ross, Karen and Vinayaka, C R and Wang, Yuqi and Zhang, Jian}, title = {UniProt: the Universal Protein Knowledgebase in 2025}, journal = {Nucleic Acids Research}, volume = {53}, number = {D1}, pages = {D609–D617}, publisher = {Oxford University Press (OUP)}, year = {2024}, doi = {10.1093/nar/gkae1010}}

@article{richardson2022, author = {Richardson, Lorna and Allen, Ben and Baldi, Germana and Beracochea, Martin and Bileschi, Maxwell L and Burdett, Tony and Burgin, Josephine and Caballero-Pérez, Juan and Cochrane, Guy and Colwell, Lucy J and Curtis, Tom and Escobar-Zepeda, Alejandra and Gurbich, Tatiana A and Kale, Varsha and Korobeynikov, Anton and Raj, Shriya and Rogers, Alexander B and Sakharova, Ekaterina and Sanchez, Santiago and Wilkinson, Darren J and Finn, Robert D}, title = {MGnify: the microbiome sequence data analysis resource in 2023}, journal = {Nucleic Acids Research}, volume = {51}, number = {D1}, pages = {D753–D759}, publisher = {Oxford University Press (OUP)}, year = {2022}, doi = {10.1093/nar/gkac1080}}

@article{durairaj2023, author = {Durairaj, Janani and Waterhouse, Andrew M. and Mets, Toomas and Brodiazhenko, Tetiana and Abdullah, Minhal and Studer, Gabriel and Akdel, Mehmet and Andreeva, Antonina and Bateman, Alex and Tenson, Tanel and Hauryliuk, Vasili and Schwede, Torsten and Pereira, Joana}, title = {What is hidden in the darkness? Deep-learning assisted large-scale protein family curation uncovers novel protein families and folds}, publisher = {openRxiv}, year = {2023}, doi = {10.1101/2023.03.14.532539}}

@article{decrecylagard2022, author = {de Crécy-lagard, Valérie and Amorin de Hegedus, Rocio and Arighi, Cecilia and Babor, Jill and Bateman, Alex and Blaby, Ian and Blaby-Haas, Crysten and Bridge, Alan J and Burley, Stephen K and Cleveland, Stacey and Colwell, Lucy J and Conesa, Ana and Dallago, Christian and Danchin, Antoine and de Waard, Anita and Deutschbauer, Adam and Dias, Raquel and Ding, Yousong and Fang, Gang and Friedberg, Iddo and Gerlt, John and Goldford, Joshua and Gorelik, Mark and Gyori, Benjamin M and Henry, Christopher and Hutinet, Geoffrey and Jaroch, Marshall and Karp, Peter D and Kondratova, Liudmyla and Lu, Zhiyong and Marchler-Bauer, Aron and Martin, Maria-Jesus and McWhite, Claire and Moghe, Gaurav D and Monaghan, Paul and Morgat, Anne and Mungall, Christopher J and Natale, Darren A and Nelson, William C and O’Donoghue, Seán and Orengo, Christine and O’Toole, Katherine H and Radivojac, Predrag and Reed, Colbie and Roberts, Richard J and Rodionov, Dmitri and Rodionova, Irina A and Rudolf, Jeffrey D and Saleh, Lana and Sheynkman, Gloria and Thibaud-Nissen, Francoise and Thomas, Paul D and Uetz, Peter and Vallenet, David and Carter, Erica Watson and Weigele, Peter R and Wood, Valerie and Wood-Charlson, Elisha M and Xu, Jin}, title = {A roadmap for the functional annotation of protein families: a community perspective}, journal = {Database}, volume = {2022}, publisher = {Oxford University Press (OUP)}, year = {2022}, doi = {10.1093/database/baac062}}

@article{martinezcuesta2015, author = {Martínez Cuesta, Sergio and Rahman, Syed Asad and Furnham, Nicholas and Thornton, Janet M.}, title = {The Classification and Evolution of Enzyme Function}, journal = {Biophysical Journal}, volume = {109}, number = {6}, pages = {1082–1086}, publisher = {Elsevier BV}, year = {2015}, doi = {10.1016/j.bpj.2015.04.020}}

@article{westroberts2024, author = {West-Roberts, Jacob and Kravitz, Joshua and Jha, Nishant and Cornman, Andre and Hwang, Yunha}, title = {Diverse Genomic Embedding Benchmark for functional evaluation across the tree of life}, publisher = {openRxiv}, year = {2024}, doi = {10.1101/2024.07.10.602933}}

@article{decrecylagard2025, author = {de Crécy-Lagard, Valérie and Dias, Raquel and Sexson, Nick and Friedberg, Iddo and Yuan, Yifeng and Swairjo, Manal A}, title = {Limitations of current machine learning models in predicting enzymatic functions for uncharacterized proteins}, journal = {G3: Genes, Genomes, Genetics}, editor = {Rokas, A}, volume = {15}, number = {10}, publisher = {Oxford University Press (OUP)}, year = {2025}, doi = {10.1093/g3journal/jkaf169}}

@article{hedstrom2002, author = {Hedstrom, Lizbeth}, title = {Serine Protease Mechanism and Specificity}, journal = {Chemical Reviews}, volume = {102}, number = {12}, pages = {4501–4524}, publisher = {American Chemical Society (ACS)}, year = {2002}, doi = {10.1021/cr000033x}}

@article{richard2024, author = {Richard, A. Clay and Pantazes, Robert J.}, title = {Using Short Molecular Dynamics Simulations to Determine the Important Features of Interactions in Antibody–Protein Complexes}, journal = {Proteins: Structure, Function, and Bioinformatics}, volume = {93}, number = {4}, pages = {812–830}, publisher = {Wiley}, year = {2024}, doi = {10.1002/prot.26773}}

@article{wankowicz2022, author = {Wankowicz, Stephanie A and de Oliveira, Saulo H and Hogan, Daniel W and van den Bedem, Henry and Fraser, James S}, title = {Ligand binding remodels protein side-chain conformational heterogeneity}, journal = {eLife}, volume = {11}, publisher = {eLife Sciences Publications, Ltd}, year = {2022}, doi = {10.7554/elife.74114}}

@article{gibbs1995, author = {Gibbs, C. S. and Coutré, S. E. and Tsiang, M. and Li, W-X. and Jain, A. K. and Dunn, K. E. and Law, V. S. and Mao, C. T. and Matsumura, S. Y. and Mejza, S. Y. and Paborsky, L. R. and Leung, L. L. K.}, title = {Conversion of thrombin into an anticoagulant by protein engineering}, journal = {Nature}, volume = {378}, number = {6555}, pages = {413–416}, publisher = {Springer Science and Business Media LLC}, year = {1995}, doi = {10.1038/378413a0}}

@article{markin2023, author = {Markin, Craig J. and Mokhtari, Daniel A. and Du, Siyuan and Doukov, Tzanko and Sunden, Fanny and Cook, Jordan A. and Fordyce, Polly M. and Herschlag, Daniel}, title = {Decoupling of catalysis and transition state analog binding from mutations throughout a phosphatase revealed by high-throughput enzymology}, journal = {Proceedings of the National Academy of Sciences}, volume = {120}, number = {29}, publisher = {National Academy of Sciences}, year = {2023}, doi = {10.1073/pnas.2219074120}}

@article{markin2021, author = {Markin, C. J. and Mokhtari, D. A. and Sunden, F. and Appel, M. J. and Akiva, E. and Longwell, S. A. and Sabatti, C. and Herschlag, D. and Fordyce, P. M.}, title = {Revealing enzyme functional architecture via high-throughput microfluidic enzyme kinetics}, journal = {Science}, volume = {373}, number = {6553}, publisher = {American Association for the Advancement of Science (AAAS)}, year = {2021}, doi = {10.1126/science.abf8761}}

@article{tsuboyama2023, author = {Tsuboyama, Kotaro and Dauparas, Justas and Chen, Jonathan and Laine, Elodie and Mohseni Behbahani, Yasser and Weinstein, Jonathan J. and Mangan, Niall M. and Ovchinnikov, Sergey and Rocklin, Gabriel J.}, title = {Mega-scale experimental analysis of protein folding stability in biology and design}, journal = {Nature}, volume = {620}, number = {7973}, pages = {434–444}, publisher = {Springer Science and Business Media LLC}, year = {2023}, doi = {10.1038/s41586-023-06328-6}}

@article{bernett2024, author = {Bernett, Judith and Blumenthal, David B and List, Markus}, title = {Cracking the black box of deep sequence-based protein–protein interaction prediction}, journal = {Briefings in Bioinformatics}, volume = {25}, number = {2}, publisher = {Oxford University Press (OUP)}, year = {2024}, doi = {10.1093/bib/bbae076}}

@article{pejaver2022, author = {Pejaver, Vikas and Byrne, Alicia B. and Feng, Bing-Jian and Pagel, Kymberleigh A. and Mooney, Sean D. and Karchin, Rachel and O’Donnell-Luria, Anne and Harrison, Steven M. and Tavtigian, Sean V. and Greenblatt, Marc S. and Biesecker, Leslie G. and Radivojac, Predrag and Brenner, Steven E. and Biesecker, Leslie G. and Harrison, Steven M. and Tayoun, Ahmad A. and Berg, Jonathan S. and Brenner, Steven E. and Cutting, Garry R. and Ellard, Sian and Greenblatt, Marc S. and Kang, Peter and Karbassi, Izabela and Karchin, Rachel and Mester, Jessica and O’Donnell-Luria, Anne and Pesaran, Tina and Plon, Sharon E. and Rehm, Heidi L. and Strande, Natasha T. and Tavtigian, Sean V. and Topper, Scott}, title = {Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria}, journal = {The American Journal of Human Genetics}, volume = {109}, number = {12}, pages = {2163–2177}, publisher = {Elsevier BV}, year = {2022}, doi = {10.1016/j.ajhg.2022.10.013}}

@article{richards2015, author = {Richards, Sue and Aziz, Nazneen and Bale, Sherri and Bick, David and Das, Soma and Gastier-Foster, Julie and Grody, Wayne W. and Hegde, Madhuri and Lyon, Elaine and Spector, Elaine and Voelkerding, Karl and Rehm, Heidi L.}, title = {Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology}, journal = {Genetics in Medicine}, volume = {17}, number = {5}, pages = {405–424}, publisher = {Elsevier BV}, year = {2015}, doi = {10.1038/gim.2015.30}}

@article{hayes2025, author = {Hayes, Thomas and Rao, Roshan and Akin, Halil and Sofroniew, Nicholas J. and Oktay, Deniz and Lin, Zeming and Verkuil, Robert and Tran, Vincent Q. and Deaton, Jonathan and Wiggert, Marius and Badkundri, Rohil and Shafkat, Irhum and Gong, Jun and Derry, Alexander and Molina, Raul S. and Thomas, Neil and Khan, Yousuf A. and Mishra, Chetan and Kim, Carolyn and Bartie, Liam J. and Nemeth, Matthew and Hsu, Patrick D. and Sercu, Tom and Candido, Salvatore and Rives, Alexander}, title = {Simulating 500 million years of evolution with a language model}, journal = {Science}, volume = {387}, number = {6736}, pages = {850–858}, publisher = {American Association for the Advancement of Science (AAAS)}, year = {2025}, doi = {10.1126/science.ads0018}}

@article{ullanat2026, author = {Ullanat, Varun and Jing, Bowen and Sledzieski, Samuel and Berger, Bonnie}, title = {Learning the language of protein-protein interactions}, journal = {Nature Communications}, volume = {17}, number = {1}, publisher = {Springer Science and Business Media LLC}, year = {2026}, doi = {10.1038/s41467-025-67971-3}}

@article{akiyama2026, author = {Akiyama, Yo and Zhang, Zhidian and Tang, Olivia and Kim, Rachel Seongeun and Mirdita, Milot and Steinegger, Martin and Ovchinnikov, Sergey}, title = {Expanding the scope of protein language modeling to protein-protein interactions with MSA Pairformer}, journal = {Cell}, volume = {189}, number = {16}, pages = {4964–4979.e8}, publisher = {Elsevier BV}, year = {2026}, doi = {10.1016/j.cell.2026.06.029}}

@article{notin2023, author = {Notin, Pascal and Kollasch, Aaron W. and Ritter, Daniel and van Niekerk, Lood and Paul, Steffanie and Spinner, Hansen and Rollins, Nathan and Shaw, Ada and Weitzman, Ruben and Frazer, Jonathan and Dias, Mafalda and Franceschi, Dinko and Orenbuch, Rose and Gal, Yarin and Marks, Debora S.}, title = {ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction}, publisher = {openRxiv}, year = {2023}, doi = {10.1101/2023.12.07.570727}}

@article{didi2026, author = {Didi, Kieran and Alamdari, Sarah and Lu, Alex X. and Wittmann, Bruce and Johnston, Kadina E. and Amini, Ava P. and Madani, Ali and Czeneszew, Maya and Dallago, Christian and Yang, Kevin K.}, title = {FLIP2: Expanding Protein Fitness Landscape Benchmarks for Real-World Machine Learning Applications}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.02.23.707496}}

@article{gillis2013, author = {Gillis, Jesse and Pavlidis, Paul}, title = {Characterizing the state of the art in the computational assignment of gene function: lessons from the first critical assessment of functional annotation (CAFA)}, journal = {BMC Bioinformatics}, volume = {14}, number = {S3}, publisher = {Springer Science and Business Media LLC}, year = {2013}, doi = {10.1186/1471-2105-14-s3-s15}}

@article{thomas2012, author = {Thomas, Paul D. and Wood, Valerie and Mungall, Christopher J. and Lewis, Suzanna E. and Blake, Judith A. and}, title = {On the Use of Gene Ontology Annotations to Assess Functional Similarity among Orthologs and Paralogs: A Short Report}, journal = {PLoS Computational Biology}, editor = {Bourne, Philip E.}, volume = {8}, number = {2}, pages = {e1002386}, publisher = {Public Library of Science (PLoS)}, year = {2012}, doi = {10.1371/journal.pcbi.1002386}}

@article{kaufman2012, author = {Kaufman, Shachar and Rosset, Saharon and Perlich, Claudia and Stitelman, Ori}, title = {Leakage in data mining: Formulation, detection, and avoidance}, journal = {ACM Transactions on Knowledge Discovery from Data}, volume = {6}, number = {4}, pages = {1–21}, publisher = {Association for Computing Machinery (ACM)}, year = {2012}, doi = {10.1145/2382577.2382579}}

@article{bernett2024a, author = {Bernett, Judith and Blumenthal, David B. and Grimm, Dominik G. and Haselbeck, Florian and Joeres, Roman and Kalinina, Olga V. and List, Markus}, title = {Guiding questions to avoid data leakage in biological machine learning applications}, journal = {Nature Methods}, volume = {21}, number = {8}, pages = {1444–1453}, publisher = {Springer Science and Business Media LLC}, year = {2024}, doi = {10.1038/s41592-024-02362-y}}

@inproceedings{hermann2024, author = {{Hermann L} and {Fiedler T} and {Nguyen HA} and {Nowicka M} and {Bartoszewicz JM}}, title = {Beware of Data Leakage from Protein LLM Pretraining}, booktitle = {Proceedings of the 19th Machine Learning in Computational Biology meeting}, editor = {{Knowles DA} and {Mostafavi S}}, volume = {261}, pages = {106--116}, publisher = {PMLR}, year = {2024}, url = {https://proceedings.mlr.press/v261/hermann24a.html}}

@article{vieira2026, author = {Vieira, Luiz C. and Lin, Sophia and Wilke, Claus O.}, title = {Intrinsic dataset features drive mutational effect prediction by protein language models}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.03.08.710389}}

@article{joeres2025, author = {Joeres, Roman and Blumenthal, David B. and Kalinina, Olga V.}, title = {Data splitting to avoid information leakage with DataSAIL}, journal = {Nature Communications}, volume = {16}, number = {1}, publisher = {Springer Science and Business Media LLC}, year = {2025}, doi = {10.1038/s41467-025-58606-8}}

@article{graber2025, author = {Graber, David and Stockinger, Peter and Meyer, Fabian and Mishra, Siddhartha and Horn, Claus and Buller, Rebecca}, title = {Resolving data bias improves generalization in binding affinity prediction}, journal = {Nature Machine Intelligence}, volume = {7}, number = {10}, pages = {1713–1725}, publisher = {Springer Science and Business Media LLC}, year = {2025}, doi = {10.1038/s42256-025-01124-5}}

@article{li2026, author = {Li, Jie and Guan, Xingyi and Zhang, Oufan and Sun, Kunyang and Wang, Yingze and Bagni, Dorian and Head-Gordon, Teresa}, title = {Leak Proof PDBBind: A Reorganized Data Set of Protein–Ligand Complexes for More Generalizable Binding Affinity Prediction}, journal = {The Journal of Physical Chemistry B}, volume = {130}, number = {2}, pages = {730–740}, publisher = {American Chemical Society (ACS)}, year = {2026}, doi = {10.1021/acs.jpcb.5c08598}}

@article{riesselman2018, author = {Riesselman, Adam J. and Ingraham, John B. and Marks, Debora S.}, title = {Deep generative models of genetic variation capture the effects of mutations}, journal = {Nature Methods}, volume = {15}, number = {10}, pages = {816–822}, publisher = {Springer Science and Business Media LLC}, year = {2018}, doi = {10.1038/s41592-018-0138-4}}

@article{fallahpour2026, author = {Fallahpour, Adibvafa and Seyed-Ahmadi, Arman and Idehpour, Parsa and Ibrahim, Omar and Choi, Benedict M.H. and Gupta, Purav and Naimer, Jack and Zhu, Kevin and Adduri, Abhinav and Shah, Arnav and Ma, Shihao and Güloglu, Talu and Liu, Nuo and Cui, Haotian and Jain, Arihant and de Castro, Max and Fallahpour, Amirfaham and Cembellin-Prieto, Antonio and Stiles, John S. and Nemčko, Filip and Nevue, Alexander A. and Moon, Hyungseok C. and Sosnick, Lucas and Markham, Olivia and Duan, Haonan and Lee, Michelle Y. Y. and Salvador, Andrea F. M. and Maddison, Chris J. and Thaiss, Christoph A. and Ricci-Tam, Chiara and Plosky, Brian S. and Burke, Dave P. and Hsu, Patrick D. and Goodarzi, Hani and Wang, Bo}, title = {BioReason-Pro: Advancing Protein Function Prediction with Multimodal Biological Reasoning}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.03.19.712954}}

@misc{wanglabbioreason, title = {wanglab/bioreason-pro-test-data · Datasets at Hugging Face}, url = {https://huggingface.co/datasets/wanglab/bioreason-pro-test-data}}

@article{liu2024, author = {Liu, Tiqing and Hwang, Linda and Burley, Stephen K and Nitsche, Carmen I and Southan, Christopher and Walters, W Patrick and Gilson, Michael K}, title = {BindingDB in 2024: a FAIR knowledgebase of protein-small molecule binding data}, journal = {Nucleic Acids Research}, volume = {53}, number = {D1}, pages = {D1633–D1644}, publisher = {Oxford University Press (OUP)}, year = {2024}, doi = {10.1093/nar/gkae1075}}

@article{maity2025, author = {Maity, Dibyajyoti and Qiao, Baofu}, title = {AlloBench: A Data Set Pipeline for the Development and Benchmarking of Allosteric Site Prediction Tools}, journal = {ACS Omega}, volume = {10}, number = {17}, pages = {17973–17982}, publisher = {American Chemical Society (ACS)}, year = {2025}, doi = {10.1021/acsomega.5c01263}}

@article{tian2023, author = {Tian, Hao and Xiao, Sian and Jiang, Xi and Tao, Peng}, title = {PASSerRank: Prediction of allosteric sites with learning to rank}, journal = {Journal of Computational Chemistry}, volume = {44}, number = {28}, pages = {2223–2229}, publisher = {Wiley}, year = {2023}, doi = {10.1002/jcc.27193}}

@article{huang2010, author = {Huang, Z. and Zhu, L. and Cao, Y. and Wu, G. and Liu, X. and Chen, Y. and Wang, Q. and Shi, T. and Zhao, Y. and Wang, Y. and Li, W. and Li, Y. and Chen, H. and Chen, G. and Zhang, J.}, title = {ASD: a comprehensive database of allosteric proteins and modulators}, journal = {Nucleic Acids Research}, volume = {39}, number = {Database}, pages = {D663–D669}, publisher = {Oxford University Press (OUP)}, year = {2010}, doi = {10.1093/nar/gkq1022}}

@article{bednar2015, author = {Bednar, David and Beerens, Koen and Sebestova, Eva and Bendl, Jaroslav and Khare, Sagar and Chaloupkova, Radka and Prokop, Zbynek and Brezovsky, Jan and Baker, David and Damborsky, Jiri}, title = {FireProt: Energy- and Evolution-Based Computational Design of Thermostable Multiple-Point Mutants}, journal = {PLOS Computational Biology}, editor = {Dokholyan, Nikolay V.}, volume = {11}, number = {11}, pages = {e1004556}, publisher = {Public Library of Science (PLoS)}, year = {2015}, doi = {10.1371/journal.pcbi.1004556}}

@article{dieckhaus2024, author = {Dieckhaus, Henry and Brocidiacono, Michael and Randolph, Nicholas Z. and Kuhlman, Brian}, title = {Transfer learning to leverage larger datasets for improved prediction of protein stability changes}, journal = {Proceedings of the National Academy of Sciences}, volume = {121}, number = {6}, publisher = {National Academy of Sciences}, year = {2024}, doi = {10.1073/pnas.2314853121}}

@article{ashburner2000, author = {Ashburner, Michael and Ball, Catherine A. and Blake, Judith A. and Botstein, David and Butler, Heather and Cherry, J. Michael and Davis, Allan P. and Dolinski, Kara and Dwight, Selina S. and Eppig, Janan T. and Harris, Midori A. and Hill, David P. and Issel-Tarver, Laurie and Kasarskis, Andrew and Lewis, Suzanna and Matese, John C. and Richardson, Joel E. and Ringwald, Martin and Rubin, Gerald M. and Sherlock, Gavin}, title = {Gene Ontology: tool for the unification of biology}, journal = {Nature Genetics}, volume = {25}, number = {1}, pages = {25–29}, publisher = {Springer Science and Business Media LLC}, year = {2000}, doi = {10.1038/75556}}

@article{maynardsmith1970, author = {MAYNARD SMITH, JOHN}, title = {Natural Selection and the Concept of a Protein Space}, journal = {Nature}, volume = {225}, number = {5232}, pages = {563–564}, publisher = {Springer Science and Business Media LLC}, year = {1970}, doi = {10.1038/225563a0}}

@article{ross1981, author = {Ross, Philip D. and Subramanian, S.}, title = {Thermodynamics of protein association reactions: forces contributing to stability}, journal = {Biochemistry}, volume = {20}, number = {11}, pages = {3096–3102}, publisher = {American Chemical Society (ACS)}, year = {1981}, doi = {10.1021/bi00514a017}}

@article{eisenberg2000, author = {Eisenberg, David and Marcotte, Edward M. and Xenarios, Ioannis and Yeates, Todd O.}, title = {Protein function in the post-genomic era}, journal = {Nature}, volume = {405}, number = {6788}, pages = {823–826}, publisher = {Springer Science and Business Media LLC}, year = {2000}, doi = {10.1038/35015694}}

@article{chothia1975, author = {Chothia, Cyrus and Janin, Joël}, title = {Principles of protein–protein recognition}, journal = {Nature}, volume = {256}, number = {5520}, pages = {705–708}, publisher = {Springer Science and Business Media LLC}, year = {1975}, doi = {10.1038/256705a0}}

@article{karplus1979, author = {KARPLUS, M. and MCCAMMON, J. A.}, title = {Protein structural fluctuations during a period of 100 ps}, journal = {Nature}, volume = {277}, number = {5697}, pages = {578–578}, publisher = {Springer Science and Business Media LLC}, year = {1979}, doi = {10.1038/277578a0}}

@article{vandermeersche2023, author = {Vander Meersche, Yann and Cretin, Gabriel and Gheeraert, Aria and Gelly, Jean-Christophe and Galochkina, Tatiana}, title = {ATLAS: protein flexibility description from atomistic molecular dynamics simulations}, journal = {Nucleic Acids Research}, volume = {52}, number = {D1}, pages = {D384–D392}, publisher = {Oxford University Press (OUP)}, year = {2023}, doi = {10.1093/nar/gkad1084}}

@article{szklarczyk2024, author = {Szklarczyk, Damian and Nastou, Katerina and Koutrouli, Mikaela and Kirsch, Rebecca and Mehryary, Farrokh and Hachilif, Radja and Hu, Dewei and Peluso, Matteo E and Huang, Qingyao and Fang, Tao and Doncheva, Nadezhda T and Pyysalo, Sampo and Bork, Peer and Jensen, Lars J and von Mering, Christian}, title = {The STRING database in 2025: protein networks with directionality of regulation}, journal = {Nucleic Acids Research}, volume = {53}, number = {D1}, pages = {D730–D737}, publisher = {Oxford University Press (OUP)}, year = {2024}, doi = {10.1093/nar/gkae1113}}

@article{chen2019, author = {Chen, Muhao and Ju, Chelsea J -T and Zhou, Guangyu and Chen, Xuelu and Zhang, Tianran and Chang, Kai-Wei and Zaniolo, Carlo and Wang, Wei}, title = {Multifaceted protein–protein interaction prediction based on Siamese residual RCNN}, journal = {Bioinformatics}, volume = {35}, number = {14}, pages = {i305–i314}, publisher = {Oxford University Press (OUP)}, year = {2019}, doi = {10.1093/bioinformatics/btz328}}

@article{steinegger2017, author = {Steinegger, Martin and Söding, Johannes}, title = {MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets}, journal = {Nature Biotechnology}, volume = {35}, number = {11}, pages = {1026–1028}, publisher = {Springer Science and Business Media LLC}, year = {2017}, doi = {10.1038/nbt.3988}}

@article{xu2010, author = {Xu, Jinrui and Zhang, Yang}, title = {How significant is a protein structure similarity with TM-score = 0.5?}, journal = {Bioinformatics}, volume = {26}, number = {7}, pages = {889–895}, publisher = {Oxford University Press (OUP)}, year = {2010}, doi = {10.1093/bioinformatics/btq066}}

@article{zhang2005, author = {Zhang, Y.}, title = {TM-align: a protein structure alignment algorithm based on the TM-score}, journal = {Nucleic Acids Research}, volume = {33}, number = {7}, pages = {2302–2309}, publisher = {Oxford University Press (OUP)}, year = {2005}, doi = {10.1093/nar/gki524}}

@article{bajusz2015, author = {Bajusz, Dávid and Rácz, Anita and Héberger, Károly}, title = {Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?}, journal = {Journal of Cheminformatics}, volume = {7}, number = {1}, publisher = {Springer Science and Business Media LLC}, year = {2015}, doi = {10.1186/s13321-015-0069-3}}

@article{zhou2024, author = {Zhou, Hongyi and Skolnick, Jeffrey}, title = {Utility of the Morgan Fingerprint in Structure-Based Virtual Ligand Screening}, journal = {The Journal of Physical Chemistry B}, volume = {128}, number = {22}, pages = {5363–5370}, publisher = {American Chemical Society (ACS)}, year = {2024}, doi = {10.1021/acs.jpcb.4c01875}}

@article{vankempen2023, author = {van Kempen, Michel and Kim, Stephanie S. and Tumescheit, Charlotte and Mirdita, Milot and Lee, Jeongjae and Gilchrist, Cameron L. M. and Söding, Johannes and Steinegger, Martin}, title = {Fast and accurate protein structure search with Foldseek}, journal = {Nature Biotechnology}, volume = {42}, number = {2}, pages = {243–246}, publisher = {Springer Science and Business Media LLC}, year = {2023}, doi = {10.1038/s41587-023-01773-0}}

@misc{elangovan2021, author = {Elangovan, Aparna and He, Jiayuan and Verspoor, Karin}, title = {Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation}, publisher = {arXiv}, year = {2021}, doi = {10.48550/arXiv.2102.01818}}

@misc{steshin2023, author = {Steshin, Simon}, title = {Lo-Hi: Practical ML Drug Discovery Benchmark}, publisher = {arXiv}, year = {2023}, doi = {10.48550/arXiv.2310.06399}}

@inproceedings{cornaz2014, author = {Cornaz, Denis and Furini, Fabio and Lacroix, Mathieu and Malaguti, Enrico and Mahjoub, A. Ridha and Martin, Sebastien}, title = {Mathematical formulations for the Balanced Vertex k-Separator Problem}, booktitle = {2014 International Conference on Control, Decision and Information Technologies (CoDIT)}, pages = {176–181}, publisher = {IEEE}, year = {2014}, doi = {10.1109/codit.2014.6996889}}

@article{balas2005, author = {Balas, Egon and Souza, Cid C. de}, title = {The vertex separator problem: a polyhedral investigation}, journal = {Mathematical Programming}, volume = {103}, number = {3}, pages = {583–608}, publisher = {Springer Science and Business Media LLC}, year = {2005}, doi = {10.1007/s10107-005-0574-7}}

@misc{huggingface, author = {{Hugging Face}}, title = {Hugging Face}, url = {https://huggingface.co}}

@inproceedings{li2024, author = {{Li FZ} and {Amini AP} and {Yue Y} and {Yang KK} and {Lu AX}}, title = {Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, editor = {{Salakhutdinov R} and others}, volume = {235}, pages = {27351--27375}, publisher = {PMLR}, year = {2024}, url = {https://proceedings.mlr.press/v235/li24a.html}}

@article{meier2021, author = {Meier, Joshua and Rao, Roshan and Verkuil, Robert and Liu, Jason and Sercu, Tom and Rives, Alexander}, title = {Language models enable zero-shot prediction of the effects of mutations on protein function}, publisher = {openRxiv}, year = {2021}, doi = {10.1101/2021.07.09.450648}}

@misc{chithrananda2020, author = {Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath}, title = {ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction}, publisher = {arXiv}, year = {2020}, doi = {10.48550/arXiv.2010.09885}}

@article{weininger1988, author = {Weininger, David}, title = {SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules}, journal = {Journal of Chemical Information and Computer Sciences}, volume = {28}, number = {1}, pages = {31–36}, publisher = {American Chemical Society (ACS)}, year = {1988}, doi = {10.1021/ci00057a005}}

@article{steenwyk2026, author = {Steenwyk, Jacob L}, title = {A geometric atlas of how ESM3 organizes modalities across depth}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.07.08.737319}}

@article{zou2026, author = {Zou, Ziyue and Hazelwood, Isaiah and Estevam, Gabriella and Rao, Jingyou and Chrispens, Karson and Howard, Matthew K. and Coyote-Maestas, Willow and Fraser, James S}, title = {Learning inhibitor interactions from protein deep mutational scanning data and deep learning models}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.11.01.763444}}

@article{zhang2024, author = {Zhang, Zhidian and Wayment-Steele, Hannah K. and Brixi, Garyk and Wang, Haobo and Kern, Dorothee and Ovchinnikov, Sergey}, title = {Protein language models learn evolutionary statistics of interacting sequence motifs}, journal = {Proceedings of the National Academy of Sciences}, volume = {121}, number = {45}, publisher = {National Academy of Sciences}, year = {2024}, doi = {10.1073/pnas.2406285121}}

@article{woolley2026, author = {Woolley, Phillip R. and Feller, Aaron L. and Ellington, Andrew D. and Wilke, Claus O.}, title = {Inherent limitations of zero-shot mutation prediction in protein engineering}, publisher = {openRxiv}, year = {2026}, doi = {10.64898/2026.06.04.730121}}

@article{liu2007, author = {Liu, T. and Lin, Y. and Wen, X. and Jorissen, R. N. and Gilson, M. K.}, title = {BindingDB: a web-accessible database of experimentally determined protein-ligand binding affinities}, journal = {Nucleic Acids Research}, volume = {35}, number = {Database}, pages = {D198–D201}, publisher = {Oxford University Press (OUP)}, year = {2007}, doi = {10.1093/nar/gkl999}}

@article{rost1999, author = {Rost, Burkhard}, title = {Twilight zone of protein sequence alignments}, journal = {Protein Engineering, Design and Selection}, volume = {12}, number = {2}, pages = {85–94}, publisher = {Oxford University Press (OUP)}, year = {1999}, doi = {10.1093/protein/12.2.85}}

@misc{yang2014, author = {Yang, Bishan and Yih, Wen-tau and He, Xiaodong and Gao, Jianfeng and Deng, Li}, title = {Embedding Entities and Relations for Learning and Inference in Knowledge Bases}, publisher = {arXiv}, year = {2014}, doi = {10.48550/arXiv.1412.6575}}

@article{ribeiro2017, author = {Ribeiro, António J M and Holliday, Gemma L and Furnham, Nicholas and Tyzack, Jonathan D and Ferris, Katherine and Thornton, Janet M}, title = {Mechanism and Catalytic Site Atlas (M-CSA): a database of enzyme reaction mechanisms and active sites}, journal = {Nucleic Acids Research}, volume = {46}, number = {D1}, pages = {D618–D623}, publisher = {Oxford University Press (OUP)}, year = {2017}, doi = {10.1093/nar/gkx1012}}

@article{berman2000, author = {Berman, H. M.}, title = {The Protein Data Bank}, journal = {Nucleic Acids Research}, volume = {28}, number = {1}, pages = {235–242}, publisher = {Oxford University Press (OUP)}, year = {2000}, doi = {10.1093/nar/28.1.235}}

@article{passaro2025, author = {Passaro, Saro and Corso, Gabriele and Wohlwend, Jeremy and Reveiz, Mateo and Thaler, Stephan and Somnath, Vignesh Ram and Getz, Noah and Portnoi, Tally and Roy, Julien and Stark, Hannes and Kwabi-Addo, David and Beaini, Dominique and Jaakkola, Tommi and Barzilay, Regina}, title = {Boltz-2: Towards Accurate and Efficient Binding Affinity Prediction}, publisher = {openRxiv}, year = {2025}, doi = {10.1101/2025.06.14.659707}}

@article{dice1945, author = {Dice, Lee R.}, title = {Measures of the Amount of Ecologic Association Between Species}, journal = {Ecology}, volume = {26}, number = {3}, pages = {297–302}, publisher = {Wiley}, year = {1945}, doi = {10.2307/1932409}}

@article{needleman1970, author = {Needleman, Saul B. and Wunsch, Christian D.}, title = {A general method applicable to the search for similarities in the amino acid sequence of two proteins}, journal = {Journal of Molecular Biology}, volume = {48}, number = {3}, pages = {443–453}, publisher = {Elsevier BV}, year = {1970}, doi = {10.1016/0022-2836(70)90057-4}}
:::::

::::::
