Calibration, Not Compounding: What Experimental Feedback Did for a Protein-Stability Model

When you feed a protein-stability model your own measurements, does that create cumulative value for future proteins — or only for the one you measured? Experimental feedback could eventually sharpen predictions through target-specific adaptation, given enough of the target's own labels — but the cross-protein correction formulations we tested did not transfer to held-out targets. This post separates the part of that result that is close to a mathematical identity from the part that is empirical and scoped, and says plainly where the evidence stops.

"Self-improving" property predictors are usually pitched as compounding: your data makes the model better everywhere, over time. In practice, experimental feedback can enter a model many ways — fine-tuning, active learning, Gaussian-process or Bayesian-optimization layers, adapters, in-context conditioning, or target-specific supervised models. It helps to keep three mechanisms distinct: scalar calibration (rescale a frozen score), a cross-protein residual correction (a learned head that reads the mutation), and target-specific refitting (a model trained on the target's own labels). This post is about the first two — the lightweight, "compounding"-flavored ones.

Scalar recalibration cannot re-rank, by construction

A feedback loop of this kind fits a correction g on top of a frozen predictor f; the deployed score is g(f(x)), learned from labels. When g is a low-parameter, monotone map of the scalar output — an affine or isotonic recalibration — one fact is definitional. A strictly increasing scalar transformation preserves rankings exactly; a strictly decreasing one reverses them; a non-strict isotonic map can introduce ties but cannot materially reorder. So for any rank-based endpoint, the only rank-relevant degree of freedom in a scalar recalibration is direction: keep or invert. An affine correction inverts only when its slope is negative.

Figure 1. A scalar calibration g sits on top of the frozen base model f — the only learned part (≈2 parameters). Because it is a monotone map of a single score, for a ranking metric it can only keep the order (positive slope) or flip it (negative slope); it cannot reorder within.

Which means that if you evaluate scalar calibration by a ranking metric, the outcome is largely predetermined — it can flip a systematically backwards model, and otherwise does nothing to the order. That is a property of the operation, not a finding. The empirical question is separate: can a feature-conditioned correction — one that reads the mutation, not just the score — learn something on some proteins that improves ranking on others?

Feature-aware corrections did not transfer, in the settings we tested

We evaluated this on a public protein-stability corpus. The endpoint is continuous ΔΔG, and the metric is per-protein Spearman ρ between predicted and measured ΔΔG under protein-disjoint held-out evaluation, across three disjoint protein sets: one to train the base, a separate set to fit the correction (on out-of-sample residuals), and held-out targets to score. Details and sign conventions are in Methods.

Three results:

• A feature-aware residual head did not improve held-out ranking. Median per-protein ρ was 0.489 (base) vs 0.490 (base + head). Paired across the 70 held-out proteins, the median difference was +0.003 (95% bootstrap CI [−0.008, +0.014]); the head improved ranking on 53% of proteins; Wilcoxon p = 0.93 — no detectable improvement in this sample.
• Richer features did not help transfer, and could hurt it. A base restricted to three substitution features (hydropathy, volume, and charge deltas) generalized better across held-out proteins (median ρ 0.524) than one on the full 12-feature set (0.489), and adding the full features back through a head pulled it down (0.503).
• No directional inversions among the held-out proteins. 0 of 70 held-out domains had negative per-protein ρ. But these are all densely measured domains — the corpus does not probe the sparse, poorly-covered regime where a backwards base (the case a sign flip fixes) is most likely, so we cannot claim anything about that regime from this dataset (see Limitations).

Figure 2. (A) Per-protein held-out ρ, base vs base + feature-aware head, paired across 70 proteins — points cluster around the diagonal (median Δ +0.003, 95% CI [−0.008, +0.014], 53% improved, Wilcoxon p = 0.93). (B) Per-protein ρ for a full-feature vs a substitution-only base; the lower-context base generalizes better and no protein is inverted. Public stability corpus, protein-disjoint held-out evaluation.

So, in these settings, the correction did not travel — a statement about the calibration and low-capacity residual-head formulations tested, on this dataset and base class.

Why the base tends to absorb the transferable signal

There is a mechanism that makes this the expected outcome for a good base — without making it a theorem. Decompose a mutation's effect into a shared component — average effects associated with hydropathy, volume, and charge changes, together with recurring local-context effects that hold broadly across proteins — and a target-specific component — a particular protein's exact packing, burial, electrostatics, and long-range coupling. The evolutionary-biochemistry literature draws the same line between nonspecific, near-additive effects and specific, context-dependent epistasis (Starr & Thornton, 2016).

A base trained across many proteins fits the shared, near-additive component; that is what its cross-protein generalization is. The substitution-only base generalized better, too — consistent with the possibility that its lower-context features captured more broadly shared effects, while the richer base absorbed context-specific patterns that did not survive the cross-protein shift (a bias–variance trade under distribution shift).

The tempting next step — "so the residual is target-specific and cannot transfer" — is too strong, and worth resisting. A residual is what a particular base failed to explain; it can hold fold-class, assay, or structural signal that is shared across proteins but absent from that base's feature set. The defensible statement is narrower: for a sufficiently expressive, well-trained base, the residual is increasingly dominated by target-specific effects, and in our experiments the tested residual parameterizations captured little transferable signal. A correction rich enough to capture transferable residual structure would functionally become part of a richer base model, rather than remaining a lightweight calibration layer.

Where more data goes: base coverage, and per-target models

If not into a transferable head, added data flows to two places we could measure. It improves the base's cross-protein generalization — which, along the single trajectory we sampled, showed no further improvement past ~50 training proteins (median per-protein ρ ~0.45 at 10, ~0.50 at 50, flat to ~190). And it improves target-specific models refit on a single protein's own labels: such a model exceeded the general model only above roughly 10² of that target's labels (within-target ρ ~0.42 → ~0.52 from 50 to 200 labels, versus a base of ~0.48 at zero) — below that, the base was better. This target-specific refitting is a different mechanism from the cross-protein correction above; it does not transfer, and it is what "your data helps your target" actually rests on.

Figure 3. (A) Held-out per-protein ρ vs. number of training proteins — no further gain past ~50 along this single sampled trajectory (not a robust saturation estimate; see Limitations). (B) A target-specific model exceeds the general base above ~10² of the target's own labels — an observed trajectory (single draw per point; see Methods), not a stable crossover estimate. Both are specific to this dataset, base class, and feature set.

That ~10² crossover is a property of this setup, not a universal number: it moves with assay noise, target length, representation quality, regularization, and whether variants are selected randomly or actively. Biswas et al. (2021) achieved useful target adaptation from as few as ~24 assayed variants — with a richer representation, a different endpoint, and family-tuned features. The general point survives the specific number: there was no low-label regime, in the formulations tested, where a correction learned on other targets beat the base on a new one.

What may transfer more reliably: knowing where the base is weak

Our exploratory reading — of these results and the wider literature — is that reliability may be more portable than a learned correction: where a model is likely to fail often tracks a protein's distance to the training distribution, and is estimable before any measurements exist on a new target. A well-calibrated uncertainty estimate would then flag likely failure up front — arguably the more useful signal on a cold target than a correction that does not generalize. We did not measure reliability transfer end-to-end here, so this is a direction, not a result; the uncertainty-aware work it points to (Groth et al., 2024; Gurev et al., 2025) is adjacent, not a direct test on stability ΔΔG.

Keep your negative measurements

Fitting g estimates a relation between prediction and outcome, and one tail of the outcome distribution poorly constrains the other — particularly sign, intercept, and behavior in the opposite tail. Because ~91% of the single mutations in the dataset are destabilizing, discarding them would remove most of the data available to estimate that relation and leave the opposite tail poorly constrained. This is the same imbalance that leaves off-the-shelf predictors markedly less accurate on stabilizing than destabilizing mutations — the mutations engineers actually want (Pucci et al., 2018). Retaining both classes is not a nicety. (Separately, paired forward and reverse mutations offer a thermodynamic consistency test via the expected antisymmetry relation ΔΔG(A→B) = −ΔΔG(B→A), which most predictors violate on the Ssym benchmark; JanusDDG, 2026.)

Figure 4. Measured ΔΔG across the corpus is overwhelmingly destabilizing (~91%). Stabilizing "hits" are the minority tail — so discarding non-hits would remove most of the data available to fit the prediction–outcome relation. (Values clipped to [−6, 10] for display.)

How this fits with prior work

None of this contradicts the strong evidence that cross-protein learning improves variant-effect prediction; it delimits which gains exist. Two established kinds are not what we tested. Better base models — transferring representations from pretrained networks (ThermoMPNN; Dieckhaus et al., 2024) or meta-predictors combining many scores across proteins (CPT-1; Jagota et al., 2023) — improve accuracy on new proteins, which is our point restated: a correction rich enough to transfer is a better base trained across proteins. Cross-target offset calibration — learning that one protein's variants are, on average, more severe than another's — lifts global ranking but is orthogonal to within-target re-ranking, and can conceal heterogeneous within-target performance in pooled metrics (Tejura et al., 2024). Task matters too: pathogenicity prediction rides on conservation, a strongly transferable signal, whereas biophysical ΔΔG residuals are dominated by structure-specific context. Our claim is the narrow one: for the frozen-base + low-capacity-correction formulations tested, feedback did not transfer across targets.

Methods

• Data. Public deep mutational stability corpus (Tsuboyama et al., 2023, ml_ready experimental ΔΔG release), restricted to protein domains with ≥90 measured single mutations (capped per protein for balance). The two analyses draw independently from this corpus, at slightly different sample sizes (stated in their split bullets below). Sign convention: we used the released ddG values as-is; ~91% of single mutations are positive (mean +2.25 kcal/mol), i.e. larger positive values are more destabilizing, and per-protein Spearman is computed between predicted and measured values under that single, consistent orientation. (Ranking metrics are invariant to a global sign flip; we state the convention explicitly because some cited work uses the opposite orientation.)
• Base model. Gradient-boosted trees (scikit-learn HistGradientBoostingRegressor; max_iter 250, max_depth 4, learning_rate 0.08, min_leaf 40) on 12 sequence features: Kyte–Doolittle / volume / charge deltas, aromatic- and polar-status change, wild-type hydropathy and volume, local-window hydropathy/aromaticity/charge, and proline/glycine indicators. The "substitution-only" base uses the three substitution deltas only.
• Correction and cross-fitting. A residual head (same model class) predicts measured ΔΔG − base prediction. Base-training, correction-donor, and held-out-target proteins are three disjoint sets (130 / 100 / 70 — 300 domains, ~108,000 mutations): the head's residual labels come from correction-donor proteins the base was never trained on, so they are out-of-sample, not in-sample stacking.
• Scaling and per-target curves. These use a separate draw of 280 domains, split protein-disjoint into 90 held-out targets and 190 training-pool proteins. The scaling curve (Fig 2A) trains the base on the first N pool proteins and scores the 90 held-out. The per-target curve (Fig 2B): for each of the 90 held-out proteins that has ≥300 measured mutations (all 90 qualify), we hold out a fixed 120-mutation test set, train a target-specific model (HistGradientBoostingRegressor; max_iter 200, max_depth 3, learning_rate 0.1, min_leaf 15) on N randomly chosen mutations, and score the 120. Each (protein, N) is a single random draw — no repeats; each curve point is the median per-protein ρ across the 90 proteins. Figure 2B is therefore an observed trajectory, not a repeated-sampling crossover estimate.
• Statistics. For the correction we report the paired per-protein difference, a paired bootstrap 95% interval (5,000 resamples), the fraction of proteins improved, and a Wilcoxon signed-rank test.
• Limitations (material). (1) The scaling curve is a single donor-protein trajectory, not repeated subsampling with confidence intervals — the plateau past ~50 could reflect dataset redundancy or family composition. (2) The per-target curve is a single draw per point (no repeats). (3) Protein-disjoint held-out evaluation does not remove close homologues; a sequence-identity-clustered hold-out at several thresholds would be a stronger out-of-distribution test, and is the main methodological upgrade to pursue. (4) Coverage was not operationalized on this corpus; the "sparse-regime / backwards-base" discussion is an expectation drawn from separate observations, not measured here. (5) One base class and feature set — a PLM- or structure-based base, and an explicit comparison against fine-tuning, are needed to generalize the conclusion.

Practical implications

• Guard the correction with target-internal cross-validation. Adapt only if it beats the base on held-out folds of that target's own data; otherwise keep the base.
• Report per-protein, not pooled. Pooled correlation can be dominated by between-protein differences and conceal heterogeneous within-target performance (Tejura et al., 2024).
• Evaluate protein-held-out, ideally sequence-cluster-held-out. Within-protein splits measure an easier problem.
• Retain both label classes, including destabilizing/negative results.
• For zero-data targets, lean on a reliability/OOD estimate rather than a borrowed correction.
• To lift accuracy on a hard class, extend base-model coverage of that class. In our tests, calibration was not a shortcut around it.

References

1. Tsuboyama, K., Dauparas, J., Chen, J., et al. Mega-scale experimental analysis of protein folding stability in biology and design. Nature 620, 434–444 (2023). doi:10.1038/s41586-023-06328-6
2. Pucci, F., Bernaerts, K.V., Kwasigroch, J.M., Rooman, M. Quantification of biases in predictions of protein stability changes upon mutations. Bioinformatics 34(21), 3659–3665 (2018). doi:10.1093/bioinformatics/bty348
3. Pancotti, C., Benevenuta, S., Birolo, G., et al. Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset (S669). Briefings in Bioinformatics 23(2), bbab555 (2022). doi:10.1093/bib/bbab555 — general background on out-of-distribution stability-prediction accuracy; not the source of the specific per-protein ρ values here.
4. Dieckhaus, H., Brocidiacono, M., Randolph, N.Z., Kuhlman, B. Transfer learning to leverage larger datasets for improved prediction of protein stability changes (ThermoMPNN). PNAS 121(6), e2314853121 (2024). doi:10.1073/pnas.2314853121
5. Jagota, M., et al. Cross-protein transfer learning substantially improves disease variant prediction (CPT-1). Genome Biology 24, 182 (2023). doi:10.1186/s13059-023-03024-6
6. Starr, T.N., Thornton, J.W. Epistasis in protein evolution. Protein Science 25(7), 1204–1218 (2016). doi:10.1002/pro.2897
7. Tejura, M., Fayer, S., McEwen, A.E., Flynn, J., Starita, L.M., Fowler, D.M. Calibration of variant effect predictors on genome-wide data masks heterogeneous performance across genes. American Journal of Human Genetics 111(9), 2031–2043 (2024). PubMed 39173626
8. Biswas, S., Khimulya, G., Alley, E.C., Esvelt, K.M., Church, G.M. Low-N protein engineering with data-efficient deep learning. Nature Methods 18, 389–396 (2021). doi:10.1038/s41592-021-01100-y
9. Groth, P.M., et al. Kermut: Composite kernel regression for protein variant effects. NeurIPS (2024). arXiv:2407.00002
10. Gurev, S., Youssef, N., Jain, N., Mehrotra, A., Leung, S.R.M., Jackson, A., Marks, D.S. Evaluating variant effect prediction across viruses (EVEREST). bioRxiv 2025.08.04.668549 (2025). biorxiv — reliability metrics for viral variant-effect prediction; adjacent, not stability ΔΔG.
11. Barducci, G., Rossi, I., Codicè, F., Rollo, C., Repetto, V., Pancotti, C., Iannibelli, V., Sanavia, T., Fariselli, P. JanusDDG: a thermodynamics-compliant model for sequence-based protein stability via two-fronts multi-head attention. Communications Biology (2026); preprint arXiv:2504.03278