Benchmarking Virtual Cell Models: AI Predicts Perturbations

Jonathan Alles

EVOBYTE Digital Biology

Introduction: When Virtual Cell Predictions Become Useful

Imagine using an AI model to choose which gene knockdowns deserve a costly follow-up experiment. Its predictions look convincing in familiar cells. However, your discovery program studies a different cellular context. Will the model transfer useful biology—or simply produce plausible-looking gene expression?

That is the question behind zero-shot perturbation prediction. A virtual cell model, in this setting, predicts how a cell’s gene expression changes after an intervention. The practical test is whether those predictions remain useful where measured responses are unavailable.

The 2026 Virtual Cell Challenge makes this problem concrete. Models receive unperturbed expression profiles and gene targets, then predict responses in cellular contexts whose perturbation measurements are withheld.

For discovery teams, this provides a useful evaluation template. Before committing experimental budgets, ask whether a model generalizes beyond familiar contexts and predicts responses specific to each intervention.

Held-Out Cellular Contexts: Define What “Unseen” Means

The challenge uses CRISPR interference, or CRISPRi, to reduce the activity of targeted genes. Perturb-seq links these interventions to single-cell gene expression measurements.

Its 2026 task spans six cell lines from different tissues of origin. Three support validation, while three are reserved for final testing. There is no challenge-specific training set, although participants may train on external data. Models receive cells carrying non-targeting guides to describe each context’s basal state.

Therefore, “zero-shot” does not mean receiving no information about the destination cells. It means predicting without measured perturbation responses from those contexts.

Mirror this distinction in your own benchmark. If deployment involves a new cell line, hold out that line’s entire perturbation dataset. Randomly separating individual cells from the same experiment tests something easier: predictions among closely related observations.

Also separate unseen contexts from unseen interventions. Predicting a familiar knockdown in a new cell line tests context transfer. Predicting a new knockdown there adds another generalization challenge. Report these settings separately rather than hiding them inside one score.

Importantly, success across cell lines does not establish success in primary tissues or patient-derived cells. Your test boundary should match the biological claim.

Before training, audit data leakage: information entering model development that would not be available when making real predictions. Track source datasets and inspect pretrained model provenance. A nominally held-out context may already appear in a public training collection.

Fit learned preprocessing on training data, except for explicitly permitted adaptation using target-context controls. Never use withheld perturbed cells to select model features or tune parameters. Keep a final context untouched after validation; repeated leaderboard feedback is still feedback.

Simple Baselines: Make Complexity Earn Its Place

A sophisticated model should beat a simple alternative that could support the same decision.

Start with a no-change baseline that predicts the target context’s control expression for every knockdown. This asks whether the model learns intervention effects beyond basal cell identity.

Next, add a generic-response baseline: estimate the average perturbation-induced change from training contexts and apply it to the destination control profile. It ignores target identity, so it tests whether apparent accuracy comes from predicting a common response.

When the same target has been perturbed elsewhere, include a transfer baseline. Add its average training-context effect to the destination control profile, using a consistent expression scale. This asks whether the model improves on straightforward reuse of known effects.

For example, suppose a model predicts a shared stress response for nearly every knockdown. Its profiles may resemble measured perturbed cells, yet offer little help choosing between targets. A generic-response baseline makes that limitation visible.

Published benchmarks have found simple baselines competitive with deep learning models in tested settings. However, newer calibration research also shows that some metrics obscure genuine improvements. The lesson is not that complex models cannot work. It is that both models and evaluation methods must earn trust.

Use identical inputs and preprocessing rules across comparisons. A baseline calculated from hidden target-context perturbation outcomes can serve as an evaluator’s reference, but not as a deployable zero-shot competitor.

Perturbation-Specific Metrics: Measure More Than Plausible Expression

Begin with the response rather than the final expression profile. Conceptually, the response is perturbed expression minus matched control expression. Evaluate predicted and measured changes on the same scale.

This matters because reproducing basal expression is not the same as predicting an intervention. Moreover, control-referenced correlation alone can still reward shared changes that do not distinguish one knockdown from another.

The 2026 challenge combines six metrics covering perturbation discrimination, expression error, and several aspects of differential expression. Its differential-expression measures examine response direction, agreement between responding-gene sets, and fold-change accuracy. Scores are anchored against a mean-response reference and split-data reproducibility.

For your internal evaluation, retain these complementary views instead of reporting only an aggregate.

A perturbation discrimination score asks whether a prediction matches its own measured response more closely than competing responses. This directly challenges a model that produces essentially the same answer for every target.

Meanwhile, assess effect size alongside direction. Predicting that a gene increases is less useful if the model greatly exaggerates the change. For differentially expressed genes, meaning genes with evidence of a response, examine recovery without ignoring false positives.

Treat measured outcomes as evaluation labels, not model-development inputs. Selecting strongly responding genes from hidden measurements can support a predefined scoring analysis; feeding that selection back into training would leak information.

Also check downstream effects separately from the targeted gene itself. Predicting that the knocked-down gene decreases is not evidence that the model understands the wider response. The challenge excludes each target’s own expression from scoring and uses a broader target-gene exclusion for discrimination.

Finally, inspect results by context and response strength. Compare gains with experimental reproducibility, and estimate uncertainty using biological replicates where available. Thousands of cells from one experiment are not thousands of independent experiments.

Takeaways: Tie Virtual Cell Benchmarks to Experimental Decisions

A useful virtual cell model should outperform realistic baselines on genuinely held-out contexts while recovering intervention-specific biology.

Before a large campaign, define the decision it must improve. If the goal is selecting a small follow-up panel, test whether its ranking enriches for experimentally confirmed responses—not merely whether its expression profiles look realistic.

Then run a blinded pilot in the intended context. Predefine what improvement would justify broader use, including the cost of false positives and missed targets.

The strongest evidence is not a single leaderboard position. It is a reproducible improvement in choosing what to test next. Would your model still earn its experimental budget if the simplest baseline were beside it?