From Prompt to Paper: AI Scientist in‑Silico Loops

Jonathan Alles

EVOBYTE Digital Biology

Introduction

A growing wave of teams is asking a provocative question: what if most of the grind of early‑stage research—scoping ideas, running baselines, iterating on experiments, and drafting the paper—could be automated? The concept of an “AI scientist” pushes beyond coding copilots and literature chatbots. It is a system that loops through ideation, experimentation, and evaluation, improves itself with structured self‑reflection, and then assembles publishable results, figures included. Recent prototypes already chain these steps end‑to‑end for machine learning (ML) studies, producing iterative experiments and full manuscript drafts. While imperfect, they show a path toward research workflows that are faster, more reproducible, and easier to audit.

In this post, we unpack how an AI scientist works, why self‑reflection matters, what it takes to draft responsibly, and where the approach is landing first—especially in ML development and computational biology. Along the way, we’ll ground core ideas in practical examples and a few compact code snippets you can adapt.

What we mean by an “AI scientist”

At heart, an AI scientist is an agentic system built on a large language model (LLM) that can call tools. It doesn’t just predict the next token; it plans, acts, checks, and revises. Two ideas make this tractable in practice.

First, reasoning‑and‑acting patterns let the model interleave thought with tool use. In “ReAct,” for instance, the agent alternates between reasoning traces and actions like querying datasets, running scripts, or searching papers. This gives a scaffold where the model both explains its plan and executes it with external tools, rather than hallucinating results.

Second, deliberate search over candidate thoughts improves the chance of finding better solutions. “Tree of Thoughts” generalizes chain‑of‑thought into a search process over intermediate reasoning steps, pruning weaker branches and exploring stronger ones. In research settings, this looks like proposing several hypotheses or ablation plans, rating them, and then advancing the most promising candidate.

Combine those with a structured pipeline—idea generation, literature grounding, experiment planning, code execution, analysis, and write‑up—and you have a minimal AI scientist. A recent demonstration even coordinates these stages into a full loop that iterates until predefined quality bars are met, then assembles a paper‑style report for review.

The research loop: ideation, experimentation, evaluation

To picture the loop, imagine you’re exploring a new regularization trick for small‑data vision. The AI scientist starts by reading a seed prompt and quickly sketches a handful of candidate ideas. It then grounds those ideas by scanning the literature, uses ranking heuristics to pick one, and produces an executable plan: datasets to use, baselines to compare against, hyperparameters to sweep, and diagnostics to collect. After running the experiments, it evaluates results against predeclared thresholds and surfaces a next step: keep, refine, or discard.

Under the hood, reasoning‑and‑acting enables that choreography. The agent alternates between “thinking” (e.g., planning ablations) and “doing” (e.g., launching training jobs, parsing logs, updating a results table). Because every action is externalized, it’s debuggable and auditable, which is essential when you’re making claims you want to publish.

A compact skeleton looks like this:

# minimal sketch of an AI scientist loop
while not done:
    idea = agent.propose_hypotheses(context)
    grounded = agent.lit_review(idea)              # calls search APIs, builds notes
    plan = agent.design_experiments(grounded)      # datasets, baselines, metrics
    runs = tools.launch_jobs(plan)                  # spins up training/eval
    results = tools.collect_metrics(runs)           # tables, plots, error bars
    critique = agent.self_reflect(idea, plan, results)
    decision = agent.select_next_step(critique)     # keep / refine / drop
    context = agent.update_research_log(decision, results)
    done = agent.met_publication_bar(context)       # e.g., stat. sig., effect size

That last line is more than a footnote. Some pipelines now encode explicit “publication bars” (for example, a minimum delta over strong baselines and ablation evidence), using them to gate whether to iterate or move to write‑up. The point isn’t to replace judgment, but to automate the mechanical parts of the loop and make your criteria explicit.

Self‑reflection: the feedback that compounds progress

The step that often changes outcomes most is self‑reflection. Rather than relying solely on external feedback, the agent generates its own critiques, distills failure patterns, and then re‑plans accordingly. In the “Reflexion” framework, verbal feedback—generated by the model or supplied by the environment—guides the next attempt, functioning like lightweight reinforcement without weight updates. In benchmarks, this simple idea boosted success rates across sequential decision‑making, coding, and reasoning tasks. When you transplant the same principle into research loops, you get fewer repeated mistakes, better experiment selection, and clearer error analysis.

Deliberate search and self‑reflection also complement each other. Tree‑style searches propose diverse options, while reflection prunes dead ends and improves surviving branches. In practice, that might mean the agent writes a brief “lab notebook” after each batch run—what worked, what failed, what to try next—and uses those notes to update prompts and plans before the next iteration.

From results to manuscript: drafting responsibly, and why figures are tricky

Once results clear the predefined bar, the AI scientist assembles a first‑pass manuscript: title and abstract; a methods section that cites code, datasets, and compute; a results narrative with ablations and error analysis; and a discussion that situates the work in the literature. Again, the goal is acceleration and consistency, not replacing authors. Current editorial guidance is clear that AI tools should be disclosed and cannot be authors because they cannot assume responsibility for accuracy and integrity. If you use an AI system in any part of the manuscript or analysis, you must say so and explain how.

Figures are the hardest part to automate well. Many publishers disallow or heavily restrict generative images for scientific figures, and all require strict image integrity. Even when you’re not using generative tools, the safest path is code‑first plotting from source data with full provenance, version pins, and checksums. That way reviewers—and your future self—can regenerate every panel exactly. Major publishers also maintain policies and detection workflows for manipulation; again, transparency and reproducibility are your best safeguards.

A tiny, reproducible pattern helps:

# generate a figure with provenance
import hashlib, json, matplotlib.pyplot as plt, pandas as pd, seaborn as sns

df = pd.read_csv("results.csv")                     # raw metrics per run
fig = sns.lineplot(data=df, x="epoch", y="val_acc", hue="method").get_figure()
meta = {"code_commit": "a1c2e5f", "data_sha256": hashlib.sha256(df.to_csv(index=False).encode()).hexdigest(),
        "env": {"python": "3.11", "lib": {"pandas": "2.2.2", "seaborn": "0.13.2"}}}
fig.savefig("val_curves.png", dpi=300, metadata={"provenance": json.dumps(meta)})

This isn’t fancy, but it’s powerful. You now have an image you can trace to exact inputs and versions, which is the standard you want if an automated system is drafting your figures. If your pipeline also stores the generation script and logs alongside the image, you’re much closer to a clean reproduction package.

Use cases taking shape in ML and computational biology

ML development is a natural early home for AI scientists because the entire loop—from idea to result—is already digital. One representative pipeline automates preliminary viability checks, hyperparameter sweeps, the main agenda, and ablations, then writes a structured report that aligns with common ML conference formats. You still decide which ideas to green‑light and how to frame contributions, but the system reduces turnaround time and raises the floor on experiment hygiene.

The same pattern is bearing fruit in computational biology, where in‑silico workflows are mature and high‑value. Protein structure prediction illustrates how AI can unlock whole new design spaces. The AlphaFold ecosystem now covers well over two hundred million sequences, providing predicted structures and confidence metrics that accelerate hypothesis generation, target triage, and mechanism exploration. An AI scientist can treat these resources as tools in its loop—query a predicted structure, plan a docking experiment, analyze variant effects, and propose the next test. With careful validation, this shortens paths from idea to in‑vitro follow‑up.

There’s also a pragmatic middle ground: use the AI scientist for literature triage, dataset construction, and code scaffolding, while reserving study design, safety checks, and interpretation for humans. In regulated or safety‑critical spaces, that human‑in‑the‑loop framing isn’t optional. Editorial policies and institutional expectations already assume human accountability, and reviewers increasingly ask for explicit descriptions of how AI tools were used.

Finally, don’t forget data and artifact stewardship. The FAIR principles—findable, accessible, interoperable, reusable—are an excellent north star when your pipeline is generating many intermediate assets automatically. If your AI scientist writes out machine‑readable manifests for datasets, code, environment specs, and figures, you get traceability almost for free. That, in turn, makes your work easier to reproduce and extend.

Summary / Takeaways

The AI scientist is not a robot author. It’s an automation layer over the most mechanical parts of research: proposing candidates, running clean experiments, checking thresholds, and assembling an auditable draft. Reason‑and‑act patterns let it use tools responsibly. Search over thoughts broadens exploration. And self‑reflection compresses the learning cycle so each iteration gets smarter than the last.

For ML and computational biology, the benefits are immediate: faster ablations, tidier baselines, clearer provenance, and quicker paths from hunch to in‑silico result. The hard parts are the ones that should be hard: deciding what matters, guarding against shortcuts that threaten integrity, and making figures that are traceable from raw data. Follow publisher guidance on AI use, avoid generative images for scientific figures unless policies allow and you disclose clearly, and pin everything needed to regenerate your results. The research loop gets faster. Your bar for rigor stays high.