Parse PubMed Data in Python

Why this matters

PubMed is one of the most important sources for biomedical literature. If you work in bioinformatics, clinical NLP, literature mining, or evidence synthesis, being able to search PubMed and turn article records into structured Python data is a very practical skill.

This quickstarter shows how to access PubMed programmatically, understand the returned data structure, and parse useful fields such as PMID, title, journal, abstract, authors, DOI, keywords, and MeSH terms.

Quick summary

In this tutorial, you will:

1. Set up a small Python environment
2. Search PubMed for a query
3. Fetch article records as XML
4. Parse the XML into Python dictionaries
5. Convert the parsed data into a DataFrame and CSV
6. See how to do the same workflow with a Python SDK-style wrapper

Expected outcome: by the end, you will have a reusable script that turns PubMed results into structured tabular data you can analyze downstream.

Requirements

• Basic Python knowledge
• Python 3.10+
• No GPU required
• Libraries: requests, pandas, biopython

---

Step 1: Create the environment

Start with a clean virtual environment and install the packages.

python -m venv .venv
source .venv/bin/activate
pip install requests pandas biopython

If you are on Windows, activate the environment with the PowerShell activation script instead.

---

Step 2: Understand the PubMed access model

For a quickstarter, the simplest PubMed workflow is:

• ESearch to find PMIDs for a query
• EFetch to download full article records
• XML parsing to extract the fields you care about

A practical mental model is:

Search query
   -> ESearch
   -> list of PMIDs
   -> EFetch
   -> PubMed XML
   -> Python dictionaries / DataFrame

Two output shapes are especially useful:

• ESearch JSON for lightweight searching
• EFetch XML for richer article metadata

A fetched record usually contains nested sections like this:

PubmedArticleSet
└── PubmedArticle
    ├── MedlineCitation
    │   ├── PMID
    │   ├── Article
    │   │   ├── ArticleTitle
    │   │   ├── Abstract
    │   │   ├── AuthorList
    │   │   └── Journal
    │   ├── MeshHeadingList
    │   └── KeywordList
    └── PubmedData
        └── ArticleIdList

The important takeaway is that PubMed data is deeply nested and many fields are optional. Your parser should always handle missing values safely.

---

Step 3: Search PubMed with ESearch

In this step, we send a search query and collect a small set of PMIDs.

import os
import requests

BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
EMAIL = "your_email@example.com"
API_KEY = os.getenv("NCBI_API_KEY")

def ncbi_get(endpoint, params):
    params = {
        **params,
        "tool": "pubmed_quickstart",
        "email": EMAIL,
    }
    if API_KEY:
        params["api_key"] = API_KEY

    response = requests.get(f"{BASE_URL}/{endpoint}", params=params, timeout=30)
    response.raise_for_status()
    return response

def search_pubmed(query, retmax=5):
    response = ncbi_get(
        "esearch.fcgi",
        {
            "db": "pubmed",
            "term": query,
            "retmode": "json",
            "retmax": retmax,
            "sort": "pub_date",
        },
    )
    payload = response.json()["esearchresult"]
    pmids = payload["idlist"]
    total_count = int(payload["count"])
    return pmids, total_count

query = "machine learning[Title/Abstract] AND radiology[Title/Abstract] AND 2025[pdat]"
pmids, total = search_pubmed(query, retmax=5)

print("Total matches:", total)
print("PMIDs:", pmids)

What happens here

• db=pubmed tells NCBI which database to search
• term contains the PubMed query
• retmode=json makes the search result easy to parse
• retmax=5 limits the number of returned PMIDs

At this point you only have identifiers, not the full article content.

---

Step 4: Fetch full PubMed records as XML

Now that you have PMIDs, fetch the corresponding article records.

def fetch_pubmed_xml(pmids):
    response = ncbi_get(
        "efetch.fcgi",
        {
            "db": "pubmed",
            "id": ",".join(pmids),
            "retmode": "xml",
        },
    )
    return response.text

xml_text = fetch_pubmed_xml(pmids)
print(xml_text[:1000])

What happens here

• EFetch downloads richer records than ESearch
• retmode=xml is a good choice when you want abstracts, author lists, DOI, keywords, and other nested metadata

For small experiments, loading the XML as a string is fine. For larger jobs, batch the PMIDs and parse incrementally.

---

Step 5: Parse the XML into Python dictionaries

Now we extract structured fields from the XML using the standard library.

from xml.etree import ElementTree as ET

def text_or_none(node, path):
    found = node.find(path)
    if found is not None and found.text:
        return found.text.strip()
    return None

def itertext_or_none(node):
    if node is None:
        return None
    text = "".join(node.itertext()).strip()
    return text or None

def parse_authors(article):
    authors = []
    for author in article.findall("./MedlineCitation/Article/AuthorList/Author"):
        collective = text_or_none(author, "CollectiveName")
        fore = text_or_none(author, "ForeName")
        last = text_or_none(author, "LastName")

        if collective:
            authors.append(collective)
        elif fore or last:
            authors.append(" ".join(part for part in [fore, last] if part))
    return authors

def parse_abstract(article):
    sections = []
    for item in article.findall("./MedlineCitation/Article/Abstract/AbstractText"):
        label = item.attrib.get("Label")
        text = "".join(item.itertext()).strip()
        if not text:
            continue
        sections.append(f"{label}: {text}" if label else text)
    return " ".join(sections) or None

def parse_keywords(article):
    values = []
    for kw in article.findall("./MedlineCitation/KeywordList/Keyword"):
        text = "".join(kw.itertext()).strip()
        if text:
            values.append(text)
    return values

def parse_mesh_terms(article):
    values = []
    for mh in article.findall("./MedlineCitation/MeshHeadingList/MeshHeading"):
        descriptor = text_or_none(mh, "DescriptorName")
        if descriptor:
            values.append(descriptor)
    return values

def parse_pub_date(article):
    pub_date = article.find("./MedlineCitation/Article/Journal/JournalIssue/PubDate")
    if pub_date is None:
        return None

    year = text_or_none(pub_date, "Year")
    month = text_or_none(pub_date, "Month")
    day = text_or_none(pub_date, "Day")
    medline_date = text_or_none(pub_date, "MedlineDate")

    if year:
        return "-".join(part for part in [year, month, day] if part)
    return medline_date

def parse_doi(article):
    for aid in article.findall("./PubmedData/ArticleIdList/ArticleId"):
        if aid.attrib.get("IdType") == "doi":
            return "".join(aid.itertext()).strip()
    return None

def parse_articles(xml_text):
    root = ET.fromstring(xml_text)
    records = []

    for article in root.findall(".//PubmedArticle"):
        title_node = article.find("./MedlineCitation/Article/ArticleTitle")

        record = {
            "pmid": text_or_none(article, "./MedlineCitation/PMID"),
            "title": itertext_or_none(title_node),
            "journal": text_or_none(article, "./MedlineCitation/Article/Journal/Title"),
            "publication_date": parse_pub_date(article),
            "language": text_or_none(article, "./MedlineCitation/Article/Language"),
            "doi": parse_doi(article),
            "authors": parse_authors(article),
            "abstract": parse_abstract(article),
            "keywords": parse_keywords(article),
            "mesh_terms": parse_mesh_terms(article),
        }
        records.append(record)

    return records

records = parse_articles(xml_text)
print(records[0])

What happens here

This parser handles three common PubMed realities:

1. Nested fields such as author lists and abstract sections
2. Optional fields such as DOI or keywords
3. Mixed content where XML tags may appear inside titles or abstracts

That is why itertext() is often safer than reading only .text.

---

Step 6: Convert records into a DataFrame

Once records are parsed into dictionaries, moving into pandas is straightforward.

import pandas as pd

df = pd.DataFrame(records)
print(df.head())
print(df.columns.tolist())

If you want a more analysis-friendly version, flatten the list fields.

df_flat = df.copy()
df_flat["authors"] = df_flat["authors"].apply(lambda x: "; ".join(x) if isinstance(x, list) else None)
df_flat["keywords"] = df_flat["keywords"].apply(lambda x: "; ".join(x) if isinstance(x, list) else None)
df_flat["mesh_terms"] = df_flat["mesh_terms"].apply(lambda x: "; ".join(x) if isinstance(x, list) else None)

print(df_flat[["pmid", "title", "journal", "doi", "authors"]].head())
df_flat.to_csv("pubmed_results.csv", index=False)

Expected output

You should now have a table with columns similar to:

• pmid
• title
• journal
• publication_date
• language
• doi
• authors
• abstract
• keywords
• mesh_terms

This is a very usable starting point for text mining, metadata analysis, or downstream curation.

---

Step 7: Use Biopython as a lightweight SDK

If you prefer a higher-level Python interface, Biopython provides a convenient wrapper around the same Entrez API.

import os
from Bio import Entrez

Entrez.email = "your_email@example.com"
Entrez.tool = "pubmed_quickstart"
Entrez.api_key = os.getenv("NCBI_API_KEY")

query = "single-cell transcriptomics[Title/Abstract]"

with Entrez.esearch(db="pubmed", term=query, retmax=3, sort="pub_date") as handle:
    search_result = Entrez.read(handle)

pmids = search_result["IdList"]
print("PMIDs:", pmids)

with Entrez.efetch(db="pubmed", id=",".join(pmids), retmode="xml") as handle:
    xml_text = handle.read()

records = parse_articles(xml_text)
print(records[0]["title"])

Why use Biopython

Biopython is useful because it:

• wraps common Entrez calls in Python functions
• reduces boilerplate
• integrates nicely with other bioinformatics workflows

A simple rule of thumb:

• use requests when you want full control over raw API calls
• use Biopython when you want a cleaner Pythonic workflow

---

Step 8: Add batching for larger result sets

For anything beyond a tiny demo, fetch records in batches.

import time

def chunked(items, size):
    for i in range(0, len(items), size):
        yield items[i:i + size]

def fetch_many(pmids, batch_size=100, pause_seconds=0.4):
    all_records = []

    for batch in chunked(pmids, batch_size):
        xml_text = fetch_pubmed_xml(batch)
        all_records.extend(parse_articles(xml_text))
        time.sleep(pause_seconds)

    return all_records

large_pmids, total = search_pubmed("cancer genomics[Title/Abstract]", retmax=20)
all_records = fetch_many(large_pmids, batch_size=5)
print("Parsed records:", len(all_records))

What happens here

Batching helps you:

• keep requests manageable
• reduce the chance of timeouts
• stay polite to the service
• build a parser that still works when your query grows

If you regularly process large PubMed result sets, batching should be your default design.

---

Common parsing pitfalls

Before you productionize your script, keep these issues in mind:

Missing abstracts

Not every PubMed record has an abstract. Your parser should allow None values.

Structured abstracts

Some abstracts are split into labeled sections such as Background, Methods, and Results. Joining all AbstractText nodes is usually the safest quick solution.

Variable date formats

Publication dates are not always complete. Sometimes you get year only, or a free-form value instead of year-month-day.

Repeated fields

Authors, keywords, publication types, and MeSH headings are repeated elements. Store them as lists first, then flatten only if needed.

---

Recap

You now have a working quickstart for parsing PubMed data in Python:

• search PubMed with ESearch
• fetch article records with EFetch
• parse nested XML safely
• convert the result into a DataFrame or CSV
• optionally use Biopython as a lightweight SDK wrapper

This pattern is enough for many practical tasks such as literature mining, abstract collection, search dashboards, and metadata analysis.

Further Reading

• Entrez Programming Utilities Help — official overview of the Entrez API family.
• E-utilities In-Depth — parameters, response formats, batching concepts, and request details.
• Download PubMed Data — official guidance for baseline files, updates, and bulk access options.
• PubMed XML Help for Data Providers — useful reference for XML tags and field structure.
• Biopython Tutorial and Entrez Docs — practical examples for Bio.Entrez.

FAQ

1. Should I use XML or JSON for PubMed parsing?

Use JSON for lightweight search metadata and XML for richer article records. In practice, many parsing workflows use JSON for ESearch and XML for EFetch.

2. Why are some DOI, abstract, or keyword fields empty?

Because PubMed records are not perfectly uniform. Some records simply do not contain those fields, and some fields appear only for certain article types or indexing states.

3. How do I handle more than a small demo-sized result set?

Use batching from the start. For very large collections, split queries into smaller date ranges or move to bulk PubMed downloads instead of trying to fetch everything in one pass.