Converting the v6.0 Jupyter notebook into a standalone, cron-ready Python script — with a parallel PDF-mining stage rebuilt from the ground up to handle a 1200-paper Google Drive library without stalling or flooding the console.
Standalone .py, not Jupyter
Cron/scheduler-ready, unattended
1200+ papers, parallel mining
Identical to notebook v6.0
The previous post in this series walked through MxCy_pipeline_v6_0.ipynb
cell by cell. This post documents a different artifact built from the same logic:
MxCy_pipeline_v6_0.py — a standalone script meant to run unattended,
on a schedule, against a literature library that has now grown past 1200 PDFs
synced from Google Drive. The ML core (27 features, domain-split gap ensembles,
bootstrap CI) is unchanged from the notebook. What changed is everything around
it: configuration, logging, and — most significantly — the PDF mining stage,
which needed a full rewrite to stay usable at this scale.
A Jupyter notebook assumes a human is watching — reading cell output, deciding when to re-run a cell, manually copying accepted rows into a dictionary. None of that scales to a 1200-paper library that grows weekly. A script can be triggered by cron, logs everything to a timestamped file instead of disappearing when the kernel restarts, and is configured entirely through environment variables — no notebook cell needs editing between runs.
1. What Stayed Identical
Every piece of physics and every ML decision from the notebook carries over unchanged: the 27-feature set (including the five v6.0 additions — zsa_delta, soc_sq, stoner_split, d_fill_ratio, goldschmidt_t), the 23-metal × 4-chalcogen × 9-stoichiometry combinatorial grid, the domain-split gap ensembles (oxide / non-oxide / all, with isotonic calibration), the ordering-label-feeds- magnetisation fix, and the 50-resample bootstrap with 90% CI coverage checking. None of that needed to change — only the scaffolding around it.
| Component | Notebook | Script |
|---|---|---|
| 27-feature engineering | Cell 5 | compute_all_features_v6() — identical |
| Materials Project fetch | Cell 7 | fetch_real_data_v6() — identical |
| Full dataset builder | Cell 8 | build_full_dataset_v6() — identical |
| ML training (gap/mag/ordering) | Cell 9 | train_ml_v6() — identical |
| SHAP, Excel export, plots | Cells 10-13 | Same functions, called from main() |
| PDF literature mining | Cell 6b — sequential, console print | Rebuilt — parallel, file-based, scale-aware |
2. Configuration — Environment Variables Instead of Notebook Cells
Where the notebook hardcoded API_KEY = "..." and
PAPER_DIR = r"G:\..." directly in a cell, the script reads
everything from environment variables, with sane defaults. This is what
makes unattended scheduling possible — the same script file runs identically
whether triggered by cron, a Windows Task Scheduler job, or by hand.
MP_API_KEY = os.environ.get("MP_API_KEY", "")
OUTPUT_DIR = os.environ.get("MXCY_OUTPUT_DIR", "./MxCy_outputs_v6")
PAPER_DIR = os.environ.get("MXCY_PAPER_DIR", "")
PAPER_DB_CSV_PATH = os.environ.get("MXCY_PAPER_CSV", "")
# PDF mining performance tuning
PDF_MAX_WORKERS = int(os.environ.get("MXCY_PDF_WORKERS", "8"))
PDF_BATCH_SIZE = int(os.environ.get("MXCY_PDF_BATCH", "50"))
PDF_TIMEOUT_SEC = int(os.environ.get("MXCY_PDF_TIMEOUT", "45"))
PDF_SKIP_LARGER_THAN_MB = float(os.environ.get("MXCY_PDF_MAX_MB", "60"))
export MP_API_KEY="your_key_here"
export MXCY_PAPER_DIR="/home/you/GoogleDrive/AllPapers"
export MXCY_PDF_WORKERS=12
python MxCy_pipeline_v6_0.py
The script reads PDFs through the normal filesystem — it does not call
the Google Drive web API. Your 1200-paper folder needs to already be
mounted or synced locally first: Google Drive for desktop (Windows/Mac),
or rclone mount / rclone sync (Linux). Once
that's set up, MXCY_PAPER_DIR just points at the resulting
ordinary folder path.
3. Logging — stdout + Timestamped File, Not Cell Output
Notebook cell output vanishes the moment the kernel restarts. A scheduled script needs a permanent record of what happened on each run — especially important for a multi-hour PDF mining pass against 1200 files, where you want to know exactly how far it got if something interrupts it.
OUTPUT_DIR, f"mxcy_run_{datetime.now():%Y%m%d_%H%M%S}.log"
)
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-7s %(message)s",
handlers=[
logging.StreamHandler(sys.stdout), # still visible if run interactively
logging.FileHandler(_log_path, encoding="utf-8"),
]
)
Every print() call from the notebook's training and export cells
became a log.info() call — same information, but now permanently
recorded with a timestamp, and easy to grep for errors after an
unattended overnight run.
4. The Rebuilt PDF Mining Stage — Designed for 1200+ Files
This is where the script genuinely diverges from a straight notebook-to-.py conversion. The notebook's Cell 6b scanned one PDF at a time, printed the first few matches per file to the console, and relied on the researcher manually retyping accepted rows. None of that works at 1200 papers.
| Concern at 1200+ papers | Notebook Cell 6b | Script |
|---|---|---|
| Total runtime | Sequential — one file fully processed before the next starts | ThreadPoolExecutor with PDF_MAX_WORKERS=8 — files processed concurrently |
| Progress visibility | Per-file print; no sense of overall completion for a long run | Logged every PDF_BATCH_SIZE files with completion %, rate (files/s), and ETA |
| Subfolder organisation | Top-level glob only — misses papers sorted into subfolders |
Path.rglob("*.pdf") — recursive, finds PDFs at any depth (e.g. inside pdf_organizer.py's category subfolders) |
| One bad PDF stalling everything | No timeout — a corrupt or huge scanned PDF could hang the whole notebook | PDF_TIMEOUT_SEC per-file timeout on the pdftotext call; oversized files (PDF_SKIP_LARGER_THAN_MB) skipped before extraction is attempted |
| Reviewing results | First handful of matches printed to console — unusable for 1200 papers' worth of output | Every candidate row written to pdf_mining_candidates_v6.csv; a JSON summary with per-run stats and errors |
| Wrapped table rows | Only scans the same line as the formula match | Falls back to checking the next line too — catches formula-in-one-line, numbers-wrapped-to-next-line table layouts common in journal PDFs |
The parallel scanning core
pdf_files = sorted(Path(paper_dir).rglob("*.pdf"))
n_total = len(pdf_files)
log.info("Found %d PDF file(s) ... mining with %d parallel workers",
n_total, PDF_MAX_WORKERS)
all_records, n_done = [], 0
t_start = time.time()
with ThreadPoolExecutor(max_workers=PDF_MAX_WORKERS) as executor:
futures = {executor.submit(_scan_one_pdf, p): p for p in pdf_files}
for future in as_completed(futures):
pdf_path, recs, err = future.result()
n_done += 1
if recs: all_records.extend(recs)
if n_done % PDF_BATCH_SIZE == 0 or n_done == n_total:
rate = n_done / (time.time() - t_start)
eta = (n_total - n_done) / rate
log.info(" [%d/%d] %.1f%% | %d rows | ETA %.0fs",
n_done, n_total, 100*n_done/n_total, len(all_records), eta)
pd.DataFrame(all_records).to_csv(
os.path.join(OUTPUT_DIR, "pdf_mining_candidates_v6.csv"), index=False)
return all_records
Safety gate: candidates are never auto-trusted
Whether mining 5 papers or 1200, the script deliberately stops short of
feeding extracted values directly into training. Every candidate row goes
to pdf_mining_candidates_v6.csv for human review first.
Pass 1: run the script with MXCY_PAPER_DIR
set. It mines all 1200 PDFs and writes every candidate to the CSV —
nothing is used for training yet.
Review: open the CSV, delete false positives (a table
caption like "Table 2, page 47" can superficially match the regex), keep
confirmed rows.
Pass 2: set MXCY_PAPER_CSV to the cleaned
file and re-run. Now those values merge into the training set — but only
where Materials Project has no entry for that exact compound and
structure; MP-DFT values always take precedence where both exist.
5. Validated End to End
Before relying on this against a real 1200-paper library, the script was tested with five synthetic PDFs containing known FeS₂ and NiO data lines. Every test PDF was correctly mined in parallel, with the exact expected band gap and magnetisation values extracted:
22:06:25 INFO Found 5 PDF file(s) under /tmp/test_papers (recursive scan)
22:06:25 INFO Mining with 8 parallel workers, batch report every 50 files
22:06:25 INFO [5/5] 100.0% | 10 candidate rows so far | 59.3 files/s | ETA 0s
22:06:25 INFO PDF mining complete: 5 files in 0.1s (59.0 files/s)
22:06:25 INFO Files with >=1 candidate row: 5 / 5
22:06:25 INFO Total candidate rows extracted: 10
22:06:25 INFO Candidate rows written to: ./MxCy_outputs_v6/pdf_mining_candidates_v6.csv
With MP_API_KEY unset, the dataset-build and graceful-skip logic
was also confirmed: the script still assembles the full 828-compound
combinatorial grid (23 metals × 4 chalcogens × 9 stoichiometries) using
empirical physics formulas, logs a clear warning that training will be
skipped without real reference data, and exits cleanly rather than crashing.
6. Output Files — Same as the Notebook, Plus a Run Log
| File | Contents |
|---|---|
mxcy_run_<timestamp>.log | Full run log — new in the script version |
pdf_mining_candidates_v6.csv | Every candidate row extracted from the literature, for review |
pdf_mining_summary_v6.json | Per-run stats: files scanned, hit rate, errors, elapsed time |
MxCy_dataset_v6_raw.csv | Full 828-compound feature dataset |
MxCy_model_bundle_v6.pkl | Trained gap/magnetisation/ordering models |
MxCy_v6_Full_Results.xlsx | 6-sheet workbook — same structure as the notebook's Cell 12 |
shap_gap_v6.png, MxCy_v6_summary.png | SHAP and 6-panel results dashboard |
MxCy_results_v6.zip | Everything bundled — new in the script version, for easy retrieval after an unattended run |
My name is Dr Abderrahmane Reggad. I'm a 54 year old and I am a researcher in materials' sciences.
0 Comments