A researcher's notebook: how I built a two-part toolchain for my transition-metal chalcogenide work — an AI-powered PDF library organiser that ingests the literature, and a physics-constrained ML pipeline (v4.0) that fetches Materials Project data, engineers GKA/Mott/ZSA descriptors, trains a stacking ensemble, and explains predictions with SHAP.
PDF Library Organiser + Gemini AI
MₓCᵧ Pipeline v4.0
Materials Project + empirical DFT
Stacking RF + SHAP explainability
My DFT work on transition-metal chalcogenides (MₓCᵧ: M = Fe, Ni, Co, Mn, Cr, Ti; C = S, Se, Te) has produced a growing library of first-principles results alongside an even larger pile of literature PDFs. This post documents the two Python tools I built to handle both: a PDF library organiser that classifies hundreds of papers automatically with Gemini AI, and the MₓCᵧ Pipeline v4.0 that feeds those papers — together with Materials Project data — into a physics-constrained ML model for band gap, magnetisation, and magnetic ordering prediction.
This pipeline grew directly out of my doctoral work (Reggad, Université de Sidi-Bel-Abbès) and subsequent publications (Reggad et al., Physica B 526, 2017, 89–95). The goal is not to replace DFT — it is to extract cross-compound trends that single calculations cannot reveal, and to build a predictor that flags which MₓCᵧ candidates are worth computing next.
Part 1 — PDF Library Organiser with Gemini AI
Before any ML pipeline can ingest the literature, the literature needs to be
organised. My G:\MyResearch\AllPapers folder had accumulated hundreds
of PDFs with meaningless filenames like 1-s2.0-S0921452617302386.pdf.
The organiser solves this in three steps: extract real titles from inside each PDF,
rename them, classify them into four categories using Gemini AI, move them to
matching subfolders, and export a colour-coded Excel report.
Architecture
Four output categories
| Folder | Category | Description | Excel colour |
|---|---|---|---|
4_Subject_Papers | subject_paper | Research on MₓCᵧ: band gap, magnetic properties, DFT calculations | Green |
2_Physics_Books | physics_book | Textbooks: solid-state physics, quantum mechanics, thermodynamics | Yellow |
3_Manuscripts | manuscript | PhD theses, dissertations, memoires de magistère | Orange |
1_Out_of_Subject | out_of_subject | Unrelated: biology, ML, economics, etc. | Red |
Title extraction strategy
The script uses a three-level fallback: (1) PDF metadata title field
if it is non-trivial (length > 5, contains spaces), (2) first meaningful line
from page 1 text — skipping lines that match journal headers, DOIs, dates, or
single words, (3) the original filename stem as last resort.
import pypdf
reader = pypdf.PdfReader(str(pdf_path))
# 1. Metadata field
meta = reader.metadata
if meta and getattr(meta, "title", None):
t = meta.title.strip()
if len(t) > 5 and not re.fullmatch(r"[\w\-]+", t):
return t
# 2. First meaningful line of page 1
lines = [l.strip() for l in reader.pages[0].extract_text().splitlines() if l.strip()]
for line in lines[:15]:
skip = re.search(
r"^(abstract|doi|http|journal|vol\.\s*\d|received|copyright)",
line, re.IGNORECASE)
if not skip and 8 <= len(line) <= 150:
return line
return pdf_path.stem # 3. Filename fallback
Gemini AI classification prompt
Classification uses Gemini 1.5 Flash — free at 1500 requests/day, no GPU required. The prompt sends title + abstract extract (max 600 chars) and asks for exactly one of four category keys. A keyword-based fallback activates automatically when the API quota is hit or returns an unexpected response.
PROMPT = """Classify the document into EXACTLY ONE of:
subject_paper — MxCy compounds, DFT, band gap, magnetic properties
physics_book — textbook on solid-state physics, QM, thermodynamics
manuscript — PhD thesis, dissertation, memoire de magistère
out_of_subject — anything else
Respond with ONLY the category key."""
# Rate limit: 4.5 s between calls → stays under 15 RPM free tier
RATE_LIMIT_SEC = 4.5
Running the script
python pdf_organizer.py "G:\MyResearch\AllPapers"
# Or just double-click and enter path when prompted
python pdf_organizer.py
# Output structure created automatically:
# G:\MyResearch\AllPapers\
# ├── 4_Subject_Papers\
# ├── 3_Manuscripts\
# ├── 2_Physics_Books\
# ├── 1_Out_of_Subject\
# └── PDF_Library_Report.xlsx
If you hard-code a path in a Python script, always use a raw string:
r"G:\MyResearch\AllPapers" not "G:\MyResearch\AllPapers".
The backslash before M, A etc. creates escape sequences
(\M is not valid, causing a SyntaxError).
Raw strings treat backslashes literally.
Part 2 — MₓCᵧ Pipeline v4.0: Physics-Constrained ML
The pipeline is a 13-cell Jupyter notebook that runs identically on Windows (local), Google Colab, and Kaggle. Version 4.0 introduced three major changes over v3.9: the Materials Project hull filter was raised to 0.15 eV/atom to include more metastable polymorphs (>300 training entries), magnetic ordering (NM/AFM/FM) was added as a third classification target, and an explicit 80/20 train/test split replaced pure cross-validation for more honest reporting.
Pipeline overview (13 cells)
| Cell | Purpose | Key outputs |
|---|---|---|
| 1 — Setup | Install packages, detect platform (local/Colab/Kaggle), restart kernel once | PLATFORM, output dir |
| 2 — Imports | All imports — run after kernel restart | All libraries ready |
| 3 — Platform & API Key | Re-detect platform, set MP API key, configure output paths | OUTPUT_DIR, API_KEY |
| 4 — Atomic Constants | Physics lookup table: n_d, r_cov, χ, IE, U_Hubbard, r_ionic for each element | ATOMIC_DATA |
| 5 — Physics Formulas | Bond length, bandwidth W, crystal field Δ, Mott criterion, ZSA classifier, GKA flag | Feature engineering functions |
| 6 — Data Fetch | Materials Project API query — band gap, magnetisation, ordering, E-hull | real_dict (~300 entries) |
| 7 — Train | Stacking RF ensemble — 80/20 split, 3 targets (gap, mag, ordering) | Model bundle, CV + test metrics |
| 7b — Compare | Predicted vs real scatter plots, Bland-Altman, confusion matrix | Comparison figures |
| 8 — Fine-Tune | Blend empirical physics corrections into model | Tuned model bundle |
| 9 — Predict | Single-compound prediction API | predict_compound(M, C, structure) |
| 10 — SHAP | TreeExplainer on RF gap sub-model, beeswarm summary plot | Feature importance ranking |
| 11 — Heatmap | Periodic-table style heatmap of all MₓCᵧ predictions | Publication-ready figure |
| 12 — Excel Export | 5-worksheet workbook: dataset, features, CV results, predictions, SHAP | MxCy_v4_Full_Results.xlsx |
Physics-informed feature table
| Feature | Physical meaning | Encodes |
|---|---|---|
| n_d | d-electron count | Crystal field occupancy, Hund's rule coupling |
| Δ (crystal field) | t₂g–eₘ splitting | Ligand field strength → gap vs metallic d-band |
| U/W | Hubbard U / bandwidth W | Mott criterion — correlated insulator boundary |
| ΔEN | Electronegativity difference | Ionic/covalent character → ZSA charge-transfer |
| GKA flag | Goodenough–Kanamori–Anderson | 180° superexchange → AFM if half-filled d-shell |
| bond length | M–C ionic radii sum × structure factor | Overlap integral → bandwidth |
| chalcogen period | S=3, Se=4, Te=5 | Covalency trend: S < Se < Te → decreasing gap |
Cell 4 — Atomic constants table (excerpt)
ATOMIC_DATA = {
'Ti': [2, 1.40, 1.54, 6.82, 3.0, 0.86],
'V': [3, 1.35, 1.63, 6.74, 3.5, 0.79],
'Cr': [4, 1.28, 1.66, 6.77, 3.5, 0.87],
'Mn': [5, 1.27, 1.55, 7.43, 4.0, 0.97],
'Fe': [6, 1.26, 1.83, 7.90, 4.5, 0.92],
'Co': [7, 1.25, 1.88, 7.88, 5.0, 0.88],
'Ni': [8, 1.24, 1.91, 7.64, 5.5, 0.83],
'Cu': [9, 1.28, 1.90, 7.73, 7.0, 0.87],
}
# Sources: r_cov (Alvarez 2008), chi (Pauling scale), IE (NIST),
# U_Hubbard (Dudarev scheme, typical DFT+U values),
# r_ionic (Shannon 1976, 6-coordinate high-spin)
Cell 6 — Fetching data from Materials Project (v4.0)
# v4.0: hull raised to 0.15 eV/atom → more polymorphs → ~300+ entries
CHEMSYS = [f"{m}-{c}" for m in ["Fe","Ni","Co","Mn","Cr","Ti"]
for c in ["S","Se","Te"]]
with MPRester() as mpr:
docs = mpr.materials.summary.search(
chemsys=CHEMSYS,
energy_above_hull=(0, 0.15), # raised from 0.10 in v3.9
fields=["material_id", "formula_pretty",
"band_gap", "ordering", # NEW in v4.0
"total_magnetization", "energy_above_hull",
"crystal_system", "spacegroup_number"]
)
Cell 7 — Training: three targets simultaneously
TARGETS = {
'band_gap_eV': 'regression', # eV — continuous
'magnetization_muB': 'regression', # μB/f.u. — continuous
'ordering_label': 'classification', # NM=0 / AFM=1 / FM=2
}
# Stacking ensemble: two RF regressors + one RF classifier
# rf_gap trained on log1p(band_gap) to handle the heavy zero-gap spike
# rf_mag trained on log1p(|magnetization|) then sign restored
# rf_ord trained on 3-class ordering labels
# 80/20 stratified split — ensures ≥250 train rows, ≥50 test rows
X_train, X_test, y_train, y_test = train_test_split(
X_real, Y_real, test_size=0.20, random_state=42)
Cell 9 — Single-compound prediction API
result = predict_compound(M='Ni', C='S', structure='zincblende')
# Example output:
# ┌─────────────────────────────────────────────┐
# │ NiS (zincblende) │
# │ Band gap : 0.43 eV (semiconductor) │
# │ Magnetisation : 1.82 μB/f.u. │
# │ Magnetic order : AFM │
# │ Half-metal : No │
# └─────────────────────────────────────────────┘
Cell 10 — SHAP analysis
# Extract RF gap sub-model from v4.0 bundle dict
bundle = joblib.load(model_path)
gap_model = bundle['rf_gap'] # (legacy: bundle.estimators_[0])
explainer = shap.TreeExplainer(gap_model)
shap_vals = explainer.shap_values(X_real[FEATURES])
shap.summary_plot(shap_vals, X_real[FEATURES],
feature_names=FEATURES, plot_type='dot')
Lessons Learned — Common Issues and Fixes
| Issue | Root cause | Fix applied |
|---|---|---|
| Windows path SyntaxError | PAPER_DIR = "G:\MyResearch\AllPapers" — backslash creates escape sequences |
Use raw string: r"G:\MyResearch\AllPapers" |
| G:\ not found on Colab | Colab is a Linux server — has no access to local Windows drives | Use Google Drive path: /content/drive/MyDrive/AllPapers after mounting |
| Drive not mounted on Colab | from google.colab import drive; drive.mount() was missing |
Auto-mount only on Colab: if PLATFORM == 'colab': drive.mount('/content/drive') |
| Gemini rate limit (429) | Free tier: 15 RPM. Hitting limit on large libraries | 4.5 s sleep between calls + automatic 30 s retry on 429 |
| predict_compound KeyError | v3.8 model bundle saved as MultiOutputRegressor; v4.0 expects dict with rf_gap key |
Added compatibility branch: isinstance(bundle, dict) check |
| SHAP on wrong sub-model | Calling TreeExplainer on full MultiOutputRegressor instead of inner RF | Extract with bundle['rf_gap'] before passing to TreeExplainer |
| Magnetic ordering missing | v3.9 did not fetch the ordering field from MP API |
Added to v4.0 field list; labels mapped: NM=0, AFM/FiM=1, FM=2 |
Both scripts are available on GitHub:
🐍 pdf_organizer.py
— standalone Windows Python script, no Jupyter required.
📓 MxCy_pipeline_v4_0.ipynb
— runs on local Python, Colab, and Kaggle.
My name is Dr Abderrahmane Reggad. I'm a 54 year old and I am a researcher in materials' sciences.
0 Comments