The v5.1 analysis report claimed R²(gap) = 0.994 and R²(magnetization) = 0.984 on cross-validation. The actual CV results sheet in the same results bundle says 0.353 and −0.446. This post is about finding out why, and what's actually true.
Every post so far in this series has been hypothesis and method — what I expect, how I'd test it, what code would run the test. This one is different: it's the first post built entirely from an actual pipeline output, the v5.1 results bundle. And the first thing that output taught me wasn't a physics result. It was that two documents from the same run disagreed with each other, and figuring out why turned out to be the more useful finding.
Two numbers that don't agree
The analysis report's header states R²(band gap, CV) = 0.994 ± 0.004 and R²(magnetization, CV) = 0.984 ± 0.008. The MxCy_v5_Full_Results.xlsx file from the same v5.1 run has a sheet called “ML CV Results” with the numbers that were actually computed during training:
| Metric | Reported in PDF | Actual CV results sheet |
|---|---|---|
| R²(band gap, CV) | 0.994 ± 0.004 | 0.353 ± 0.245 |
| R²(magnetization, CV) | 0.984 ± 0.008 | −0.446 ± 0.816 |
| R²(band gap, held-out test) | — | 0.563 (MAE 0.115 eV) |
| R²(magnetization, held-out test) | — | −0.754 (MAE 2.50 μB) |
| Ordering accuracy, held-out test | — | 78.6% |
A negative R² means the model does worse than just predicting the mean every time. That's not a subtle gap between two reasonable estimates — it's two different stories about whether the magnetization model works at all.
Tracing it to the two-lane protocol
My own ML_Guide_SelectAndTrain.docx already names the two most likely explanations, written before I'd even found the discrepancy. One: reporting training R² instead of CV R² — training R² for a random forest is expected to land around 0.95+ regardless of whether the model generalizes, and the guide calls presenting it as the real result a serious mistake. Two: mixing the two data pools the guide is explicit about keeping separate — real Materials Project rows (Is_Real = True) for training, and empirical-formula rows (Is_Real = False) for prediction on hypothetical compounds only, never both in the same loop.
Checking the actual dataset sheet settles which one: of 828 total compound rows, only 282 are Is_Real = True; the other 546 are empirical/extrapolated. The CV results sheet's own training and test counts — 226 + 56 — add up to exactly 282. So the honest CV numbers were computed correctly, on the real subset only. Whatever produced the PDF's 0.994/0.984 likely wasn't.
The clearest evidence is in the dataset statistics themselves. Splitting the 828 rows by the Is_Real flag:
| Subset | n | Mean band gap | Fraction with gap = 0 (metallic) |
|---|---|---|---|
Is_Real = True (actual DFT/MP data) | 282 | 0.256 eV | 71.6% |
Is_Real = False (empirical/extrapolated) | 546 | 6.327 eV | 0.9% |
| All rows combined | 828 | 4.260 eV | — |
The real chalcogenides in this dataset are overwhelmingly metallic — nearly three in four have an exactly-zero measured gap. The analysis report's dataset overview says the opposite: “Insulators (Eg > 2.5 eV) 288/324 (89%).” That 89% insulating picture is real, but it's a description of the model's own extrapolated predictions on hypothetical compounds, not of the measured dataset. Whoever or whatever generated the report's overview table appears to have summarized all 828 rows together rather than isolating Is_Real = True first — the exact leakage pattern my own guide warns about, just showing up in a summary table rather than a training loop.
Where the gap model gets real signal
None of this means the band gap model is worthless — R² = 0.56 on a genuine held-out test, with MAE under 0.12 eV, is a real, moderate signal on a small and noisy dataset. The SHAP importances from the actual trained model (computed on the real 282, per the saved mxcy_base_model_v51.pkl) line up with the physical hypotheses in my own feature rationale table:
exchange_corr_ratio and U_W_ratio top the list, with Hubbard_U and chi_diff close behind — the Mott criterion and ionicity descriptors this whole series has been building around, ranking exactly where the physics says they should. bond_angle_deg sits near the bottom for the gap model, which is expected — the GKA angle is a magnetic-ordering descriptor, not a gap one, and seeing it rank low here is itself a small confirmation that the model isn't just picking up noise.
Where it breaks: magnetization
The held-out test predictions explain the negative magnetization R² directly. A few real examples from the Prediction_vs_Real sheet:
| Compound | Real mag (μB) | Predicted (μB) | Real ordering | Predicted ordering |
|---|---|---|---|---|
| MnO₂ | 0 | 15.94 | AFM | FM |
| FeO | 0 | 12.99 | AFM | FM |
| NbSe₂ | 1.81 | 0 | FM | NM |
| V₃Te₄ | 7.01 | 7.96 | FM | AFM |
The MnO₂ and FeO errors are the most informative ones, because Mn and Fe sit in exactly the d⁴–d⁶ window the GKA half-filling rule (post 9, and rule 5 in the analysis report) singles out as the AFM window. There's also a class-imbalance problem the 78.6% ordering accuracy hides: of the 282 real compounds, 191 are non-magnetic, 67 are FM, and only 24 are AFM. A classifier that always predicted “non-magnetic” would already score 67.7% — so 78.6% is real improvement, but it's a much smaller improvement than it sounds, and it's being measured on the class with the fewest real examples and the most physics riding on it.
Testing post 13's hypothesis on the only data that's trustworthy
Post 13 predicted a roughly monotonic drift — lower U/W and more charge-transfer character going from 3d to 4d to 5d. Restricting strictly to the 282 real compounds and grouping by row gives an actual answer, not a schematic one:
| Row | n (real) | Mean U/W | Std | Mean gap (eV) |
|---|---|---|---|---|
| 3d | 164 | 7.53 | 6.37 | 0.257 |
| 4d | 73 | 10.41 | 18.37 | 0.185 |
| 5d | 45 | 3.59 | 2.18 | 0.369 |
That's not the monotonic drift post 13 expected — 4d has the highest mean U/W of the three rows, not the lowest. But the standard deviation for 4d (18.4, larger than the mean itself) is the real story: a row-level mean is being dragged around by a small number of extreme U/W compounds, most likely narrow-band d⁸⁰-adjacent cases (Ag, Pd) where W is close to zero and U/W blows up. Post 13's hypothesis isn't confirmed or refuted by this — it's the wrong level of aggregation to test it at. The next version of that test needs per-metal medians, not per-row means.
What this means for the rule bank
The per-structure formulas, the Hund-Stoner magnetization fit, and most of the twelve master rules in the analysis report were very likely fit on the full 828-row set, or on the model's own smooth extrapolations rather than the noisy real measurements — which would explain both the suspiciously clean R² = 0.95–0.99 per-structure fits and why they don't match the honestly-validated CV numbers. None of those rules should be treated as confirmed until they're re-derived on Is_Real = True rows only, and the magnetization-side rules specifically can't be trusted at all while the underlying model's CV R² is negative.
# refit on Is_Real==True only, compare against the all-data version
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
df = pd.read_excel("MxCy_v5_Full_Results.xlsx", sheet_name="Dataset v5")
def refit_and_compare(features, target="band_gap_eV"):
real = df[df["Is_Real"] == True].dropna(subset=features + [target])
all_rows = df.dropna(subset=features + [target])
for label, sub in [("Is_Real only", real), ("All rows (leaky)", all_rows)]:
m = LinearRegression().fit(sub[features], sub[target])
r2 = r2_score(sub[target], m.predict(sub[features]))
print(ff"{label}: n={len(sub)}, R\u00b2={r2:.3f}, coefs={dict(zip(features, m.coef_))}")
refit_and_compare(["U_W_ratio", "chi_diff"])
If the “Is_Real only” line and the “All rows” line print similar R² and similar coefficients, that specific rule survives contact with real data. If they diverge the way the headline CV numbers did, the rule needs to be rewritten using only the real-data fit, with the smaller, noisier R² reported honestly — not the larger one.
Concrete next steps
Four things before any more “master rules” get written: re-derive every existing rule with the function above, restricted to real data, and keep only what survives. Diagnose the magnetization model specifically — my own guide's prescribed fix for unstable CV is splitting the dataset (e.g. by gap = 0 vs gap > 0, or trying GroupKFold by element to test generalization to unseen metals rather than random splits). Treat the AFM class's 24 real examples as the actual bottleneck behind several rules in the report, not a detail — more real Cr/Mn/Fe data would do more for the magnetic side of this project than any amount of additional feature engineering. And separate, going forward, what's a validated finding from what's a screening prediction on the 546 hypothetical compounds — both are useful, but only one of them is currently a fact about real materials.
My name is Dr Abderrahmane Reggad. I'm a 54 year old and I am a researcher in materials' sciences.
0 Comments