↓ Skip to main content

Benchmarking GNoME, NequIP, and Allegro for Au–Pd Formation Energies

·422 words·2 mins
Ismail Can Oguz
Author
Ismail Can Oguz
I build ML-accelerated atomistic workflows (equivariant GNNs + DFT) to discover catalysts and understand surfaces.

Why I built this benchmark
#

Foundation machine-learning interatomic potentials are increasingly used to screen materials before expensive density-functional-theory calculations. However, a model can only be interpreted properly when its predictions are compared with a clearly defined electronic-structure reference.

In this project, I benchmarked released GNoME/WBM formation-energy predictions and three openly available equivariant potentials:

  • NequIP-MP-L
  • NequIP-OAM-L
  • Allegro-OAM-L

The reference values come from the fully relaxed Au–Pd calculations reported by Sluiter, Colinet, and Pasturel in Physical Review B 73, 174204 (2006): DOI: 10.1103/PhysRevB.73.174204.

DFT and training-data context
#

The article used VASP, PAW potentials, and the PW91 generalized-gradient approximation, with full relaxation of volume, cell shape, and internal coordinates. The open ML models are instead connected mainly to PBE/PBE+U training data.

  • NequIP-MP-L was trained on MPTrj, the Materials Project trajectory dataset.
  • NequIP-OAM-L and Allegro-OAM-L were pretrained on OMat24 and fine-tuned on sAlex and MPTrj.
  • The released GNoME values used here are predictions for formula-matched structures from the WBM benchmark, not confirmed matches to the exact L1₂ or L1₀ prototypes.

For metallic Au–Pd, Hubbard \(U\) is not relevant, so the Materials Project branch is effectively ordinary PBE.

Main results
#

PhasePW91-DFTGNoME/WBMNequIP-MP-LAllegro-OAM-LNequIP-OAM-L
Au3Pd−97.00−78.93−83.30−82.26−75.08
AuPd−104.00−89.48−102.92−83.04−75.93
AuPd3−65.00—−54.65−52.88−51.50

Across the three prototype-matched compositions, the mean absolute deviations from the published DFT values are:

  • NequIP-MP-L: 8.38 meV/atom
  • Allegro-OAM-L: 15.94 meV/atom
  • NequIP-OAM-L: 21.16 meV/atom

The closest agreement occurs for equiatomic AuPd: NequIP-MP-L predicts −102.92 meV/atom, compared with the article’s L1₀ value of −104 meV/atom.

Interpretation
#

For this narrow ordered-alloy benchmark, the MPTrj-only model transfers particularly well to the published Au–Pd energetics. The broader OMat24-based models systematically predict weaker ordering for these selected structures.

This is not a universal ranking of the models. OAM models cover a much broader chemical and configurational space, while this comparison focuses on only three ordered metallic phases. Differences can also arise from PW91 versus PBE, pseudopotential choices, relaxation constraints, and heterogeneous training-data conventions.

The released GNoME/WBM values reproduce the negative formation-energy trend, but they should be interpreted as composition-level comparisons because the corresponding WBM structures have not been confirmed to match the exact article prototypes.

Reproducible workflow
#

The GitHub repository contains the Colab notebook, extracted DFT reference table, model-comparison plots, CSV outputs, and discussion of the DFT provenance:

Open the complete Au–Pd benchmark on GitHub

For the ordered Au–Pd structures tested here, NequIP-MP-L is the closest open model to the prototype-matched PW91-GGA literature values, while the broader OMat24-based models predict less-negative formation energies.