IsingFormer: Augmenting Parallel Tempering With Learned Proposals

arXiv:2509.23043v2 Announce Type: replace-cross Abstract: Generative models have been extensively used to accelerate MCMC mixing for sampling and optimization, but their effective integration with standard MCMC remains an open question. Here, we introduce a global proposal move in which finite-temperature configurations from an external generator are used as proposals within Parallel Tempering (PT). We examine a specific generator, IsingFormer, a Transformer trained on long-run MCMC configurations intended to approximate equilibrium distributions, and call the resulting framework Transformer-Augmented Parallel Tempering (TAPT). The IsingFormer exhibits two useful capabilities: interpolation to untrained $\beta$ values and conditional completion under clamped settings absent from training. On 3D spin-glass instances, TAPT reaches substantially lower residual energies than standard PT in fewer Monte Carlo sweeps. On integer factorization, we show how a structured problem encoding can amortize the training cost by training IsingFormer once and reusing the same model across target products that were not imposed during training. Finally, in a scaling study that uses long-run MCMC configurations as proposals, with proposal generation excluded from the timing, TAPT reduces the fitted time-to-solution exponent by approximately $33\%$ relative to PT over the tested problem sizes.