Improved Parallelization of the SIESTA Magneto-hydrodynamic Equilibrium Code Using Cyclic Reduction
Abstract
SIESTA is a parallel three-dimensional plasma equilibrium code capable of resolving magnetic islands at high spatial resolutions for toroidal plasmas. Originally designed to exploit small-scale parallelism, SIESTA has now been scaled to execute efficiently over several thousands of processors P. This scaling improvement was accomplished with minimal intrusion to the execution flow of the original version. First, the efficiency of the iterative solutions was improved by integrating the parallel tridiagonal block solver code BCYCLIC. Krylov-space generation in GMRES was then accelerated using a customized parallel matrix-vector multiplication algorithm. Novel parallel Hessian generation algorithms were integrated and memory access latencies were dramatically reduced through loop nest optimizations and data layout rearrangement. These optimizations sped up equilibria calculations by factors of 30-50. It is possible to compute solutions with granularity $N/P$ near unity on extremely fine radial meshes ($N extgreater 1024$ points). Grid separation in SIESTA, which manifests itself primarily in the resonant components of the pressure far from rational surfaces, is strongly suppressed by finer meshes. Large problem sizes of up to 300K simultaneous non-linear coupled equations have been solved on the NERSC supercomputers.
Kalyan Perumalla
R&D Manager
As a Federal Program Manager in Advanced Scientific Computing Research at the U.S. Dept. of Energy, Office of Science, Kalyan Perumalla manages a $100-million R&D portfolio covering AI, HPC, Quantum, SciDAC, and Basic Computer Science. He previously led advanced research and development as Distinguished Research Staff Member at the Oak Ridge National Laboratory (ORNL), where he spent 17 years developing scalable software and applications on the world’s largest supercomputers. He also held senior faculty and adjunct appointments at UTK, GT, and UNL, and was a Fellow at the Institute of Advanced Studies in Durham University, UK.