The divide–expand–consolidate MP2 scheme goes massively parallel

Abstract

For large molecular systems conventional implementations of second order Møller–Plesset (MP2) theory encounter a scaling wall, both memory- and time-wise. We describe how this scaling wall can be removed. We present a massively parallel algorithm for calculating MP2 energies and densities using the divide–expand–consolidate scheme where a calculation on a large system is divided into many small fragment calculations employing local orbital spaces. The resulting algorithm is linear-scaling with system size, exhibits near perfect parallel scalability, removes memory bottlenecks and does not involve any I/O. The algorithm employs three levels of parallelisation combined via a dynamic job distribution scheme. Results on two molecular systems containing 528 and 1056 atoms (4278 and 8556 basis functions) using 47,120 and 94,240 cores are presented. The results demonstrate the scalability of the algorithm both with respect to the number of cores and with respect to system size. The presented algorithm is thus highly suited for large super computer architectures and allows MP2 calculations on large molecular systems to be carried out within a few hours – for example, the correlated calculation on the molecular system containing 1056 atoms took 2.37 hours using 94240 cores.

Keywords:

• massively parallel
• DEC
• linear scaling
• MP2
• large molecules

Acknowledgements

We would like to congratulate Prof. Trygve Helgaker on the occasion of his 60th birthday and thank him for many years of fruitful collaboration as well as his engagement in teaching future quantum chemists. This research used the Titan supercomputer of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. The research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP/2007-2013)/ERC Grant Agreement No. 291371. Support from the National Institute for Computational Sciences (NICS), the Danish Center for Scientific Computing (DCSC) and The Danish Council for Independent Research – Natural Sciences is gratefully acknowledged. S.R. acknowledges support from the Norwegian Research council through the CoE Centre for Theoretical and Computational Chemistry (CTCC) Grant No. 179568/V30. J.J. would like to acknowledge the support from the National Science Foundation (Grant No. ARRA-NSF-EPS-0919436).

Notes

^aNumber of NUMA nodes. Each NUMA node runs one MPI process which employs OpenMP parallelisation over eight cores.

^bTime-to-solution for all fragment calculations.

^aValue used in efficiency criterion in Equation (18).

^bPercentage loss within the fragment slots (Figure 3).

^cPercentage loss associated with idle time at the end of the calculation and MPI communication between global master and local slaves (Figure 4).

^dSum of local and global losses.

^aTotal number of petaFLOP for all fragment calculations.

^bAverage number of GFLOP per second per core.

^aNumber of occupied AOS orbitals in fragment.

^bNumber of virtual AOS orbitals in fragment.

^cTime measured by local master.

^dNumber of GFLOP per second per core calculated using effective times, where the MPI associated losses in Table 3 have been removed (see text).