Model and simulation of exascale communication networks

Abstract

Exascale supercomputers will have millions or even hundreds of millions of processing cores and the potential for nearly billion-way parallelism. Exascale compute and data storage architectures will be critically dependent on the interconnection network. The most popular interconnection network for current and future supercomputer systems is the torus (eg, k-ary, n-cube). This paper focuses on the modelling and simulation of ultra-large-scale torus networks using Rensselaer's Optimistic Simulator System. We compare real communication delays between our model and the actual torus network from Blue Gene/L using 2048 processors. Our performance experiments demonstrate the ability to simulate million-node to billion-node torus networks. The torus network model for a 16-million-node configuration shows a high degree of strong scaling when going from 1024 cores to 32 768 cores on Blue Gene/L, with a peak event-rate of nearly 5 billion events per second. We also demonstrate the performance of our torus network model configured with 1 billion nodes on both Blue Gene/L and Blue Gene/P systems. The observed best event rate at 128 K cores is 12.36 billion per second on Blue Gene/P.

Keywords:

• parallel discrete-event simulation
• torus network
• exascale
• discrete-event model

Acknowledgements

This work was supported in part by the Office of Advanced Scientific Computer Research, Office of Science, US Dept. of Energy, under Contracts DE-AC02-06CH11357 and DE-FC02-10ER25989/DE-SC0004875, and in part by the NSF CNS NeTS Program, Contract #0435259. This research used resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the US Department of Energy under contract DE-AC02-06CH11357. Computing time on Intrepid was provided by a US Department of Energy INCITE award. Rensselaer's Computational Center for Nanotechnology Innovations (CCNI) provided the Blue Gene/L computing resources.