Future exascale high-performance computers require a new kind of software that allows dynamic workloads to run with maximum energy efficiency on the best-suited hardware available in the system. The Technical University of Munich (TUM) and the Leibniz Supercomputing Centre (LRZ) are working together to create a production-ready software stack to enable low- energy, high-performance exascale systems.

Exascale supercomputers are knocking at the door. They might be a game-changer to the way we design and use high-performance computing (HPC) systems. As exascale performance drives more HPC systems toward heterogenous architectures that mix traditional CPUs with accelerators like GPUs and FPGAs, computational scientists will have to design more dynamic applications and workloads in order to get massive performance increases for their applications.

The question is: How will applications leverage these different technologies efficiently and effectively Power management and dynamic resource allocation will become the most important aspects of this new area of HPC. Stated more simply: how do HPC centres ensure that users are getting the most science per Joule

Optimization is challenging

Optimizing application performance on heterogeneous systems under power and energy constraints poses some challenges. Some are quite sophisticated, like the dynamic phase behaviour of applications. And some are basic hardware issues like the variability of processors: Due to manufacturing limitations, low-power operation of CPUs can cause a wide variety of frequencies across the cores. Adding to these is the ever-growing complexity and heterogeneity on node-level.

A software stack for such heterogeneous exascale systems will have to meet some specific demands. It has to be dynamic, work with highly heterogeneous integrated systems, and adapt to existing hardware. TUM and LRZ are working closely together to build a software stack based on existing and proven solutions. Among others, MPI and its various implementations, SLURM, PMIx or DCDB are well-known parts of this Munich Software Stack.

“The basic stack is already running on the SuperMUC-NG supercomputer at the LRZ”, says Martin Schulz, Chair for Computer Architecture and Parallel Systems at the Technical University of Munich and Director at the Leibniz Supercomputing Centre. “Right now, we are engaged in two European research projects for further development of this stack on more heterogeneous, deeper integrated and dynamic systems, as they will become commonplace in the exascale era: REGALE and DEEP-SEA.” One of the foundations for the next generation of this software stack is the HPC PowerStack[1], an initiative, with TUM as one of the co-founders, for better standardization and homogenization of approaches for power and energy optimized systems.

EU projects address the questions

REGALE aims to define an open architecture, build a prototype system, and incorporate in this system appropriate sophistication in order to equip supercomputing systems with the mechanisms and policies for effective resource utilization and execution of complex applications. DEEP-SEA will deliver the programming environment for future European exascale systems, capable of adapting at all levels of the software stack. While the basic technologies will be implemented and used in DEEP-SEA, the control chain will play a major role in REGALE.

Both projects are focused on making existing codes more dynamic so they can leverage existing accelerators: Many codes today are static and might only be partially ready for more dynamics. This will require some refactoring, and in some cases, complete rewrites of certain parts of the codes. But it will also require novel and elaborate scheduling methods that must be developed by HPC centres themselves. Part of the upcoming research in DEEP-SEA and REGALE will be to find ways to determine where targeted efforts on top of an existing software stack can yield the greatest result. To this end, agile development approaches will play a role: Continuous Integration with elaborate testing and automation are being established on BEAST (Bavarian Energy-, Architecture- and Software-Testbed) at the LRZ, the testbed for the Munich Software Stack.

Need for holistic approach

“Most research in the field of power and energy management today is done site-specific,” Schulz said. “We see little integration of the components; we have a lack of standardized interfaces that work on all layers of the software stack. In the end, this leads to suboptimal performance of the applications and increases the power needed by the system.. With the Munich Software Stack, TUM and LRZ are working on an open, holistic, and scalable approach to an integrated power and energy management  in order to get the most out of supercomputers to come.”