Projects

D3: Invasive Computing and HPC

Principal Investigators:

Prof. H.-J. Bungartz, Prof. M. Gerndt, Prof. M. Bader

Scientific Researchers:

S. Narvaez, J. John 

Abstract

The research agenda of D3 is threefold: First, we consider numerical core routines widespread in both supercomputing and embedded applications with respect to an invasive enhancement. Second, we investigate advantages of invasive computing on HPC systems by integrating such invasive algorithms into real-life simulation scenarios. Third, concepts are developed to support invasion in standard HPC programming models on state-of-the-art HPC systems.

In the first funding phase, we demonstrated that invasion is a promising paradigm for a wide range of highly relevant numerical algorithms by defining requirements for a set of representative numerical building blocks. Furthermore, invasion allows for more flexible resource usage of HPC architectures by extending the standard programming model for shared-memory systems with invasive run-time mechanisms dynamically redistributing cores iOMP.

In the second funding phase, we developed an extension of the Message Passing Interface (MPI) for invasive programming. Four additional MPI routines allow to write malleable applications. The MPICH based implementation of iMPI was integrated with the widely used HPC batch system SLURM. Invasive MPI application can thus be run inside of a SLURM batch job on any HPC cluster. The Tsunami application as well as highly parallel inverse solver were implemented with iMPI and used in experiments on SuperMUC, the 6 Petaflop supercomputer at Leibniz Supercomputing Center in Garching.

In the third funding phase,the project will investigate the ability to dynamically enforce powerbounds on HPC systems with the help of invasive Resource management. It will develop an invasive checkpointing system that adapts to the application requirements by dynamically allocating or releasing nodes used for in-memory checkpoints. The focus on the application side will be on using the x86 implementation of the iRTSS on a cluser of Xeon Phi processors for tuning applications with respect to complex memory hierarchies.

Synopsis

The overarching goal of the third funding phase is to show that invasive computing can make a difference in HPC - not only for accelerating an individual application code, but also in terms of the daily practice of operation of large systems at supercomputing centres. This phase will leverage the results from Phase I and Phase II in the areas of invasive HPC applications and programming models.

One first big concern of current centres is power consumption, in particular the lack of predictability. Invasive computing can help to increase predictability and, thus, reduce the overall energy cost. Second, centres are worried about higher failure rates due to growing core numbers. Here, fast adaptive data recovery can be realised based on a resource-aware checkpointing infrastructure. Finally, novel manycore architectures will be deployed, increasing especially the complexity of efficiently handling the memory hierarchy. Invasion of memory layers with guidance by the application will allow for improved execution.

Approach

Enforcement of Power Corridor In addition to the overall energy consumption, power stability plays an important role in current and future HPC systems. Sudden changes in the power consumption of a large-scale system are dangerous for the electricity supply infrastructure. This has to be handled in hardware to allow for short response times. Power stability is also important for the pricing of electrical energy. Typically, the contracts between the electricity providers and the HPC centres are optimised for a predictable power draw. Using more or less power than predefined in the agreed power corridor increases the costs for the HPC centre. Furthermore, dynamic adaptation of the corridor on the demand of the power grid might be part of the contract and reduces the electricity costs further \cite{Chen:2014}. The computing centre can be seen as a significant reserve to balance power usage in the grid.

We will develop novel power-corridor-aware schedulers for the invasive resource management, that perform run-time verification and enforcement of the power corridor bounds. The invasive batch scheduler will select applications to run in the invasive partition not only with respect to execution and energy efficiency, but also with respect to the potential for controlling the power consumption. The invasive runtime scheduler will use this tuning potential of the running applications to enforce the power bounds of the system. When the power usages reaches the upper limit, the application's clock frequency can be reduced and resources can be removed from power demanding applications. If the power drops below the lower bound, applications with potential higher power usage can be assigned more resources as well as a higher clock frequency.

Invasive Checkpointing Fault tolerance is of high importance when the size of HPC systems (in terms of core numbers) keeps growing with increasing pace, since the huge amount of cores and other resources will lead to increasing failure rates of the systems. General strategies in HPC to provide fault tolerance are to perform redundant computations, to use fault-tolerant algorithms, and to follow a checkpoint restart approach. While redundant computation is much too costly and fault-tolerant algorithms are very rare, our focus will be on the checkpointing approach. Depending on the progress of the application, the availability of resources, and the overheads for checkpointing, periodic checkpoints of the current application state are saved. In case of a failure, the application can then be restarted based on the last checkpoint.

In contrast to existing checkpointing systems, our invasive checkpointing is a system-level infrastructure, which supports multiple applications with checkpointing requirements. This global support allows to carefully optimise the required resources, i.e., nodes, network bandwidth and storage capacity, across multiple applications. Resource-aware scheduling of checkpointing across applications will allow to reduce the amount of required resources significantly. iCheck will also be unique because it will support malleable jobs. It will leverage the idea of invasive resource management to dynamically adapt its own resources to the requirements of the jobs. Depending on the number of jobs requiring checkpointing, the amount of data and the frequency of checkpoints, nodes will be allocated providing memory and computing capacity for in-memory checkpoints, storage will be reserved, as well as bandwidth to the nodes and to storage resources will be managed.

Invasion on Novel Architectures We will also investigate invasive computing on modern state-of-the-art manycore processors, in close collaboration with project C1. The focus will be on the Intel Xeon Scalable Processors with Persistent Memory. Intel Xeon Gold 6238 processors with Intel Optane Persistent Memory will be used for this investigation. The multiple memory architecture is a challenge on the system as well as on the application side. We will look into invasive applications for this architecture, starting with the molecular dynamics application ls1 MarDyn. The work will focus on fine-grained parallelism in HPC applications. We will look out for higher-level abstractions simplifying application development based on that API.

Invasion for Large-Scale Applications The project will continue our development efforts in invasifying large-scale, real-world applications that can fill systems such as the upcoming SuperMUC-NG, and that have a strong need for high dynamics. For that, we intend to focus on the invasification of the earthquake simulation code SeisSol and the molecular dynamics code ls1 MarDyn. In the case of SeisSol, for example, a large gain can be expected in rupture simulations, where most of the simulation time is characterised by the dynamics of the rupture process. Here, we intend to apply the lazy activation approach, and activate parallel partitions only once seismic waves reach a partition. Concerning ls1 MarDyn, one of our main simulation scenarios is nucleation processes in large-scale chemical reactors.

Snapshot of the rupture front and emanating seismic waves during a simulation of the Sumatra 2004 earthquake using SeisSol. The dynamic rupture process of the earthquake was initiated in the bottom right corner and takes a substantial part of the total simulation time to propagate through the entire domain.

ls1 MarDyn: nucleation and droplet formulation in an over-saturated fluid.

A comprehensive summary of the major achievements of the first and second funding phase can be found by accessing Project D3 first phase and Project D3 second phase websites.

Publications