Title:Transparent Checkpoint-Restart over InfiniBand

Abstract:InfiniBand is widely used for low-latency, high-throughput cluster computing. Saving the state of the InfiniBand network as part of distributed checkpointing has been a long-standing challenge for researchers. Because of a lack of a solution, typical MPI implementations have included custom checkpoint-restart services that "tear down" the network, checkpoint each node as a standalone computer, and then re-connect the network again. We present the first example of transparent, system-initiated checkpoint-restart that directly supports InfiniBand. This direct approach results in checkpoints that are found to be several times faster than with the use of a checkpoint-restart service. Scalability is shown by checkpointing 128 MPI processes across 128 nodes in a few seconds. We also describe a debugging approach in which one can copy a checkpoint image from an InfiniBand-based production cluster to a local Ethernet-bsed cluster where one can restart the computation and attach an interactive debugger. The key enabling idea is direct virtualization of InfiniBand resources, with no need to tear down and re-connect the InfiniBand connection before and after checkpointing. The generality of this approach is shown not only by checkpointing an MPI computation, but also a native UPC computation (Berkeley Unified Parallel C). UPC was run over GASNet's ibv conduit, without the use of MPI. Performance is reported for a production InfiniBand cluster with 32 nodes. The work is based on DMTCP (Distributed MultiThreaded CheckPointing) plugins. DMTCP plugins are a modular extension of the core DMTCP checkpoint-restart package.