-
Extended Abstract: Productive Parallel Programming with Parsl
Authors:
Kyle Chard,
Yadu Babuji,
Anna Woodard,
Ben Clifford,
Zhuozhao Li,
Mihael Hategan,
Ian Foster,
Mike Wilde,
Daniel S. Katz
Abstract:
Parsl is a parallel programming library for Python that aims to make it easy to specify parallelism in programs and to realize that parallelism on arbitrary parallel and distributed computing systems. Parsl relies on developers annotating Python functions-wrapping either Python or external applications-to indicate that these functions may be executed concurrently. Developers can then link together…
▽ More
Parsl is a parallel programming library for Python that aims to make it easy to specify parallelism in programs and to realize that parallelism on arbitrary parallel and distributed computing systems. Parsl relies on developers annotating Python functions-wrapping either Python or external applications-to indicate that these functions may be executed concurrently. Developers can then link together functions via the exchange of data. Parsl establishes a dynamic dependency graph and sends tasks for execution on connected resources when dependencies are resolved. Parsl's runtime system enables different compute resources to be used, from laptops to supercomputers, without modification to the Parsl program.
△ Less
Submitted 4 May, 2022; v1 submitted 3 May, 2022;
originally announced May 2022.
-
ExaWorks: Workflows for Exascale
Authors:
Aymen Al-Saadi,
Dong H. Ahn,
Yadu Babuji,
Kyle Chard,
James Corbett,
Mihael Hategan,
Stephen Herbein,
Shantenu Jha,
Daniel Laney,
Andre Merzky,
Todd Munson,
Michael Salim,
Mikhail Titov,
Matteo Turilli,
Justin M. Wozniak
Abstract:
Exascale computers will offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. These software combinations and integrations, however, are difficult to achieve due to challenges of coordination and deployment of heterogeneous software components on diverse and massive platforms.…
▽ More
Exascale computers will offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. These software combinations and integrations, however, are difficult to achieve due to challenges of coordination and deployment of heterogeneous software components on diverse and massive platforms. We present the ExaWorks project, which can address many of these challenges: ExaWorks is leading a co-design process to create a workflow software development Toolkit (SDK) consisting of a wide range of workflow management tools that can be composed and interoperate through common interfaces. We describe the initial set of tools and interfaces supported by the SDK, efforts to make them easier to apply to complex science challenges, and examples of their application to exemplar cases. Furthermore, we discuss how our project is working with the workflows community, large computing facilities as well as HPC platform vendors to sustainably address the requirements of workflows at the exascale.
△ Less
Submitted 30 August, 2021;
originally announced August 2021.
-
Toward Enabling Reproducibility for Data-Intensive Research using the Whole Tale Platform
Authors:
Kyle Chard,
Niall Gaffney,
Mihael Hategan,
Kacper Kowalik,
Bertram Ludaescher,
Timothy McPhillips,
Jarek Nabrzyski,
Victoria Stodden,
Ian Taylor,
Thomas Thelen,
Matthew J. Turk,
Craig Willis
Abstract:
Whole Tale http://wholetale.org is a web-based, open-source platform for reproducible research supporting the creation, sharing, execution, and verification of "Tales" for the scientific research community. Tales are executable research objects that capture the code, data, and environment along with narrative and workflow information needed to re-create computational results from scientific studie…
▽ More
Whole Tale http://wholetale.org is a web-based, open-source platform for reproducible research supporting the creation, sharing, execution, and verification of "Tales" for the scientific research community. Tales are executable research objects that capture the code, data, and environment along with narrative and workflow information needed to re-create computational results from scientific studies. Creating reproducible research objects that enable reproducibility, transparency, and re-execution for computational experiments requiring significant compute resources or utilizing massive data is an especially challenging open problem. We describe opportunities, challenges, and solutions to facilitating reproducibility for data- and compute-intensive research, that we call "Tales at Scale," using the Whole Tale computing platform. We highlight challenges and solutions in frontend responsiveness needs, gaps in current middleware design and implementation, network restrictions, containerization, and data access. Finally, we discuss challenges in packaging computational experiment implementations for portable data-intensive Tales and outline future work.
△ Less
Submitted 12 May, 2020;
originally announced May 2020.
-
Computing Environments for Reproducibility: Capturing the "Whole Tale"
Authors:
Adam Brinckman,
Kyle Chard,
Niall Gaffney,
Mihael Hategan,
Matthew B. Jones,
Kacper Kowalik,
Sivakumar Kulasekaran,
Bertram Ludäscher,
Bryce D. Mecum,
Jarek Nabrzyski,
Victoria Stodden,
Ian J. Taylor,
Matthew J. Turk,
Kandace Turner
Abstract:
The act of sharing scientific knowledge is rapidly evolving away from traditional articles and presentations to the delivery of executable objects that integrate the data and computational details (e.g., scripts and workflows) upon which the findings rely. This envisioned coupling of data and process is essential to advancing science but faces technical and institutional barriers. The Whole Tale p…
▽ More
The act of sharing scientific knowledge is rapidly evolving away from traditional articles and presentations to the delivery of executable objects that integrate the data and computational details (e.g., scripts and workflows) upon which the findings rely. This envisioned coupling of data and process is essential to advancing science but faces technical and institutional barriers. The Whole Tale project aims to address these barriers by connecting computational, data-intensive research efforts with the larger research process--transforming the knowledge discovery and dissemination process into one where data products are united with research articles to create "living publications" or "tales". The Whole Tale focuses on the full spectrum of science, empowering users in the long tail of science, and power users with demands for access to big data and compute resources. We report here on the design, architecture, and implementation of the Whole Tale environment.
△ Less
Submitted 1 May, 2018;
originally announced May 2018.
-
Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments
Authors:
Yong Zhao,
Ioan Raicu,
Ian Foster,
Mihael Hategan,
Veronika Nefedova,
Mike Wilde
Abstract:
The practical realization of managing and executing large scale scientific computations efficiently and reliably is quite challenging. Scientific computations often involve thousands or even millions of tasks operating on large quantities of data, such data are often diversely structured and stored in heterogeneous physical formats, and scientists must specify and run such computations over exte…
▽ More
The practical realization of managing and executing large scale scientific computations efficiently and reliably is quite challenging. Scientific computations often involve thousands or even millions of tasks operating on large quantities of data, such data are often diversely structured and stored in heterogeneous physical formats, and scientists must specify and run such computations over extended periods on collections of compute, storage and network resources that are heterogeneous, distributed and may change constantly. We present the integration of several advanced systems: Swift, Karajan, and Falkon, to address the challenges in running various large scale scientific applications in Grid environments. Swift is a parallel programming tool for rapid and reliable specification, execution, and management of large-scale science and engineering workflows. Swift consists of a simple scripting language called SwiftScript and a powerful runtime system that is based on the CoG Karajan workflow engine and integrates the Falkon light-weight task execution service that uses multi-level scheduling and a streamlined dispatcher. We showcase the scalability, performance and reliability of the integrated system using application examples drawn from astronomy, cognitive neuroscience and molecular dynamics, which all comprise large number of fine-grained jobs. We show that Swift is able to represent dynamic workflows whose structures can only be determined during runtime and reduce largely the code size of various workflow representations using SwiftScript; schedule the execution of hundreds of thousands of parallel computations via the Karajan engine; and achieve up to 90% reduction in execution time when compared to traditional batch schedulers.
△ Less
Submitted 26 August, 2008;
originally announced August 2008.
-
Plate-forme Magicien d'Oz pour l'étude de l'apport des ACAs à l'interaction
Authors:
Jérôme Simonin,
Marius Hategan,
Noëlle Carbonell
Abstract:
In order to evaluate the contribution of Embodied (Animated) Conversational Agents (ECAs) to the effectiveness and usability of human-computer interaction, we developed a software platform meant to collect usage data. This platform, which implements the wizard of Oz paradigm, makes it possible to simulate user interfaces integrating ACAs for any Windows software application. It can also save and…
▽ More
In order to evaluate the contribution of Embodied (Animated) Conversational Agents (ECAs) to the effectiveness and usability of human-computer interaction, we developed a software platform meant to collect usage data. This platform, which implements the wizard of Oz paradigm, makes it possible to simulate user interfaces integrating ACAs for any Windows software application. It can also save and "replay" a rich interaction trace including user and system events, screen captures, users' speech and eye fixations. This platform has been used to assess users' subjective judgements and reactions to a multimodal online help system meant to facilitate the use of software for the general public (Flash). The online help system is embodied using a 3D talking head (developed by FT R&D) which "says" oral help messages illustrated with Flash screen copies.
△ Less
Submitted 28 August, 2007;
originally announced August 2007.
