Right-sizing compute resource allocations for bioinformatics tools with Total Perspective Vortex
Authors:
Nuwan Goonasekera,
Catherine Bromhead,
Simon Gladman,
Nate Coraor,
Bjorn Gruning,
Enis Afgan
Abstract:
In biomedical research, computational methods have become indispensable and their use is increasing, making the efficient allocation of computing resources paramount. Practitioners routinely allocate resources far in excess of what is required for batch processing jobs, leading to not just inflated wait times and costs, but also unnecessary carbon emissions. This is not without reason however, as…
▽ More
In biomedical research, computational methods have become indispensable and their use is increasing, making the efficient allocation of computing resources paramount. Practitioners routinely allocate resources far in excess of what is required for batch processing jobs, leading to not just inflated wait times and costs, but also unnecessary carbon emissions. This is not without reason however, as accurately determining resource needs is complex, affected by the nature of tools, data size, and analysis parameters, especially on popular servers that handle numerous jobs. The Galaxy platform, a web-based hub for biomedical analysis used globally by scientists, exemplifies this challenge. Serving nearly half a million registered users and managing around 2 million monthly jobs, Galaxy's growth outpaces the resources at its disposal. This is necessitating smarter resource utilization. To address this, we have developed a tool named Total Perspective Vortex (TPV) - a software package that right-sizes resource allocations for each job. TPV is able to dynamically set resource requirements for individual jobs and perform meta-scheduling across heterogeneous resources. It also includes a first-ever community-curated database of default resource requirements for nearly 1,000 popular bioinformatics tools. Deployments in Galaxy Australia and Europe demonstrate its effectiveness with meta-scheduling user jobs and an improved experience for systems administrators managing Galaxy servers.
△ Less
Submitted 4 December, 2023;
originally announced December 2023.
CloudLaunch: Discover and Deploy Cloud Applications
Authors:
Enis Afgan,
Andrew Lonie,
James Taylor,
Nuwan Goonasekera
Abstract:
Cloud computing is a common platform for delivering software to end users. However, the process of making complex-to-deploy applications available across different cloud providers requires isolated and uncoordinated application-specific solutions, often locking-in developers to a particular cloud provider. Here, we present the CloudLaunch application as a uniform platform for discovering and deplo…
▽ More
Cloud computing is a common platform for delivering software to end users. However, the process of making complex-to-deploy applications available across different cloud providers requires isolated and uncoordinated application-specific solutions, often locking-in developers to a particular cloud provider. Here, we present the CloudLaunch application as a uniform platform for discovering and deploying applications for different cloud providers. CloudLaunch allows arbitrary applications to be added to a catalog with each application having its own customizable user interface and control over the launch process, while preserving cloud-agnosticism so that authors can easily make their applications available on multiple clouds with minimal effort. It then provides a uniform interface for launching available applications by end users across different cloud providers. Architecture details are presented along with examples of different deployable applications that highlight architectural features.
△ Less
Submitted 10 May, 2018; v1 submitted 10 May, 2018;
originally announced May 2018.
