Omnibenchmark (alpha) for continuous and open benchmarking in bioinformatics
Authors:
Izaskun Mallona,
Almut Luetge,
Ben Carrillo,
Daniel Incicau,
Reto Gerber,
Anthony Sonrel,
Charlotte Soneson,
Mark D. Robinson
Abstract:
Benchmarking in bioinformatics is a process of designing, running and disseminating rigorous performance evaluations of methods (software). Benchmarking systems facilitate the benchmarking process by providing an entrypoint to store, coordinate and execute concrete benchmarks.
We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and exec…
▽ More
Benchmarking in bioinformatics is a process of designing, running and disseminating rigorous performance evaluations of methods (software). Benchmarking systems facilitate the benchmarking process by providing an entrypoint to store, coordinate and execute concrete benchmarks.
We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts. Omnibenchmark provides a benchmark definition syntax (in a configuration YAML file), a dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using EasyBuild, lmod, Apptainer or conda. Tutorials and installation instructions are available from https://omnibenchmark.org.
△ Less
Submitted 25 September, 2024;
originally announced September 2024.
Building a continuous benchmarking ecosystem in bioinformatics
Authors:
Izaskun Mallona,
Charlotte Soneson,
Ben Carrillo,
Almut Luetge,
Daniel Incicau,
Reto Gerber,
Anthony Sonrel,
Mark D. Robinson
Abstract:
Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform t…
▽ More
Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.
△ Less
Submitted 19 November, 2024; v1 submitted 23 September, 2024;
originally announced September 2024.
