-
Omnibenchmark (alpha) for continuous and open benchmarking in bioinformatics
Authors:
Izaskun Mallona,
Almut Luetge,
Ben Carrillo,
Daniel Incicau,
Reto Gerber,
Anthony Sonrel,
Charlotte Soneson,
Mark D. Robinson
Abstract:
Benchmarking in bioinformatics is a process of designing, running and disseminating rigorous performance evaluations of methods (software). Benchmarking systems facilitate the benchmarking process by providing an entrypoint to store, coordinate and execute concrete benchmarks.
We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and exec…
▽ More
Benchmarking in bioinformatics is a process of designing, running and disseminating rigorous performance evaluations of methods (software). Benchmarking systems facilitate the benchmarking process by providing an entrypoint to store, coordinate and execute concrete benchmarks.
We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts. Omnibenchmark provides a benchmark definition syntax (in a configuration YAML file), a dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using EasyBuild, lmod, Apptainer or conda. Tutorials and installation instructions are available from https://omnibenchmark.org.
△ Less
Submitted 25 September, 2024;
originally announced September 2024.
-
Building a continuous benchmarking ecosystem in bioinformatics
Authors:
Izaskun Mallona,
Charlotte Soneson,
Ben Carrillo,
Almut Luetge,
Daniel Incicau,
Reto Gerber,
Anthony Sonrel,
Mark D. Robinson
Abstract:
Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform t…
▽ More
Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.
△ Less
Submitted 19 November, 2024; v1 submitted 23 September, 2024;
originally announced September 2024.
-
Ten simple rules for collaborating with wet lab researchers for computational researchers
Authors:
Mark D. Robinson,
Peiying Cai,
Martin Emons,
Reto Gerber,
Pierre-Luc Germain,
Samuel Gunz,
Siyuan Luo,
Giulia Moro,
Emanuel Sonder,
Anthony Sonrel,
Jiayi Wang,
David Wissel,
Izaskun Mallona
Abstract:
Computational biologists are frequently engaged in collaborative data analysis with wet lab researchers. These interdisciplinary projects, as necessary as they are to the scientific endeavour, can be surprisingly challenging due to cultural differences in operations and values. In these Ten Simple Rules guide we aim to help dry lab researchers identify sources of friction; and provide actionable t…
▽ More
Computational biologists are frequently engaged in collaborative data analysis with wet lab researchers. These interdisciplinary projects, as necessary as they are to the scientific endeavour, can be surprisingly challenging due to cultural differences in operations and values. In these Ten Simple Rules guide we aim to help dry lab researchers identify sources of friction; and provide actionable tools to facilitate respectful, open, transparent and rewarding collaborations.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
