-
acoupi: An Open-Source Python Framework for Deploying Bioacoustic AI Models on Edge Devices
Authors:
Aude Vuilliomenet,
Santiago Martínez Balvanera,
Oisin Mac Aodha,
Kate E. Jones,
Duncan Wilson
Abstract:
1. Passive acoustic monitoring (PAM) coupled with artificial intelligence (AI) is becoming an essential tool for biodiversity monitoring. Traditional PAM systems require manual data offloading and impose substantial demands on storage and computing infrastructure. The combination of on-device AI-based processing and network connectivity enables local data analysis and transmission of only relevant…
▽ More
1. Passive acoustic monitoring (PAM) coupled with artificial intelligence (AI) is becoming an essential tool for biodiversity monitoring. Traditional PAM systems require manual data offloading and impose substantial demands on storage and computing infrastructure. The combination of on-device AI-based processing and network connectivity enables local data analysis and transmission of only relevant information, greatly reducing storage needs. However, programming these devices for robust operation is challenging, requiring expertise in embedded systems and software engineering. Despite the increase in AI-based models for bioacoustics, their full potential remains unrealized without accessible tools to deploy them on custom hardware and tailor device behaviour to specific monitoring goals. 2. To address this challenge, we develop acoupi, an open-source Python framework that simplifies the creation and deployment of smart bioacoustic devices. acoupi integrates audio recording, AI-based data processing, data management, and real-time wireless messaging into a unified and configurable framework. By modularising key elements of the bioacoustic monitoring workflow, acoupi allows users to easily customise, extend, or select specific components to fit their unique monitoring needs. 3. We demonstrate the flexibility of acoupi by integrating two bioacoustic classifiers: BirdNET, for the classification of bird species, and BatDetect2, for the classification of UK bat species. We test the reliability of acoupi over a month-long deployment of two acoupi-powered devices in a UK urban park. 4. acoupi can be deployed on low-cost hardware such as the Raspberry Pi and can be customised for various applications. acoupi standardised framework and simplified tools facilitate the adoption of AI-powered PAM systems for researchers and conservationists. acoupi is on GitHub at https://github.com/acoupi/acoupi.
△ Less
Submitted 29 January, 2025;
originally announced January 2025.
-
INQUIRE: A Natural World Text-to-Image Retrieval Benchmark
Authors:
Edward Vendrow,
Omiros Pantazis,
Alexander Shepard,
Gabriel Brostow,
Kate E. Jones,
Oisin Mac Aodha,
Sara Beery,
Grant Van Horn
Abstract:
We introduce INQUIRE, a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. INQUIRE includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total match…
▽ More
We introduce INQUIRE, a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. INQUIRE includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total matches. Queries span categories such as species identification, context, behavior, and appearance, emphasizing tasks that require nuanced image understanding and domain expertise. Our benchmark evaluates two core retrieval tasks: (1) INQUIRE-Fullrank, a full dataset ranking task, and (2) INQUIRE-Rerank, a reranking task for refining top-100 retrievals. Detailed evaluation of a range of recent multimodal models demonstrates that INQUIRE poses a significant challenge, with the best models failing to achieve an mAP@50 above 50%. In addition, we show that reranking with more powerful multimodal models can enhance retrieval performance, yet there remains a significant margin for improvement. By focusing on scientifically-motivated ecological challenges, INQUIRE aims to bridge the gap between AI capabilities and the needs of real-world scientific inquiry, encouraging the development of retrieval systems that can assist with accelerating ecological and biodiversity research. Our dataset and code are available at https://inquire-benchmark.github.io
△ Less
Submitted 11 November, 2024; v1 submitted 4 November, 2024;
originally announced November 2024.
-
Deep learning-based ecological analysis of camera trap images is impacted by training data quality and quantity
Authors:
Peggy A. Bevan,
Omiros Pantazis,
Holly Pringle,
Guilherme Braga Ferreira,
Daniel J. Ingram,
Emily Madsen,
Liam Thomas,
Dol Raj Thanet,
Thakur Silwal,
Santosh Rayamajhi,
Gabriel Brostow,
Oisin Mac Aodha,
Kate E. Jones
Abstract:
Large image collections generated from camera traps offer valuable insights into species richness, occupancy, and activity patterns, significantly aiding biodiversity monitoring. However, the manual processing of these datasets is time-consuming, hindering analytical processes. To address this, deep neural networks have been adopted to automate image labelling, but the impact of classification err…
▽ More
Large image collections generated from camera traps offer valuable insights into species richness, occupancy, and activity patterns, significantly aiding biodiversity monitoring. However, the manual processing of these datasets is time-consuming, hindering analytical processes. To address this, deep neural networks have been adopted to automate image labelling, but the impact of classification error on ecological metrics remains unclear. Here, we analyse data from camera trap collections in an African savannah (82,300 images, 47 species) and an Asian sub-tropical dry forest (40,308 images, 29 species) to compare ecological metrics derived from expert-generated species identifications with those generated by deep learning classification models. We specifically assess the impact of deep learning model architecture, the proportion of label noise in the training data, and the size of the training dataset on three ecological metrics: species richness, occupancy, and activity patterns. Overall, ecological metrics derived from deep neural networks closely match those calculated from expert labels and remain robust to manipulations in the training pipeline. We found that the choice of deep learning model architecture does not impact ecological metrics, and ecological metrics related to the overall community (species richness, community occupancy) were resilient to up to 10% noise in the training dataset and a 50% reduction in the training dataset size. However, we caution that less common species are disproportionately affected by a reduction in deep neural network accuracy, and this has consequences for species-specific metrics (occupancy, diel activity patterns). To ensure the reliability of their findings, practitioners should prioritize creating large, clean training sets with balanced representation across species over exploring numerous deep learning model architectures.
△ Less
Submitted 7 May, 2025; v1 submitted 26 August, 2024;
originally announced August 2024.
-
Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics
Authors:
Ben Williams,
Bart van Merriënboer,
Vincent Dumoulin,
Jenny Hamer,
Eleni Triantafillou,
Abram B. Fleishman,
Matthew McKown,
Jill E. Munger,
Aaron N. Rice,
Ashlee Lillis,
Clemency E. White,
Catherine A. D. Hobbs,
Tries B. Razak,
Kate E. Jones,
Tom Denton
Abstract:
Machine learning has the potential to revolutionize passive acoustic monitoring (PAM) for ecological assessments. However, high annotation and compute costs limit the field's efficacy. Generalizable pretrained networks can overcome these costs, but high-quality pretraining requires vast annotated libraries, limiting its current applicability primarily to bird taxa. Here, we identify the optimum pr…
▽ More
Machine learning has the potential to revolutionize passive acoustic monitoring (PAM) for ecological assessments. However, high annotation and compute costs limit the field's efficacy. Generalizable pretrained networks can overcome these costs, but high-quality pretraining requires vast annotated libraries, limiting its current applicability primarily to bird taxa. Here, we identify the optimum pretraining strategy for a data-deficient domain using coral reef bioacoustics. We assemble ReefSet, a large annotated library of reef sounds, though modest compared to bird libraries at 2% of the sample count. Through testing few-shot transfer learning performance, we observe that pretraining on bird audio provides notably superior generalizability compared to pretraining on ReefSet or unrelated audio alone. However, our key findings show that cross-domain mixing which leverages bird, reef and unrelated audio during pretraining maximizes reef generalizability. SurfPerch, our pretrained network, provides a strong foundation for automated analysis of marine PAM data with minimal annotation and compute costs.
△ Less
Submitted 7 May, 2024; v1 submitted 25 April, 2024;
originally announced April 2024.
-
Whombat: An open-source annotation tool for machine learning development in bioacoustics
Authors:
Santiago Martinez Balvanera,
Oisin Mac Aodha,
Matthew J. Weldy,
Holly Pringle,
Ella Browning,
Kate E. Jones
Abstract:
1. Automated analysis of bioacoustic recordings using machine learning (ML) methods has the potential to greatly scale biodiversity monitoring efforts. The use of ML for high-stakes applications, such as conservation research, demands a data-centric approach with a focus on utilizing carefully annotated and curated evaluation and training data that is relevant and representative. Creating annotate…
▽ More
1. Automated analysis of bioacoustic recordings using machine learning (ML) methods has the potential to greatly scale biodiversity monitoring efforts. The use of ML for high-stakes applications, such as conservation research, demands a data-centric approach with a focus on utilizing carefully annotated and curated evaluation and training data that is relevant and representative. Creating annotated datasets of sound recordings presents a number of challenges, such as managing large collections of recordings with associated metadata, developing flexible annotation tools that can accommodate the diverse range of vocalization profiles of different organisms, and addressing the scarcity of expert annotators.
2. We present Whombat a user-friendly, browser-based interface for managing audio recordings and annotation projects, with several visualization, exploration, and annotation tools. It enables users to quickly annotate, review, and share annotations, as well as visualize and evaluate a set of machine learning predictions on a dataset. The tool facilitates an iterative workflow where user annotations and machine learning predictions feedback to enhance model performance and annotation quality.
3. We demonstrate the flexibility of Whombat by showcasing two distinct use cases: an project aimed at enhancing automated UK bat call identification at the Bat Conservation Trust (BCT), and a collaborative effort among the USDA Forest Service and Oregon State University researchers exploring bioacoustic applications and extending automated avian classification models in the Pacific Northwest, USA.
4. Whombat is a flexible tool that can effectively address the challenges of annotation for bioacoustic research. It can be used for individual and collaborative work, hosted on a shared server or accessed remotely, or run on a personal computer without the need for coding skills.
△ Less
Submitted 7 November, 2023; v1 submitted 24 August, 2023;
originally announced August 2023.
