-
Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology
Authors:
Alex Hoi Hang Chan,
Otto Brookes,
Urs Waldmann,
Hemal Naik,
Iain D. Couzin,
Majid Mirmehdi,
Noël Adiko Houa,
Emmanuelle Normand,
Christophe Boesch,
Lukas Boesch,
Mimi Arandjelovic,
Hjalmar Kühl,
Tilo Burghardt,
Fumihiro Kano
Abstract:
Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that mo…
▽ More
Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case. To support this argument, we present two disparate case studies: (1) estimating chimpanzee abundance and density with camera trap distance sampling when using a video-based behaviour classifier and (2) estimating head rotation in pigeons using a 3D posture estimator. We show that even models with strong machine learning performance (e.g., 87% mAP) can yield data that leads to discrepancies in abundance estimates compared to expert-derived data. Similarly, the highest-performing models for posture estimation do not produce the most accurate inferences of gaze direction in pigeons. Motivated by these findings, we call for researchers to integrate application-specific metrics in ecological/biological datasets, allowing for models to be benchmarked in the context of their downstream application and to facilitate better integration of models into application workflows.
△ Less
Submitted 6 May, 2025; v1 submitted 5 May, 2025;
originally announced May 2025.
-
The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition
Authors:
Otto Brookes,
Maksim Kukushkin,
Majid Mirmehdi,
Colleen Stephens,
Paula Dieguez,
Thurston C. Hicks,
Sorrel Jones,
Kevin Lee,
Maureen S. McCarthy,
Amelia Meier,
Emmanuelle Normand,
Erin G. Wessling,
Roman M. Wittig,
Kevin Langergraber,
Klaus Zuberbühler,
Lukas Boesch,
Thomas Schmid,
Mimi Arandjelovic,
Hjalmar Kühl,
Tilo Burghardt
Abstract:
Computer vision analysis of camera trap video footage is essential for wildlife conservation, as captured behaviours offer some of the earliest indicators of changes in population health. Recently, several high-impact animal behaviour datasets and methods have been introduced to encourage their use; however, the role of behaviour-correlated background information and its significant effect on out-…
▽ More
Computer vision analysis of camera trap video footage is essential for wildlife conservation, as captured behaviours offer some of the earliest indicators of changes in population health. Recently, several high-impact animal behaviour datasets and methods have been introduced to encourage their use; however, the role of behaviour-correlated background information and its significant effect on out-of-distribution generalisation remain unexplored. In response, we present the PanAf-FGBG dataset, featuring 20 hours of wild chimpanzee behaviours, recorded at over 350 individual camera locations. Uniquely, it pairs every video with a chimpanzee (referred to as a foreground video) with a corresponding background video (with no chimpanzee) from the same camera location. We present two views of the dataset: one with overlapping camera locations and one with disjoint locations. This setup enables, for the first time, direct evaluation of in-distribution and out-of-distribution conditions, and for the impact of backgrounds on behaviour recognition models to be quantified. All clips come with rich behavioural annotations and metadata including unique camera IDs and detailed textual scene descriptions. Additionally, we establish several baselines and present a highly effective latent-space normalisation technique that boosts out-of-distribution performance by +5.42% mAP for convolutional and +3.75% mAP for transformer-based models. Finally, we provide an in-depth analysis on the role of backgrounds in out-of-distribution behaviour recognition, including the so far unexplored impact of background durations (i.e., the count of background frames within foreground videos).
△ Less
Submitted 19 March, 2025; v1 submitted 28 February, 2025;
originally announced February 2025.
-
PanAf20K: A Large Video Dataset for Wild Ape Detection and Behaviour Recognition
Authors:
Otto Brookes,
Majid Mirmehdi,
Colleen Stephens,
Samuel Angedakin,
Katherine Corogenes,
Dervla Dowd,
Paula Dieguez,
Thurston C. Hicks,
Sorrel Jones,
Kevin Lee,
Vera Leinert,
Juan Lapuente,
Maureen S. McCarthy,
Amelia Meier,
Mizuki Murai,
Emmanuelle Normand,
Virginie Vergnes,
Erin G. Wessling,
Roman M. Wittig,
Kevin Langergraber,
Nuria Maldonado,
Xinyu Yang,
Klaus Zuberbuhler,
Christophe Boesch,
Mimi Arandjelovic
, et al. (2 additional authors not shown)
Abstract:
We present the PanAf20K dataset, the largest and most diverse open-access annotated video dataset of great apes in their natural environment. It comprises more than 7 million frames across ~20,000 camera trap videos of chimpanzees and gorillas collected at 14 field sites in tropical Africa as part of the Pan African Programme: The Cultured Chimpanzee. The footage is accompanied by a rich set of an…
▽ More
We present the PanAf20K dataset, the largest and most diverse open-access annotated video dataset of great apes in their natural environment. It comprises more than 7 million frames across ~20,000 camera trap videos of chimpanzees and gorillas collected at 14 field sites in tropical Africa as part of the Pan African Programme: The Cultured Chimpanzee. The footage is accompanied by a rich set of annotations and benchmarks making it suitable for training and testing a variety of challenging and ecologically important computer vision tasks including ape detection and behaviour recognition. Furthering AI analysis of camera trap information is critical given the International Union for Conservation of Nature now lists all species in the great ape family as either Endangered or Critically Endangered. We hope the dataset can form a solid basis for engagement of the AI community to improve performance, efficiency, and result interpretation in order to support assessments of great ape presence, abundance, distribution, and behaviour and thereby aid conservation efforts.
△ Less
Submitted 31 January, 2024; v1 submitted 24 January, 2024;
originally announced January 2024.
