-
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
Authors:
Arthur S. Bianchessi,
Rodrigo C. Barros,
Lucas S. Kupssinskü
Abstract:
Transformer-based language models rely on positional encoding (PE) to handle token order and support context length extrapolation. However, existing PE methods lack theoretical clarity and rely on limited evaluation metrics to substantiate their extrapolation claims. We propose the Bayesian Attention Mechanism (BAM), a theoretical framework that formulates positional encoding as a prior within a p…
▽ More
Transformer-based language models rely on positional encoding (PE) to handle token order and support context length extrapolation. However, existing PE methods lack theoretical clarity and rely on limited evaluation metrics to substantiate their extrapolation claims. We propose the Bayesian Attention Mechanism (BAM), a theoretical framework that formulates positional encoding as a prior within a probabilistic model. BAM unifies existing methods (e.g., NoPE and ALiBi) and motivates a new Generalized Gaussian positional prior that substantially improves long-context generalization. Empirically, BAM enables accurate information retrieval at $500\times$ the training context length, outperforming previous state-of-the-art context length generalization in long context retrieval accuracy while maintaining comparable perplexity and introducing minimal additional parameters.
△ Less
Submitted 28 May, 2025;
originally announced May 2025.
-
Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines
Authors:
Christian Mattjie,
Luis Vinicius de Moura,
Rafaela Cappelari Ravazio,
Lucas Silveira Kupssinskü,
Otávio Parraga,
Marcelo Mussi Delucis,
Rodrigo Coelho Barros
Abstract:
Segmentation in medical imaging is a critical component for the diagnosis, monitoring, and treatment of various diseases and medical conditions. Presently, the medical segmentation landscape is dominated by numerous specialized deep learning models, each fine-tuned for specific segmentation tasks and image modalities. The recently-introduced Segment Anything Model (SAM) employs the ViT neural arch…
▽ More
Segmentation in medical imaging is a critical component for the diagnosis, monitoring, and treatment of various diseases and medical conditions. Presently, the medical segmentation landscape is dominated by numerous specialized deep learning models, each fine-tuned for specific segmentation tasks and image modalities. The recently-introduced Segment Anything Model (SAM) employs the ViT neural architecture and harnesses a massive training dataset to segment nearly any object; however, its suitability to the medical domain has not yet been investigated. In this study, we explore the zero-shot performance of SAM in medical imaging by implementing eight distinct prompt strategies across six datasets from four imaging modalities, including X-ray, ultrasound, dermatoscopy, and colonoscopy. Our findings reveal that SAM's zero-shot performance is not only comparable to, but in certain cases, surpasses the current state-of-the-art. Based on these results, we propose practical guidelines that require minimal interaction while consistently yielding robust outcomes across all assessed contexts. The source code, along with a demonstration of the recommended guidelines, can be accessed at https://github.com/Malta-Lab/SAM-zero-shot-in-Medical-Imaging.
△ Less
Submitted 5 May, 2023; v1 submitted 28 April, 2023;
originally announced May 2023.
-
Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey
Authors:
Otávio Parraga,
Martin D. More,
Christian M. Oliveira,
Nathan S. Gavenski,
Lucas S. Kupssinskü,
Adilson Medronha,
Luis V. Moura,
Gabriel S. Simões,
Rodrigo C. Barros
Abstract:
Despite being responsible for state-of-the-art results in several computer vision and natural language processing tasks, neural networks have faced harsh criticism due to some of their current shortcomings. One of them is that neural networks are correlation machines prone to model biases within the data instead of focusing on actual useful causal relationships. This problem is particularly seriou…
▽ More
Despite being responsible for state-of-the-art results in several computer vision and natural language processing tasks, neural networks have faced harsh criticism due to some of their current shortcomings. One of them is that neural networks are correlation machines prone to model biases within the data instead of focusing on actual useful causal relationships. This problem is particularly serious in application domains affected by aspects such as race, gender, and age. To prevent models from incurring on unfair decision-making, the AI community has concentrated efforts in correcting algorithmic biases, giving rise to the research area now widely known as fairness in AI. In this survey paper, we provide an in-depth overview of the main debiasing methods for fairness-aware neural networks in the context of vision and language research. We propose a novel taxonomy to better organize the literature on debiasing methods for fairness, and we discuss the current challenges, trends, and important future work directions for the interested researcher and practitioner.
△ Less
Submitted 10 November, 2022;
originally announced November 2022.
