-
The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models
Authors:
Ariel Goldstein,
Eric Ham,
Mariano Schain,
Samuel Nastase,
Zaid Zada,
Avigail Dabush,
Bobbi Aubrey,
Harshvardhan Gazula,
Amir Feder,
Werner K Doyle,
Sasha Devore,
Patricia Dugan,
Daniel Friedman,
Roi Reichart,
Michael Brenner,
Avinatan Hassidim,
Orrin Devinsky,
Adeen Flinker,
Omer Levy,
Uri Hasson
Abstract:
Deep Language Models (DLMs) provide a novel computational paradigm for understanding the mechanisms of natural language processing in the human brain. Unlike traditional psycholinguistic models, DLMs use layered sequences of continuous numerical vectors to represent words and context, allowing a plethora of emerging applications such as human-like text generation. In this paper we show evidence th…
▽ More
Deep Language Models (DLMs) provide a novel computational paradigm for understanding the mechanisms of natural language processing in the human brain. Unlike traditional psycholinguistic models, DLMs use layered sequences of continuous numerical vectors to represent words and context, allowing a plethora of emerging applications such as human-like text generation. In this paper we show evidence that the layered hierarchy of DLMs may be used to model the temporal dynamics of language comprehension in the brain by demonstrating a strong correlation between DLM layer depth and the time at which layers are most predictive of the human brain. Our ability to temporally resolve individual layers benefits from our use of electrocorticography (ECoG) data, which has a much higher temporal resolution than noninvasive methods like fMRI. Using ECoG, we record neural activity from participants listening to a 30-minute narrative while also feeding the same narrative to a high-performing DLM (GPT2-XL). We then extract contextual embeddings from the different layers of the DLM and use linear encoding models to predict neural activity. We first focus on the Inferior Frontal Gyrus (IFG, or Broca's area) and then extend our model to track the increasing temporal receptive window along the linguistic processing hierarchy from auditory to syntactic and semantic areas. Our results reveal a connection between human language processing and DLMs, with the DLM's layer-by-layer accumulation of contextual information mirroring the timing of neural activity in high-order language areas.
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
-
RAVEN X High Performance Data Mining Toolbox for Bioacoustic Data Analysis
Authors:
Peter J. Dugan,
Holger Klinck,
Marie A. Roch,
Tyler A. Helble
Abstract:
Objective of this work is to integrate high performance computing (HPC) technologies and bioacoustics data-mining capabilities by offering a MATLAB-based toolbox called Raven-X. Raven-X will provide a hardware-independent solution, for processing large acoustic datasets - the toolkit will be available to the community at no cost. This goal will be achieved by leveraging prior work done which succe…
▽ More
Objective of this work is to integrate high performance computing (HPC) technologies and bioacoustics data-mining capabilities by offering a MATLAB-based toolbox called Raven-X. Raven-X will provide a hardware-independent solution, for processing large acoustic datasets - the toolkit will be available to the community at no cost. This goal will be achieved by leveraging prior work done which successfully deployed MATLAB based HPC tools within Cornell University's Bioacoustics Research Program (BRP). These tools enabled commonly available multi-core computers to process data at accelerated rates to detect and classify whale sounds in large multi-channel sound archives. Through this collaboration, we will expand on this effort which was featured through Mathworks research and industry forums incorporate new cutting-edge detectors and classifiers, and disseminate Raven-X to the broader bioacoustics community.
△ Less
Submitted 12 October, 2016;
originally announced October 2016.
-
Early and Late Time Acoustic Measures for Underwater Seismic Airgun Signals In Long-Term Acoustic Data Sets
Authors:
Peter Dugan,
Melania Guerra,
Dimitri Ponirakis,
Holger Klinck,
Christopher W. Clark
Abstract:
This work presents a new toolkit for describing the acoustic properties of the ocean environment before, during and after a sound event caused by an underwater seismic air-gun. The toolkit uses existing sound measures, but uniquely applies these to capture the early time period (actual pulse) and late time period (reverberation and multiple arrivals). In total, 183 features are produced for each a…
▽ More
This work presents a new toolkit for describing the acoustic properties of the ocean environment before, during and after a sound event caused by an underwater seismic air-gun. The toolkit uses existing sound measures, but uniquely applies these to capture the early time period (actual pulse) and late time period (reverberation and multiple arrivals). In total, 183 features are produced for each air-gun sound. This toolkit was utilized on data retrieved from a field deployment encompassing five marine autonomous recording units during a 46-day seismic air-gun survey in Baffin Bay, Greenland. Using this toolkit, a total of 147 million data points were identified from the Greenland deployment recordings. The feasibility of extracting a large number of features was then evaluated using two separate methods: a serial computer and a high performance system. Results indicate that data extraction performance took an estimated 216 hours for the serial system, and 18 hours for the high performance computer. This paper provides an analytical description of the new toolkit along with details for using it to identify relevant data.
△ Less
Submitted 5 May, 2016;
originally announced July 2016.
-
Phase 3: DCL System Using Deep Learning Approaches for Land-based or Ship-based Real-Time Recognition and Localization of Marine Mammals - Bioacoustic Applicaitons
Authors:
Peter J. Dugan,
Christopher W. Clark,
Yann André LeCun,
Sofie M. Van Parijs
Abstract:
Goals of this research phase is to investigate advanced detection and classification pardims useful for data-mining passive large passive acoustic archives. Technical objectives are to develop and refine a High Performance Computing, Acoustic Data Accelerator (HPC-ADA) along with MATLAB based software based on time series acoustic signal Detection cLassification using Machine learning Algorithms,…
▽ More
Goals of this research phase is to investigate advanced detection and classification pardims useful for data-mining passive large passive acoustic archives. Technical objectives are to develop and refine a High Performance Computing, Acoustic Data Accelerator (HPC-ADA) along with MATLAB based software based on time series acoustic signal Detection cLassification using Machine learning Algorithms, called DeLMA. Data scientists and biologists integrate to use the HPC-ADA and DeLMA technologies to explore data using newly developed techniques aimed at inspection of data extracted at large spatial and temporal scales.
△ Less
Submitted 5 May, 2016; v1 submitted 3 May, 2016;
originally announced May 2016.
-
Phase 4: DCL System Using Deep Learning Approaches for Land-Based or Ship-Based Real-Time Recognition and Localization of Marine Mammals - Distributed Processing and Big Data Applications
Authors:
Peter J. Dugan,
Christopher W. Clark,
Yann André LeCun,
Sofie M. Van Parijs
Abstract:
While the animal bioacoustics community at large is collecting huge amounts of acoustic data at an unprecedented pace, processing these data is problematic. Currently in bioacoustics, there is no effective way to achieve high performance computing using commericial off the shelf (COTS) or government off the shelf (GOTS) tools. Although several advances have been made in the open source and commerc…
▽ More
While the animal bioacoustics community at large is collecting huge amounts of acoustic data at an unprecedented pace, processing these data is problematic. Currently in bioacoustics, there is no effective way to achieve high performance computing using commericial off the shelf (COTS) or government off the shelf (GOTS) tools. Although several advances have been made in the open source and commercial software community, these offerings either support specific applications that do not integrate well with data formats in bioacoustics or they are too general. Furthermore, complex algorithms that use deep learning strategies require special considerations, such as very large libraiers of exemplars (whale sounds) readily available for algorithm training and testing. Detection-classification for passive acoustics is a data-mining strategy and our goals are aligned with best practices that appeal to the general data mining and machine learning communities where the problem of processing large data is common. Therefore, the objective of this work is to advance the state-of-the art for data-mining large passive acoustic datasets as they pertain to bioacoustics. With this basic deficiency recognized at the forefront, portions of the grant were dedicated to fostering deep-learning by way of international competitions (kaggle.com) meant to attract deep-learning solutions. The focus of this early work was targeted to make significant progress in addressing big data systems and advanced algorithms over the duration of the grant from 2012 to 2015. This early work provided simulataneous advances in systems-algorithms research while supporting various collaborations and projects.
△ Less
Submitted 5 May, 2016; v1 submitted 3 May, 2016;
originally announced May 2016.
-
Phase 2: DCL System Using Deep Learning Approaches for Land-based or Ship-based Real-Time Recognition and Localization of Marine Mammals - Machine Learning Detection Algorithms
Authors:
Peter J. Dugan,
Christopher W. Clark,
Yann André LeCun,
Sofie M. Van Parijs
Abstract:
Overarching goals for this work aim to advance the state of the art for detection, classification and localization (DCL) in the field of bioacoustics. This goal is primarily achieved by building a generic framework for detection-classification (DC) using a fast, efficient and scalable architecture, demonstrating the capabilities of this system using on a variety of low-frequency mid-frequency ceta…
▽ More
Overarching goals for this work aim to advance the state of the art for detection, classification and localization (DCL) in the field of bioacoustics. This goal is primarily achieved by building a generic framework for detection-classification (DC) using a fast, efficient and scalable architecture, demonstrating the capabilities of this system using on a variety of low-frequency mid-frequency cetacean sounds. Two primary goals are to develop transferable technologies for detection and classification in, one: the area of advanced algorithms, such as deep learning and other methods; and two: advanced systems, capable of real-time and archival processing. For each key area, we will focus on producing publications from this work and providing tools and software to the community where/when possible. Currently massive amounts of acoustic data are being collected by various institutions, corporations and national defense agencies. The long-term goal is to provide technical capability to analyze the data using automatic algorithms for (DC) based on machine intelligence. The goal of the automation is to provide effective and efficient mechanisms by which to process large acoustic datasets for understanding the bioacoustic behaviors of marine mammals. This capability will provide insights into the potential ecological impacts and influences of anthropogenic ocean sounds. This work focuses on building technologies using a maturity model based on DARPA 6.1 and 6.2 processes, for basic and applied research, respectively.
△ Less
Submitted 5 May, 2016; v1 submitted 3 May, 2016;
originally announced May 2016.
-
Phase 1: DCL System Research Using Advanced Approaches for Land-based or Ship-based Real-Time Recognition and Localization of Marine Mammals - HPC System Implementation
Authors:
Peter J. Dugan,
Christopher W. Clark,
Yann André LeCun,
Sofie M. Van Parijs
Abstract:
We aim to investigate advancing the state of the art of detection, classification and localization (DCL) in the field of bioacoustics. The two primary goals are to develop transferable technologies for detection and classification in: (1) the area of advanced algorithms, such as deep learning and other methods; and (2) advanced systems, capable of real-time and archival and processing. This projec…
▽ More
We aim to investigate advancing the state of the art of detection, classification and localization (DCL) in the field of bioacoustics. The two primary goals are to develop transferable technologies for detection and classification in: (1) the area of advanced algorithms, such as deep learning and other methods; and (2) advanced systems, capable of real-time and archival and processing. This project will focus on long-term, continuous datasets to provide automatic recognition, minimizing human time to annotate the signals. Effort will begin by focusing on several years of multi-channel acoustic data collected in the Stellwagen Bank National Marine Sanctuary (SBNMS) between 2006 and 2010. Our efforts will incorporate existing technologies in the bioacoustics signal processing community, advanced high performance computing (HPC) systems, and new approaches aimed at automatically detecting-classifying and measuring features for species-specific marine mammal sounds within passive acoustic data.
△ Less
Submitted 5 May, 2016; v1 submitted 3 May, 2016;
originally announced May 2016.
-
High Performance Computer Acoustic Data Accelerator: A New System for Exploring Marine Mammal Acoustics for Big Data Applications
Authors:
Peter Dugan,
John Zollweg,
Marian Popescu,
Denise Risch,
Herve Glotin,
Yann LeCun,
and Christopher Clark
Abstract:
This paper presents a new software model designed for distributed sonic signal detection runtime using machine learning algorithms called DeLMA. A new algorithm--Acoustic Data-mining Accelerator (ADA)--is also presented. ADA is a robust yet scalable solution for efficiently processing big sound archives using distributing computing technologies. Together, DeLMA and the ADA algorithm provide a powe…
▽ More
This paper presents a new software model designed for distributed sonic signal detection runtime using machine learning algorithms called DeLMA. A new algorithm--Acoustic Data-mining Accelerator (ADA)--is also presented. ADA is a robust yet scalable solution for efficiently processing big sound archives using distributing computing technologies. Together, DeLMA and the ADA algorithm provide a powerful tool currently being used by the Bioacoustics Research Program (BRP) at the Cornell Lab of Ornithology, Cornell University. This paper provides a high level technical overview of the system, and discusses various aspects of the design. Basic runtime performance and project summary are presented. The DeLMA-ADA baseline performance comparing desktop serial configuration to a 64 core distributed HPC system shows as much as a 44 times faster increase in runtime execution. Performance tests using 48 cores on the HPC shows a 9x to 12x efficiency over a 4 core desktop solution. Project summary results for 19 east coast deployments show that the DeLMA-ADA solution has processed over three million channel hours of sound to date.
△ Less
Submitted 11 September, 2015;
originally announced September 2015.
-
Bioacoustic Signal Classification Based on Continuous Region Processing, Grid Masking and Artificial Neural Network
Authors:
Mohammad Pourhomayoun,
Peter Dugan,
Marian Popescu,
Christopher Clark
Abstract:
In this paper, we develop a novel method based on machine-learning and image processing to identify North Atlantic right whale (NARW) up-calls in the presence of high levels of ambient and interfering noise. We apply a continuous region algorithm on the spectrogram to extract the regions of interest, and then use grid masking techniques to generate a small feature set that is then used in an artif…
▽ More
In this paper, we develop a novel method based on machine-learning and image processing to identify North Atlantic right whale (NARW) up-calls in the presence of high levels of ambient and interfering noise. We apply a continuous region algorithm on the spectrogram to extract the regions of interest, and then use grid masking techniques to generate a small feature set that is then used in an artificial neural network classifier to identify the NARW up-calls. It is shown that the proposed technique is effective in detecting and capturing even very faint up-calls, in the presence of ambient and interfering noises. The method is evaluated on a dataset recorded in Massachusetts Bay, United States. The dataset includes 20000 sound clips for training, and 10000 sound clips for testing. The results show that the proposed technique can achieve an error rate of less than FPR = 4.5% for a 90% true positive rate.
△ Less
Submitted 17 June, 2013; v1 submitted 15 May, 2013;
originally announced May 2013.
-
Classification for Big Dataset of Bioacoustic Signals Based on Human Scoring System and Artificial Neural Network
Authors:
Mohammad Pourhomayoun,
Peter Dugan,
Marian Popescu,
Denise Risch,
Hal Lewis,
Christopher Clark
Abstract:
In this paper, we propose a method to improve sound classification performance by combining signal features, derived from the time-frequency spectrogram, with human perception. The method presented herein exploits an artificial neural network (ANN) and learns the signal features based on the human perception knowledge. The proposed method is applied to a large acoustic dataset containing 24 months…
▽ More
In this paper, we propose a method to improve sound classification performance by combining signal features, derived from the time-frequency spectrogram, with human perception. The method presented herein exploits an artificial neural network (ANN) and learns the signal features based on the human perception knowledge. The proposed method is applied to a large acoustic dataset containing 24 months of nearly continuous recordings. The results show a significant improvement in performance of the detection-classification system; yielding as much as 20% improvement in true positive rate for a given false positive rate.
△ Less
Submitted 17 June, 2013; v1 submitted 15 May, 2013;
originally announced May 2013.
-
Bioacoustical Periodic Pulse Train Signal Detection and Classification using Spectrogram Intensity Binarization and Energy Projection
Authors:
Marian Popescu,
Peter J. Dugan,
Mohammad Pourhomayoun,
Denise Risch,
Harold W. Lewis III,
Christopher W. Clark
Abstract:
The following work outlines an approach for automatic detection and recognition of periodic pulse train signals using a multi-stage process based on spectrogram edge detection, energy projection and classification. The method has been implemented to automatically detect and recognize pulse train songs of minke whales. While the long term goal of this work is to properly identify and detect minke s…
▽ More
The following work outlines an approach for automatic detection and recognition of periodic pulse train signals using a multi-stage process based on spectrogram edge detection, energy projection and classification. The method has been implemented to automatically detect and recognize pulse train songs of minke whales. While the long term goal of this work is to properly identify and detect minke songs from large multi-year datasets, this effort was developed using sounds off the coast of Massachusetts, in the Stellwagen Bank National Marine Sanctuary. The detection methodology is presented and evaluated on 232 continuous hours of acoustic recordings and a qualitative analysis of machine learning classifiers and their performance is described. The trained automatic detection and classification system is applied to 120 continuous hours, comprised of various challenges such as broadband and narrowband noises, low SNR, and other pulse train signatures. This automatic system achieves a TPR of 63% for FPR of 0.6% (or 0.87 FP/h), at a Precision (PPV) of 84% and an F1 score of 71%.
△ Less
Submitted 28 June, 2013; v1 submitted 14 May, 2013;
originally announced May 2013.
