-
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
Authors:
Prabhat Pandey,
Rupak Vignesh Swaminathan,
K V Vijay Girish,
Arunasish Sen,
Jian Xie,
Grant P. Strimel,
Andreas Schwarz
Abstract:
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range…
▽ More
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs.
△ Less
Submitted 17 April, 2025; v1 submitted 12 April, 2025;
originally announced April 2025.
-
Adaptive dictionary based approach for background noise and speaker classification and subsequent source separation
Authors:
K V Vijay Girish,
A G Ramakrishnan,
T V Ananthapadmanabha
Abstract:
A judicious combination of dictionary learning methods, block sparsity and source recovery algorithm are used in a hierarchical manner to identify the noises and the speakers from a noisy conversation between two people. Conversations are simulated using speech from two speakers, each with a different background noise, with varied SNR values, down to -10 dB. Ten each of randomly chosen male and fe…
▽ More
A judicious combination of dictionary learning methods, block sparsity and source recovery algorithm are used in a hierarchical manner to identify the noises and the speakers from a noisy conversation between two people. Conversations are simulated using speech from two speakers, each with a different background noise, with varied SNR values, down to -10 dB. Ten each of randomly chosen male and female speakers from the TIMIT database and all the noise sources from the NOISEX database are used for the simulations. For speaker identification, the relative value of weights recovered is used to select an appropriately small subset of the test data, assumed to contain speech. This novel choice of using varied amounts of test data results in an improvement in the speaker recognition rate of around 15% at SNR of 0 dB. Speech and noise are separated using dictionaries of the estimated speaker and noise, and an improvement of signal to distortion ratios of up to 10% is achieved at SNR of 0 dB. K-medoid and cosine similarity based dictionary learning methods lead to better recognition of the background noise and the speaker. Experiments are also conducted on cases, where either the background noise or the speaker is outside the set of trained dictionaries. In such cases, adaptive dictionary learning leads to performance comparable to the other case of complete dictionaries.
△ Less
Submitted 28 October, 2016; v1 submitted 30 September, 2016;
originally announced September 2016.
-
A dictionary learning and source recovery based approach to classify diverse audio sources
Authors:
K V Vijay Girish,
T V Ananthapadmanabha,
A G Ramakrishnan
Abstract:
A dictionary learning based audio source classification algorithm is proposed to classify a sample audio signal as one amongst a finite set of different audio sources. Cosine similarity measure is used to select the atoms during dictionary learning. Based on three objective measures proposed, namely, signal to distortion ratio (SDR), the number of non-zero weights and the sum of weights, a frame-w…
▽ More
A dictionary learning based audio source classification algorithm is proposed to classify a sample audio signal as one amongst a finite set of different audio sources. Cosine similarity measure is used to select the atoms during dictionary learning. Based on three objective measures proposed, namely, signal to distortion ratio (SDR), the number of non-zero weights and the sum of weights, a frame-wise source classification accuracy of 98.2% is obtained for twelve different sources. Cent percent accuracy has been obtained using moving SDR accumulated over six successive frames for ten of the audio sources tested, while the two other sources require accumulation of 10 and 14 frames.
△ Less
Submitted 27 October, 2015;
originally announced October 2015.
-
Detection of transitions between broad phonetic classes in a speech signal
Authors:
T V Ananthapadmanabha,
K V Vijay Girish,
A G Ramakrishnan
Abstract:
Detection of transitions between broad phonetic classes in a speech signal is an important problem which has applications such as landmark detection and segmentation. The proposed hierarchical method detects silence to non-silence transitions, high amplitude (mostly sonorants) to low ampli- tude (mostly fricatives/affricates/stop bursts) transitions and vice-versa. A subset of the extremum (minimu…
▽ More
Detection of transitions between broad phonetic classes in a speech signal is an important problem which has applications such as landmark detection and segmentation. The proposed hierarchical method detects silence to non-silence transitions, high amplitude (mostly sonorants) to low ampli- tude (mostly fricatives/affricates/stop bursts) transitions and vice-versa. A subset of the extremum (minimum or maximum) samples between every pair of successive zero-crossings is selected above a second pass threshold, from each bandpass filtered speech signal frame. Relative to the mid-point (reference) of a frame, locations of the first and the last extrema lie on either side, if the speech signal belongs to a homogeneous segment; else, both these locations lie on the left or the right side of the reference, indicating a transition frame. When tested on the entire TIMIT database, of the transitions detected, 93.6% are within a tolerance of 20 ms from the hand labeled boundaries. Sonorant, unvoiced non-sonorant and silence classes and their respective onsets are detected with an accuracy of about 83.5% for the same tolerance. The results are as good as, and in some respects better than the state-of-the-art methods for similar tasks.
△ Less
Submitted 3 November, 2014;
originally announced November 2014.
