-
Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
Authors:
Andrew H. Lee,
Sina J. Semnani,
Galo Castillo-López,
Gäel de Chalendar,
Monojit Choudhury,
Ashna Dua,
Kapil Rajesh Kavitha,
Sungkyun Kim,
Prashant Kodali,
Ponnurangam Kumaraguru,
Alexis Lombard,
Mehrad Moradshahi,
Gihyun Park,
Nasredine Semmar,
Jiwon Seo,
Tianhao Shen,
Manish Shrivastava,
Deyi Xiong,
Monica S. Lam
Abstract:
Creating multilingual task-oriented dialogue (TOD) agents is challenging due to the high cost of training data acquisition. Following the research trend of improving training data efficiency, we show for the first time, that in-context learning is sufficient to tackle multilingual TOD.
To handle the challenging dialogue state tracking (DST) subtask, we break it down to simpler steps that are mor…
▽ More
Creating multilingual task-oriented dialogue (TOD) agents is challenging due to the high cost of training data acquisition. Following the research trend of improving training data efficiency, we show for the first time, that in-context learning is sufficient to tackle multilingual TOD.
To handle the challenging dialogue state tracking (DST) subtask, we break it down to simpler steps that are more compatible with in-context learning where only a handful of few-shot examples are used. We test our approach on the multilingual TOD dataset X-RiSAWOZ, which has 12 domains in Chinese, English, French, Korean, Hindi, and code-mixed Hindi-English. Our turn-by-turn DST accuracy on the 6 languages range from 55.6% to 80.3%, seemingly worse than the SOTA results from fine-tuned models that achieve from 60.7% to 82.8%; our BLEU scores in the response generation (RG) subtask are also significantly lower than SOTA.
However, after manual evaluation of the validation set, we find that by correcting gold label errors and improving dataset annotation schema, GPT-4 with our prompts can achieve (1) 89.6%-96.8% accuracy in DST, and (2) more than 99% correct response generation across different languages. This leads us to conclude that current automatic metrics heavily underestimate the effectiveness of in-context learning.
△ Less
Submitted 16 June, 2024; v1 submitted 28 May, 2024;
originally announced May 2024.
-
Direction-dependent effects on global 21-cm detection
Authors:
Yash Agrawal,
K. Kavitha,
Saurabh Singh
Abstract:
Cosmic dawn represents critical juncture in cosmic history when the first population of stars emerged. The astrophysical processes that govern this transformation need to be better understood. The detection of redshifted 21-cm radiation emitted from neutral hydrogen during this era offers a direct window into the thermal and ionization state of the universe. This emission manifests as differential…
▽ More
Cosmic dawn represents critical juncture in cosmic history when the first population of stars emerged. The astrophysical processes that govern this transformation need to be better understood. The detection of redshifted 21-cm radiation emitted from neutral hydrogen during this era offers a direct window into the thermal and ionization state of the universe. This emission manifests as differential brightness between spin temperature and the cosmic microwave background (CMB). SARAS experiment aims to detect the sky-averaged signal in the frequency range 40-200 MHz. SARAS's unique design and operation strategy to float the antenna over a water body minimizes spectral features that may arise due to stratified ground beneath the antenna. However, the antenna environment can be prone to configuration changes due to variations in critical design parameters such as conductivity and antenna tilts. In this paper, we connect the variations in antenna properties to signal detection prospects. By using realistic simulations of a direction and frequency-dependent radiation pattern of the SARAS antenna and its transfer function, we establish critical parameters and estimate bias in the detectability of different models of the global 21-cm signal. We find a correlation between the nature of chromaticity in antenna properties and the bias in the recovered spectral profiles of 21-cm signals. We also find stringent requirements for transfer function corrections, which can otherwise make detection prospects prohibitive. We finally explore a range of critical parameters that allow robust signal detection.
△ Less
Submitted 12 April, 2025; v1 submitted 19 January, 2024;
originally announced January 2024.
-
A Floating Octave Bandwidth Cone-Disc Antenna for Detection of Cosmic Dawn
Authors:
Agaram Raghunathan,
Ravi Subrahmanyan,
N. Udaya Shankar,
Saurabh Singh,
Jishnu Nambissan,
K. Kavitha,
Nivedita Mahesh,
R. Somashekar,
Gaddam Sindhu,
B. S. Girish,
K. S. Srivani,
Mayuri S. Rao
Abstract:
The critical component of radio astronomy radiometers built to detect redshifted 21-cm signals from Cosmic Dawn is the antenna element. We describe the design and performance of an octave bandwidth cone disc antenna built to detect this signal in the band 40 to 90 MHz. The Cosmic Dawn signal is predicted to be a wideband spectral feature orders of magnitude weaker than sky and ground radio brightn…
▽ More
The critical component of radio astronomy radiometers built to detect redshifted 21-cm signals from Cosmic Dawn is the antenna element. We describe the design and performance of an octave bandwidth cone disc antenna built to detect this signal in the band 40 to 90 MHz. The Cosmic Dawn signal is predicted to be a wideband spectral feature orders of magnitude weaker than sky and ground radio brightness. Thus, the engineering challenge is to design an antenna at low frequencies that is able to provide with high fidelity the faint cosmological signal, along with foreground sky, to the receiver. The antenna characteristics must not compromise detection by imprinting any confusing spectral features on the celestial radiation, ground emission or receiver noise. An innovation in the present design is making the antenna electrically smaller than half wavelength and operating it on the surface of a sufficiently large water body. The homogeneous and high permittivity medium beneath the small cone-disc antenna results in an achromatic beam pattern, high radiation efficiency and minimum unwanted confusing spectral features. The antenna design was optimized in WIPL-D and FEKO. A prototype was constructed and deployed on a lake to validate its performance with field measurements.
Index Terms: Antenna measurements, radio astronomy, reflector antennas.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
Evaluation of Distance Measures for Feature based Image Registration using AlexNet
Authors:
K. Kavitha,
B. Thirumala Rao
Abstract:
Image registration is a classic problem of computer vision with several applications across areas like defence, remote sensing, medicine etc. Feature based image registration methods traditionally used hand-crafted feature extraction algorithms, which detect key points in an image and describe them using a region around the point. Such features are matched using a threshold either on distances or…
▽ More
Image registration is a classic problem of computer vision with several applications across areas like defence, remote sensing, medicine etc. Feature based image registration methods traditionally used hand-crafted feature extraction algorithms, which detect key points in an image and describe them using a region around the point. Such features are matched using a threshold either on distances or ratio of distances computed between the feature descriptors. Evolution of deep learning, in particular convolution neural networks, has enabled researchers to address several problems of vision such as recognition, tracking, localization etc. Outputs of convolution layers or fully connected layers of CNN which has been trained for applications like visual recognition are proved to be effective when used as features in other applications such as retrieval. In this work, a deep CNN, AlexNet, is used in the place of handcrafted features for feature extraction in the first stage of image registration. However, there is a need to identify a suitable distance measure and a matching method for effective results. Several distance metrics have been evaluated in the framework of nearest neighbour and nearest neighbour ratio matching methods using benchmark dataset. Evaluation is done by comparing matching and registration performance using metrics computed from ground truth.
Keywords: Distance measures; deep learning; feature detection; feature descriptor; image matching
△ Less
Submitted 20 July, 2019;
originally announced July 2019.
