-
Humanity's Last Exam
Authors:
Long Phan,
Alice Gatti,
Ziwen Han,
Nathaniel Li,
Josephina Hu,
Hugh Zhang,
Chen Bo Calvin Zhang,
Mohamed Shaaban,
John Ling,
Sean Shi,
Michael Choi,
Anish Agrawal,
Arnav Chopra,
Adam Khoja,
Ryan Kim,
Richard Ren,
Jason Hausenloy,
Oliver Zhang,
Mantas Mazeika,
Dmitry Dodonov,
Tung Nguyen,
Jaeho Lee,
Daron Anderson,
Mikhail Doroshenko,
Alun Cennyth Stokes
, et al. (1084 additional authors not shown)
Abstract:
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of…
▽ More
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
△ Less
Submitted 19 April, 2025; v1 submitted 24 January, 2025;
originally announced January 2025.
-
Artificially Synthesising Data for Audio Classification and Segmentation to Improve Speech and Music Detection in Radio Broadcast
Authors:
Satvik Venkatesh,
David Moffat,
Alexis Kirke,
Gözel Shakeri,
Stephen Brewster,
Jörg Fachner,
Helen Odell-Miller,
Alex Street,
Nicolas Farina,
Sube Banerjee,
Eduardo Reck Miranda
Abstract:
Segmenting audio into homogeneous sections such as music and speech helps us understand the content of audio. It is useful as a pre-processing step to index, store, and modify audio recordings, radio broadcasts and TV programmes. Deep learning models for segmentation are generally trained on copyrighted material, which cannot be shared. Annotating these datasets is time-consuming and expensive and…
▽ More
Segmenting audio into homogeneous sections such as music and speech helps us understand the content of audio. It is useful as a pre-processing step to index, store, and modify audio recordings, radio broadcasts and TV programmes. Deep learning models for segmentation are generally trained on copyrighted material, which cannot be shared. Annotating these datasets is time-consuming and expensive and therefore, it significantly slows down research progress. In this study, we present a novel procedure that artificially synthesises data that resembles radio signals. We replicate the workflow of a radio DJ in mixing audio and investigate parameters like fade curves and audio ducking. We trained a Convolutional Recurrent Neural Network (CRNN) on this synthesised data and outperformed state-of-the-art algorithms for music-speech detection. This paper demonstrates the data synthesis procedure as a highly effective technique to generate large datasets to train deep neural networks for audio segmentation.
△ Less
Submitted 19 February, 2021;
originally announced February 2021.
-
Analysis of the Q^2-dependence of charged-current quasielastic processes in neutrino-nucleus interactions
Authors:
Artur M. Ankowski,
Omar Benhar,
Nicola Farina
Abstract:
We discuss the observed disagreement between the Q^2 distributions of neutrino-nucleus quasielastic events, measured by a number of recent experiments, and the predictions of Monte Carlo simulations based on the relativistic Fermi gas model. The results of our analysis suggest that these discrepancies are likely to be ascribable to both the breakdown of the impulse approximation and the limitation…
▽ More
We discuss the observed disagreement between the Q^2 distributions of neutrino-nucleus quasielastic events, measured by a number of recent experiments, and the predictions of Monte Carlo simulations based on the relativistic Fermi gas model. The results of our analysis suggest that these discrepancies are likely to be ascribable to both the breakdown of the impulse approximation and the limitations of the Fermi gas description. Several issues related to the extraction of the Q^2 distributions from the experimental data are also discussed, and new kinematical variables, which would allow for an improved analysis, are proposed.
△ Less
Submitted 12 July, 2010; v1 submitted 4 January, 2010;
originally announced January 2010.
-
Correlation effects on the weak response of nuclear matter
Authors:
Omar Benhar,
Nicola Farina
Abstract:
The consistent description of the nuclear response at low and high momentum transfer requires a unified dynamical model, suitable to account for both short- and long-range correlation effects. We report the results of a study of the charged current weak response of symmetric nuclear matter, carried out using an effective interaction obtained from a realistic model of the nucleon-nucleon force wi…
▽ More
The consistent description of the nuclear response at low and high momentum transfer requires a unified dynamical model, suitable to account for both short- and long-range correlation effects. We report the results of a study of the charged current weak response of symmetric nuclear matter, carried out using an effective interaction obtained from a realistic model of the nucleon-nucleon force within the formalism of correlated basis functions. Our approach allows for a clear identification of the kinematical regions in which different interaction effects dominate.
△ Less
Submitted 15 April, 2009;
originally announced April 2009.
-
Weak Response of Nuclear Matter at low Momentum transfer
Authors:
Nicola Farina
Abstract:
A quantitative understanding of the weak nuclear response is a prerequisite for the computer simulations of astrophysical phenomena like supernov$æ$ explosions and neutron star cooling. In order to reduce the systematic uncertainties associated with the simulations, a consistent framework, able to take into account dynamical correlation effects, is needed to compute neutrino-nucleon and neutrino…
▽ More
A quantitative understanding of the weak nuclear response is a prerequisite for the computer simulations of astrophysical phenomena like supernov$æ$ explosions and neutron star cooling. In order to reduce the systematic uncertainties associated with the simulations, a consistent framework, able to take into account dynamical correlation effects, is needed to compute neutrino-nucleon and neutrino-nucleus reaction rates. In this paper we describe the many-body theory of the weak nuclear response at low energy regime. We show how to include both short and long correlations effects in a consistent fashion.
△ Less
Submitted 17 February, 2009;
originally announced February 2009.
-
Weak Response of Nuclear Matter
Authors:
Nicola Farina
Abstract:
The quantitative understanding of neutrino interactions with nuclei and nuclear matter is needed to the study of many different problems. In the astrophysics environment, neutrino-nucleon and neutrino-nucleus reaction rates are used as inputs in the simulations of phenomena like supernov$æ$ explosions and neutron star cooling. In the field of neutrino physics, the quantitative knowledge of neutr…
▽ More
The quantitative understanding of neutrino interactions with nuclei and nuclear matter is needed to the study of many different problems. In the astrophysics environment, neutrino-nucleon and neutrino-nucleus reaction rates are used as inputs in the simulations of phenomena like supernov$æ$ explosions and neutron star cooling. In the field of neutrino physics, the quantitative knowledge of neutrino-nucleus cross-section is critical to reduce the systematic uncertainty of the long baseline oscillation experiments.
It is important to realize that, while neutrinos interacting in stellar matter typically have energies of the order of few MeV, the energies involved in long baseline oscillations experiments are much larger. For example, K2K experiment takes data in the region $E_ν =0.5-3$ GeV.
In this thesis, we describe how nuclear many-body theory provide a scheme allowing for a consistent treatment of neutrino-nucleus interactions at both high and low energies. We will show our predictions of the neutrino-nucleus cross section in the high energy regime and the results of our calculations for the nuclear matter weak response in the low energy regime.
△ Less
Submitted 16 January, 2009;
originally announced January 2009.
-
Unified description of equation of state and transport properties of nuclear matter
Authors:
Omar Benhar,
Nicola Farina,
Salvatore Fiorilla,
Marco Valli
Abstract:
Correlated basis function perturbation theory and the formalism of cluster expansions have been recently employed to obtain an effective interaction from a state-of-the-art nucleon nucleon potential model. The approach based on the effective interaction allows for a consistent description of the nuclear matter ground state and nucleon-nucleon scattering in the nuclear medium. This paper reports…
▽ More
Correlated basis function perturbation theory and the formalism of cluster expansions have been recently employed to obtain an effective interaction from a state-of-the-art nucleon nucleon potential model. The approach based on the effective interaction allows for a consistent description of the nuclear matter ground state and nucleon-nucleon scattering in the nuclear medium. This paper reports the the results of numerical calculations of different properties of nuclear and neutron matter, including the equation of state and the shear viscosity and thermal conductivity transport coefficients, carried out using the effective interaction.
△ Less
Submitted 15 July, 2008;
originally announced July 2008.
-
Lepton-nucleus scattering in the impulse approximation regime
Authors:
O. Benhar,
N. Farina,
H. Nakamura,
M. Sakuda,
R. Seki
Abstract:
We discuss theoretical calculations of electron- and neutrino-nucleus scattering, carried out using realistic nuclear spectral functions and including the effect of final state interactions. Comparison between electron scattering data and the calculated inclusive cross sections off oxygen shows that the Fermi gas model fails to provide a satisfactory description of the measured cross sections, a…
▽ More
We discuss theoretical calculations of electron- and neutrino-nucleus scattering, carried out using realistic nuclear spectral functions and including the effect of final state interactions. Comparison between electron scattering data and the calculated inclusive cross sections off oxygen shows that the Fermi gas model fails to provide a satisfactory description of the measured cross sections, and inclusion of nuclear dynamics is needed. The role of Pauli blocking in charged-current neutrino induced reactions at low $Q^2$ is also analyzed.
△ Less
Submitted 20 October, 2005;
originally announced October 2005.
-
Electron- and neutrino-nucleus scattering in the impulse approximation regime
Authors:
Omar Benhar,
Nicola Farina,
Hiroki Nakamura,
Makoto Sakuda,
Ryoichi Seki
Abstract:
A quantitative understanding of the weak nuclear response is a prerequisite for the analyses of neutrino experiments such as K2K and MiniBOONE, which measure energy and angle of the muons produced in neutrino-nucleus interactions in the energy range $0.5-3$ GeV and reconstruct the incident neutrino energy to determine neutrino oscillations. In this paper we discuss theoretical calculations of el…
▽ More
A quantitative understanding of the weak nuclear response is a prerequisite for the analyses of neutrino experiments such as K2K and MiniBOONE, which measure energy and angle of the muons produced in neutrino-nucleus interactions in the energy range $0.5-3$ GeV and reconstruct the incident neutrino energy to determine neutrino oscillations. In this paper we discuss theoretical calculations of electron- and neutrino-nucleus scattering, carried out within the impulse approximation scheme using realistic nuclear spectral functions.Comparison between electron scattering data and the calculated inclusive cross section off oxygen, at beam energies ranging between 700 and 1200 MeV, show that the Fermi gas model, widely used in the analysis of neutrino oscillation experiments,fails to provide a satisfactory description of the measured cross sections,and inclusion of nuclear dynamics is needed.
△ Less
Submitted 13 June, 2005;
originally announced June 2005.
-
Neutrino-nucleus cross section in the impulse approximation regime
Authors:
Omar Benhar,
Nicola Farina
Abstract:
In the impulse approximation regime the nuclear response to a weakly interacting probe can be written in terms of the measured nucleon structure fuctions and the target spectral function, yielding the energy and momentum distribution of the constituent nucleons. We discuss a calculation of charged current neutrino-oxygen interactions in the quasielastic channel, carried out within nuclear many b…
▽ More
In the impulse approximation regime the nuclear response to a weakly interacting probe can be written in terms of the measured nucleon structure fuctions and the target spectral function, yielding the energy and momentum distribution of the constituent nucleons. We discuss a calculation of charged current neutrino-oxygen interactions in the quasielastic channel, carried out within nuclear many body theory. The proposed approach, extensively and successfully employed in the analysys of electron-nucleus scattering data, allows for a parameter free prediction of the neutrino-nucleus cross section, whose quantitative understanding will be critical to the analysis of the next genaration of high precision neutrino oscillation experiments.
△ Less
Submitted 29 July, 2004;
originally announced July 2004.
