-
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
Authors:
Marek Kadlčík,
Michal Štefánik,
Timothee Mickus,
Michal Spiegel,
Josef Kuchař
Abstract:
Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure…
▽ More
Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns.
In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings' preciseness judged by our probe's accuracy explains a large portion of LM's errors in elementary arithmetic, and show that aligning the embeddings with the pattern discovered by our probe can mitigate these errors.
△ Less
Submitted 10 June, 2025;
originally announced June 2025.
-
Negation: A Pink Elephant in the Large Language Models' Room?
Authors:
Tereza Vrabcová,
Marek Kadlčík,
Petr Sojka,
Michal Štefánik,
Michal Spiegel
Abstract:
Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language models (LLMs) and remain underexplored.
We constructed and published two new textual entailment datasets NoFEVER-ML and NoSNLI-ML in four languages (English, Czech, German, and Ukrainian) with
examples differing in n…
▽ More
Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language models (LLMs) and remain underexplored.
We constructed and published two new textual entailment datasets NoFEVER-ML and NoSNLI-ML in four languages (English, Czech, German, and Ukrainian) with
examples differing in negation. It allows investigation of the root causes of the negation problem and its exemplification: how popular LLM model properties and language impact their inability to handle negation correctly.
Contrary to previous work, we show that increasing the model size may improve the models' ability to handle negations. Furthermore, we find that both the models' reasoning accuracy and robustness to negation are language-dependent and that the length and explicitness of the premise have an impact on robustness. There is better accuracy in projective language with fixed order, such as English, than in non-projective ones, such as German or Czech.
Our entailment datasets pave the way to further research for explanation and exemplification of the negation problem, minimization of LLM hallucinations, and improvement of LLM reasoning in multilingual settings.
△ Less
Submitted 3 June, 2025; v1 submitted 28 March, 2025;
originally announced March 2025.
-
Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
Authors:
Michal Spiegel,
Michal Štefánik,
Marek Kadlčík,
Josef Kuchař
Abstract:
Can transformers learn to perform algorithmic tasks reliably across previously unseen input/output domains? While pre-trained language models show solid accuracy on benchmarks incorporating algorithmic reasoning, assessing the reliability of these results necessitates an ability to cleanse models' functional capabilities from memorization. In this paper, we propose an algorithmic benchmark compris…
▽ More
Can transformers learn to perform algorithmic tasks reliably across previously unseen input/output domains? While pre-trained language models show solid accuracy on benchmarks incorporating algorithmic reasoning, assessing the reliability of these results necessitates an ability to cleanse models' functional capabilities from memorization. In this paper, we propose an algorithmic benchmark comprising six tasks of infinite input domains where we can also disentangle and trace the correct, robust algorithm necessary for the task. This allows us to assess (i) models' ability to extrapolate to unseen types of inputs, including new lengths, value ranges or input domains, but also (ii) to assess the robustness of the functional mechanism in recent models through the lens of their attention maps. We make the implementation of all our tasks and interoperability methods publicly available at https://github.com/michalspiegel/AttentionSpan .
△ Less
Submitted 28 February, 2025;
originally announced March 2025.
-
KInIT at SemEval-2024 Task 8: Fine-tuned LLMs for Multilingual Machine-Generated Text Detection
Authors:
Michal Spiegel,
Dominik Macko
Abstract:
SemEval-2024 Task 8 is focused on multigenerator, multidomain, and multilingual black-box machine-generated text detection. Such a detection is important for preventing a potential misuse of large language models (LLMs), the newest of which are very capable in generating multilingual human-like texts. We have coped with this task in multiple ways, utilizing language identification and parameter-ef…
▽ More
SemEval-2024 Task 8 is focused on multigenerator, multidomain, and multilingual black-box machine-generated text detection. Such a detection is important for preventing a potential misuse of large language models (LLMs), the newest of which are very capable in generating multilingual human-like texts. We have coped with this task in multiple ways, utilizing language identification and parameter-efficient fine-tuning of smaller LLMs for text classification. We have further used the per-language classification-threshold calibration to uniquely combine fine-tuned models predictions with statistical detection metrics to improve generalization of the system detection performance. Our submitted method achieved competitive results, ranking at the fourth place, just under 1 percentage point behind the winner.
△ Less
Submitted 17 June, 2024; v1 submitted 21 February, 2024;
originally announced February 2024.
-
IMGTB: A Framework for Machine-Generated Text Detection Benchmarking
Authors:
Michal Spiegel,
Dominik Macko
Abstract:
In the era of large language models generating high quality texts, it is a necessity to develop methods for detection of machine-generated text to avoid harmful use or simply due to annotation purposes. It is, however, also important to properly evaluate and compare such developed methods. Recently, a few benchmarks have been proposed for this purpose; however, integration of newest detection meth…
▽ More
In the era of large language models generating high quality texts, it is a necessity to develop methods for detection of machine-generated text to avoid harmful use or simply due to annotation purposes. It is, however, also important to properly evaluate and compare such developed methods. Recently, a few benchmarks have been proposed for this purpose; however, integration of newest detection methods is rather challenging, since new methods appear each month and provide slightly different evaluation pipelines. In this paper, we present the IMGTB framework, which simplifies the benchmarking of machine-generated text detection methods by easy integration of custom (new) methods and evaluation datasets. Its configurability and flexibility makes research and development of new detection methods easier, especially their comparison to the existing state-of-the-art detectors. The default set of analyses, metrics and visualizations offered by the tool follows the established practices of machine-generated text detection benchmarking found in state-of-the-art literature.
△ Less
Submitted 21 November, 2023;
originally announced November 2023.
-
Post-CCSD(T) corrections to bond distances and vibrational frequencies: the power of $Λ$
Authors:
Maciej Spiegel,
Emmanouil Semidalas,
Jan M. L. Martin,
Megan R. Bentley,
John F. Stanton
Abstract:
The importance of post-CCSD(T) corrections as high as CCSDTQ56 for ground-state spectroscopic constants ($D_e$, $ω_e$, $ω_ex_e$, and $α_e$) has been surveyed for a sample of two dozen mostly heavy-atom diatomics spanning a broad range of static correlation strength. While CCSD(T) is known to be an unusually felicitous `Pauling point' between accuracy and computational cost, performance leaves some…
▽ More
The importance of post-CCSD(T) corrections as high as CCSDTQ56 for ground-state spectroscopic constants ($D_e$, $ω_e$, $ω_ex_e$, and $α_e$) has been surveyed for a sample of two dozen mostly heavy-atom diatomics spanning a broad range of static correlation strength. While CCSD(T) is known to be an unusually felicitous `Pauling point' between accuracy and computational cost, performance leaves something to be desired for molecules with strong static correlation. We find CCSDT(Q)$_Λ$ to be the next `sweet spot' up, of comparable or superior quality to the much more expensive CCSDTQ. A similar comparison applies to CCSDTQ(5)$_Λ$ vs. CCSDTQ5, while CCSDTQ5(6)$_Λ$ is essentially indistinguishable from CCSDTQ56. A composite of CCSD(T)-X2C/ACV5Z-X2C with [CCSDT(Q)$_Λ$ -- CCSD(T)]/cc-pVTZ or even cc-pVDZ basis sets appears highly effective for computational vibrational spectroscopy. Unlike CCSDT(Q) which breaks down for the ozone vibrational frequencies, CCSDT(Q)$_Λ$ handles them gracefully.
△ Less
Submitted 22 August, 2023; v1 submitted 27 July, 2023;
originally announced July 2023.
-
How Does Magnetic Reconnection Drive the Early Stage Evolution of Coronal Mass Ejections?
Authors:
Chunming Zhu,
Jiong Qiu,
Paulett Liewer,
Angelos Vourlidas,
Michael Spiegel,
Qiang Hu
Abstract:
Theoretically, CME kinematics are related to magnetic reconnection processes in the solar corona. However, the current quantitative understanding of this relationship is based on the analysis of only a handful of events. Here we report a statistical study of 60 CME-flare events from August 2010 to December 2013. We investigate kinematic properties of CMEs and magnetic reconnection in the low coron…
▽ More
Theoretically, CME kinematics are related to magnetic reconnection processes in the solar corona. However, the current quantitative understanding of this relationship is based on the analysis of only a handful of events. Here we report a statistical study of 60 CME-flare events from August 2010 to December 2013. We investigate kinematic properties of CMEs and magnetic reconnection in the low corona during the early phase of the eruptions, by combining limb observations from STEREO with simultaneous on-disk views from SDO. For a subset of 42 events with reconnection rate evaluated by the magnetic fluxes swept by the flare ribbons on the solar disk observed from SDO, we find a strong correlation between the peak CME acceleration and the peak reconnection rate. Also, the maximum velocities of relatively fast CMEs (> 600 km/s) are positively correlated with the reconnection flux, but no such correlation is found for slow CMEs. A time-lagged correlation analysis suggests that the distribution of the time lag of CME acceleration relative to reconnection rate exhibits three peaks, approximately 10 minutes apart, and on average, acceleration-lead events have smaller reconnection rates. We further compare the CME total mechanical energy with the estimated energy in the current sheet. The comparison suggests that, for small-flare events, reconnection in the current sheet alone is insufficient to fuel CMEs. Results from this study suggest that flare reconnection may dominate the acceleration of fast CMEs, but for events of slow CMEs and weak reconnection, other mechanisms may be more important.
△ Less
Submitted 24 March, 2020;
originally announced March 2020.
-
Scalar curvature rigidity for locally conformally flat manifolds with boundary
Authors:
Fabian Michael Spiegel
Abstract:
Inspired by the work of F. Hang and X. Wang and partial results by S. Raulot, we prove a scalar curvature rigitidy result for locally conformally flat manifolds with boundary in the spirit of the well-known Min-Oo conjecture.
Inspired by the work of F. Hang and X. Wang and partial results by S. Raulot, we prove a scalar curvature rigitidy result for locally conformally flat manifolds with boundary in the spirit of the well-known Min-Oo conjecture.
△ Less
Submitted 12 February, 2016; v1 submitted 19 November, 2015;
originally announced November 2015.
