-
Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
Authors:
Lujun Li,
Lama Sleem,
Niccolo' Gentile,
Geoffrey Nichil,
Radu State
Abstract:
The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperatur…
▽ More
The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperature, plays a crucial role in model inference. In this study, we systematically evaluated the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities, conducting statistical analyses on open source models of three different sizes: small (1B--4B), medium (6B--13B), and large (40B--80B). Our findings reveal distinct skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection in practical applications. To address this challenge, we propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt. We demonstrate that this approach can significantly improve the performance of small and medium models in the SuperGLUE datasets. Furthermore, our study extends to FP16 precision inference, revealing that temperature effects are consistent with those observed in 4-bit quantized models. By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature -- the point at which significant performance changes occur -- increases with model size.
△ Less
Submitted 8 June, 2025;
originally announced June 2025.
-
Small Language Models in the Real World: Insights from Industrial Text Classification
Authors:
Lujun Li,
Lama Sleem,
Niccolo' Gentile,
Geoffrey Nichil,
Radu State
Abstract:
With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance and flexibility, yet they suffer from inefficiency on inference due to token-by-token generation, and their effectiveness in text classification tasks heavily depends on prompt quality. Moreover, their substantial GPU reso…
▽ More
With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance and flexibility, yet they suffer from inefficiency on inference due to token-by-token generation, and their effectiveness in text classification tasks heavily depends on prompt quality. Moreover, their substantial GPU resource requirements often limit widespread adoption. Thus, the question of whether smaller language models are capable of effectively handling text classification tasks emerges as a topic of significant interest. However, the selection of appropriate models and methodologies remains largely underexplored. In this paper, we conduct a comprehensive evaluation of prompt engineering and supervised fine-tuning methods for transformer-based text classification. Specifically, we focus on practical industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. We examine the strengths and limitations of smaller models, with particular attention to both their performance and their efficiency in Video Random-Access Memory (VRAM) utilization, thereby providing valuable insights for the local deployment and application of compact models in industrial settings.
△ Less
Submitted 23 June, 2025; v1 submitted 21 May, 2025;
originally announced May 2025.
-
Is LLM the Silver Bullet to Low-Resource Languages Machine Translation?
Authors:
Yewei Song,
Lujun Li,
Cedric Lothritz,
Saad Ezzini,
Lama Sleem,
Niccolo Gentile,
Radu State,
Tegawendé F. Bissyandé,
Jacques Klein
Abstract:
Low-Resource Languages (LRLs) present significant challenges in natural language processing due to their limited linguistic resources and underrepresentation in standard datasets. While recent advances in Large Language Models (LLMs) and Neural Machine Translation have substantially improved translation capabilities for high-resource languages, performance disparities persist for LRLs, particularl…
▽ More
Low-Resource Languages (LRLs) present significant challenges in natural language processing due to their limited linguistic resources and underrepresentation in standard datasets. While recent advances in Large Language Models (LLMs) and Neural Machine Translation have substantially improved translation capabilities for high-resource languages, performance disparities persist for LRLs, particularly impacting privacy-sensitive and resource-constrained scenarios. This paper systematically evaluates current LLMs in 200 languages using the FLORES-200 benchmark and demonstrates their limitations in LRL translation capability. We also explore alternative data sources, including news articles and bilingual dictionaries, and demonstrate how knowledge distillation from large pre-trained teacher models can significantly improve the performance of small LLMs on LRL translation tasks. For example, this approach increases EN->LB with the LLM-as-a-Judge score on the validation set from 0.36 to 0.89 for Llama-3.2-3B. Furthermore, we examine different fine-tuning configurations, providing practical insights on optimal data scale, training efficiency, and the preservation of generalization capabilities of models under study.
△ Less
Submitted 5 June, 2025; v1 submitted 31 March, 2025;
originally announced March 2025.
-
A Revision of a New Chaos-Based Image Encryption System: Weaknesses and Limitations
Authors:
Hassan Noura,
Lama Sleem,
Raphaël Couturier
Abstract:
Lately, multimedia encryption has been the focus of attention in many researches. Recently, a large number of encryption algorithms has been presented to protect image contents.The main objective of modern image encryption schemes is to reduce the computation complexity in order to respond to the real time multimedia and/or limited resources requirements without degrading the high level of securit…
▽ More
Lately, multimedia encryption has been the focus of attention in many researches. Recently, a large number of encryption algorithms has been presented to protect image contents.The main objective of modern image encryption schemes is to reduce the computation complexity in order to respond to the real time multimedia and/or limited resources requirements without degrading the high level of security. In fact, most of the recent solutions are based on the chaotic theory. However, the majority of chaotic systems suffers from different limitations and their implementation is difficult at the hardware level because of the non integer operations that are employed requiring huge resources and latency. In this paper, we analyze the new chaos-based image encryption system presented in~\cite{el2016new}. It uses a static binary diffusion layer, followed by a key dependent bit-permutation layer that only iterates for one round. Based on their results in this paper, we claim that the uniformity and avalanche effect can be reached from the first round. However, we tried to verify the results but our conclusion was that these results were wrong because it was shown that at least 6 iterations are necessary to ensure the required cryptographic performance such as the plain-sensitivity property. Therefore, the required execution time must be multiplied by 6 and consequently this will increase the latency. In addition to all aforementioned problems, we find that ensuring the avalanche effect in the whole image introduces a high error propagation. In order to solve this problem, we recommend to ensure the avalanche effect in the level of blocks instead of the whole image.
△ Less
Submitted 29 January, 2017;
originally announced January 2017.
