-
How far can bias go? -- Tracing bias from pretraining data to alignment
Authors:
Marion Thaler,
Abdullatif Köksal,
Alina Leidinger,
Anna Korhonen,
Hinrich Schütze
Abstract:
As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing…
▽ More
As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing on the Dolma dataset and the OLMo model. Using zero-shot prompting and token co-occurrence analyses, we explore how biases in training data influence model outputs. Our findings reveal that biases present in pre-training data are amplified in model outputs. The study also examines the effects of prompt types, hyperparameters, and instruction-tuning on bias expression, finding instruction-tuning partially alleviating representational bias while still maintaining overall stereotypical gender associations, whereas hyperparameters and prompting variation have a lesser effect on bias expression. Our research traces bias throughout the LLM development pipeline and underscores the importance of mitigating bias at the pretraining stage.
△ Less
Submitted 28 November, 2024;
originally announced November 2024.
-
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions
Authors:
Abdullatif Köksal,
Marion Thaler,
Ayyoob Imani,
Ahmet Üstün,
Anna Korhonen,
Hinrich Schütze
Abstract:
Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tunin…
▽ More
Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tuning datasets for low-resource languages without requiring human annotators or pre-existing multilingual models. Utilizing reverse instructions and a translation pipeline, MURI produces instruction-output pairs from existing human-written texts in low-resource languages. This method ensures cultural relevance and diversity by sourcing texts from different native domains and applying filters to eliminate inappropriate content. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the approach's effectiveness for both NLU and open-ended generation. We publicly release datasets and models at https://github.com/akoksal/muri.
△ Less
Submitted 19 September, 2024;
originally announced September 2024.
-
JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models
Authors:
Ze Wang,
Zekun Wu,
Xin Guan,
Michael Thaler,
Adriano Koshiyama,
Skylar Lu,
Sachin Beepath,
Ediz Ertekin Jr.,
Maria Perez-Ortiz
Abstract:
The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new…
▽ More
The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new construct grounded in labour economics, legal principles, and critiques of current bias benchmarks: hiring bias can be categorized into two types: Level bias (difference in the average outcomes between demographic counterfactual groups) and Spread bias (difference in the variance of outcomes between demographic counterfactual groups); Level bias can be further subdivided into statistical bias (i.e. changing with non-demographic content) and taste-based bias (i.e. consistent regardless of non-demographic content). Secondly, the framework includes rigorous statistical and computational hiring bias metrics, such as Rank After Scoring (RAS), Rank-based Impact Ratio, Permutation Test, and Fixed Effects Model. Thirdly, we analyze gender hiring biases in ten state-of-the-art LLMs. Seven out of ten LLMs show significant biases against males in at least one industry. An industry-effect regression reveals that the healthcare industry is the most biased against males. Moreover, we found that the bias performance remains invariant with resume content for eight out of ten LLMs. This indicates that the bias performance measured in this paper might apply to other resume datasets with different resume qualities. Fourthly, we provide a user-friendly demo and resume dataset to support the adoption and practical use of the framework, which can be generalized to other social traits and tasks.
△ Less
Submitted 30 September, 2024; v1 submitted 17 June, 2024;
originally announced June 2024.
-
The Supply of Motivated Beliefs
Authors:
Michael Thaler
Abstract:
When people choose what messages to send to others, they often consider how others will interpret the messages. A sender may expect a receiver to engage in motivated reasoning, leading the receiver to trust good news more than bad news, relative to a Bayesian. This paper experimentally studies how motivated reasoning affects information transmission in political settings. Senders are randomly matc…
▽ More
When people choose what messages to send to others, they often consider how others will interpret the messages. A sender may expect a receiver to engage in motivated reasoning, leading the receiver to trust good news more than bad news, relative to a Bayesian. This paper experimentally studies how motivated reasoning affects information transmission in political settings. Senders are randomly matched with receivers whose political party's stances happen to be aligned or misaligned with the truth, and either face incentives to be rated as truthful or face no incentives. Incentives to be rated as truthful cause senders to be less truthful; when incentivized, senders send false information to align messages with receivers' politically-motivated beliefs. The adverse effect of incentives is not appreciated by receivers, who rate senders in both conditions as being equally likely to be truthful. A complementary experiment further identifies senders' beliefs about receivers' motivated reasoning as the mechanism driving these results. Senders are additionally willing to pay to learn the politics of their receivers, and use this information to send more false messages.
△ Less
Submitted 28 September, 2023; v1 submitted 11 November, 2021;
originally announced November 2021.
-
Overinference from Weak Signals and Underinference from Strong Signals
Authors:
Ned Augenblick,
Eben Lazarus,
Michael Thaler
Abstract:
When people receive new information, sometimes they revise their beliefs too much, and sometimes too little. In this paper, we show that a key driver of whether people overinfer or underinfer is the strength of the information. Based on a model in which people know which direction to update in, but not exactly how much to update, we hypothesize that people will overinfer from weak signals and unde…
▽ More
When people receive new information, sometimes they revise their beliefs too much, and sometimes too little. In this paper, we show that a key driver of whether people overinfer or underinfer is the strength of the information. Based on a model in which people know which direction to update in, but not exactly how much to update, we hypothesize that people will overinfer from weak signals and underinfer from strong signals. We then test this hypothesis across four different environments: abstract experiments, a naturalistic experiment, sports betting markets, and financial markets. In each environment, our consistent and robust finding is overinference from weak signals and underinference from strong signals. Our framework and findings can help harmonize apparently contradictory results from the experimental and empirical literatures.
△ Less
Submitted 30 June, 2024; v1 submitted 20 September, 2021;
originally announced September 2021.
-
Agent vs. Avatar: Comparing Embodied Conversational Agents Concerning Characteristics of the Uncanny Valley
Authors:
Markus Thaler,
Stephan Schlögl,
Aleksander Groth
Abstract:
Visual appearance is an important aspect influencing the perception and consequent acceptance of Embodied Conversational Agents (ECA). To this end, the Uncanny Valley theory contradicts the common assumption that increased humanization of characters leads to better acceptance. Rather, it shows that anthropomorphic behavior may trigger feelings of eeriness and rejection in people. The work presente…
▽ More
Visual appearance is an important aspect influencing the perception and consequent acceptance of Embodied Conversational Agents (ECA). To this end, the Uncanny Valley theory contradicts the common assumption that increased humanization of characters leads to better acceptance. Rather, it shows that anthropomorphic behavior may trigger feelings of eeriness and rejection in people. The work presented in this paper explores whether four different autonomous ECAs, specifically build for a European research project, are affected by this effect, and how they compare to two slightly more realistically looking human-controlled, i.e. face-tracked, ECAs with respect to perceived humanness, eeriness, and attractiveness. Short videos of the ECAs in combination with a validated questionnaire were used to investigate potential differences. Results support existing theories highlighting that increased perceived humanness correlates with increased perceived eeriness. Furthermore, it was found, that neither the gender of survey participants, their age, nor the sex of the ECA influences this effect, and that female ECAs are perceived to be significantly more attractive than their male counterparts.
△ Less
Submitted 22 April, 2021;
originally announced April 2021.
-
The Fake News Effect: Experimentally Identifying Motivated Reasoning Using Trust in News
Authors:
Michael Thaler
Abstract:
Motivated reasoning posits that people distort how they process information in the direction of beliefs they find attractive. This paper creates a novel experimental design to identify motivated reasoning from Bayesian updating when people have preconceived beliefs. It analyzes how subjects assess the veracity of information sources that tell them the median of their belief distribution is too hig…
▽ More
Motivated reasoning posits that people distort how they process information in the direction of beliefs they find attractive. This paper creates a novel experimental design to identify motivated reasoning from Bayesian updating when people have preconceived beliefs. It analyzes how subjects assess the veracity of information sources that tell them the median of their belief distribution is too high or too low. Bayesians infer nothing about the source veracity, but motivated beliefs are evoked. Evidence supports politically-motivated reasoning about immigration, income mobility, crime, racial discrimination, gender, climate change, and gun laws. Motivated reasoning helps explain belief biases, polarization, and overconfidence.
△ Less
Submitted 25 May, 2022; v1 submitted 2 December, 2020;
originally announced December 2020.
-
Good News Is Not a Sufficient Condition for Motivated Reasoning
Authors:
Michael Thaler
Abstract:
People often receive good news that makes them feel better about the world around them, or bad news that makes them feel worse about it. This paper studies how the valence of news affects belief updating, absent functional and ego-relevant factors. Using experiments with over 1,500 participants and 5,600 observations, I test whether people engage in motivated reasoning to overly trust good news ve…
▽ More
People often receive good news that makes them feel better about the world around them, or bad news that makes them feel worse about it. This paper studies how the valence of news affects belief updating, absent functional and ego-relevant factors. Using experiments with over 1,500 participants and 5,600 observations, I test whether people engage in motivated reasoning to overly trust good news versus bad news on valence-relevant issues like cancer survival rates, others' happiness, and infant mortality. The estimate for motivated reasoning towards good news is a precisely-estimated null. Modest effects, of one-third the size of motivated reasoning in politics and performance, can be ruled out. Complementary survey evidence shows that most people expect good news to increase happiness, but to not systematically lead to motivated reasoning. These results suggest that belief-based utility is not sufficient in leading people to distort belief updating in order to favor those beliefs.
△ Less
Submitted 18 January, 2024; v1 submitted 2 December, 2020;
originally announced December 2020.
-
Gender Differences in Motivated Reasoning
Authors:
Michael Thaler
Abstract:
Men and women systematically differ in their beliefs about their performance relative to others; in particular, men tend to be more overconfident. This paper provides support for one explanation for gender differences in overconfidence, performance-motivated reasoning, in which people distort how they process new information in ways that make them believe they outperformed others. Using a large on…
▽ More
Men and women systematically differ in their beliefs about their performance relative to others; in particular, men tend to be more overconfident. This paper provides support for one explanation for gender differences in overconfidence, performance-motivated reasoning, in which people distort how they process new information in ways that make them believe they outperformed others. Using a large online experiment, I find that male subjects distort information processing in ways that favor their performance, while female subjects do not systematically distort information processing in either direction. These statistically-significant gender differences in performance-motivated reasoning mimic gender differences in overconfidence; beliefs of male subjects are systematically overconfident, while beliefs of female subjects are well-calibrated on average. The experiment also includes political questions, and finds that politically-motivated reasoning is similar for both men and women. These results suggest that, while men and women are both susceptible to motivated reasoning in general, men find it particularly attractive to believe that they outperformed others.
△ Less
Submitted 24 July, 2021; v1 submitted 2 December, 2020;
originally announced December 2020.
-
Thermodynamics of the PNJL model
Authors:
C. Ratti,
S. Roessner,
M. A. Thaler,
W. Weise
Abstract:
QCD thermodynamics is investigated by means of the Polyakov-loop-extended Nambu Jona-Lasinio (PNJL) model, in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. The behaviour of the Polyakov loop as a function of temperature is obtained by minimizing the thermodynamic potential of the system. A Taylor series e…
▽ More
QCD thermodynamics is investigated by means of the Polyakov-loop-extended Nambu Jona-Lasinio (PNJL) model, in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. The behaviour of the Polyakov loop as a function of temperature is obtained by minimizing the thermodynamic potential of the system. A Taylor series expansion of the pressure is performed. Pressure difference and quark number density are then evaluated up to sixth order in quark chemical potential, and compared to the corresponding lattice data. The validity of the Taylor expansion is discussed within our model, through a comparison between the full results and the truncated ones.
△ Less
Submitted 21 September, 2006;
originally announced September 2006.
-
Phase diagram and thermodynamics of the PNJL model
Authors:
Claudia Ratti,
Michael A. Thaler,
Wolfram Weise
Abstract:
QCD-based thermodynamics at zero and finite quark chemical potential is studied using an extended Nambu and Jona-Lasinio approach in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. This so-called PNJL model thus includes features of both deconfinement and chiral symmetry restoration. We discuss the phase di…
▽ More
QCD-based thermodynamics at zero and finite quark chemical potential is studied using an extended Nambu and Jona-Lasinio approach in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. This so-called PNJL model thus includes features of both deconfinement and chiral symmetry restoration. We discuss the phase diagram as it emerges from this approach in close comparison with results from lattice QCD thermodynamics. The critical point, separating crossover from first order phase transition, is investigated with special focus on its quark mass dependence, starting from the relatively large masses presently accessible by lattice simulations, down to the chiral limit.
△ Less
Submitted 11 April, 2006;
originally announced April 2006.
-
Phases of QCD: lattice thermodynamics and a field theoretical model
Authors:
C. Ratti,
M. A. Thaler,
W. Weise
Abstract:
We investigate three-colour QCD thermodynamics at finite quark chemical potential. Lattice QCD results are compared with a generalized Nambu Jona-Lasinio model in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. This so-called PNJL model thus includes features of both deconfinement and chiral symmetry restor…
▽ More
We investigate three-colour QCD thermodynamics at finite quark chemical potential. Lattice QCD results are compared with a generalized Nambu Jona-Lasinio model in which quarks couple simultaneously to the chiral condensate and to a background temporal gauge field representing Polyakov loop dynamics. This so-called PNJL model thus includes features of both deconfinement and chiral symmetry restoration. The parameters of the Polyakov loop effective potential are fixed in the pure gauge sector. The chiral condensate and the Polyakov loop as functions of temperature and quark chemical potential are calculated by minimizing the thermodynamic potential of the system. The resulting equation of state, (scaled) pressure difference and quark number density at finite quark chemical potential are then confronted with corresponding Lattice QCD data.
△ Less
Submitted 13 January, 2006; v1 submitted 23 June, 2005;
originally announced June 2005.
-
Occupation times of sets of infinite measure for ergodic transformations
Authors:
Jon Aaronson,
Maximilian Thaler,
Roland Zweimueller
Abstract:
Assume that $T$ is a conservative ergodic measure preserving transformation of the infinite measure space $(X,\mathcal{A},μ)$.We study the asymptotic behaviour of occupation times of certain subsets of infinite measure. Specifically, we prove a Darling-Kac type distributional limit theorem for occupation times of barely infinite components which are separated from the rest of the space by a set…
▽ More
Assume that $T$ is a conservative ergodic measure preserving transformation of the infinite measure space $(X,\mathcal{A},μ)$.We study the asymptotic behaviour of occupation times of certain subsets of infinite measure. Specifically, we prove a Darling-Kac type distributional limit theorem for occupation times of barely infinite components which are separated from the rest of the space by a set of finite measure with c.f.-mixing return process. In the same setup we show that the ratios of occupation times of two components separated in this way diverge almost everywhere. These abstract results are illustrated by applications to interval maps with indifferent fixed points.
△ Less
Submitted 28 June, 2004;
originally announced June 2004.
-
Quasiparticle Description of Hot QCD at Finite Quark Chemical Potential
Authors:
M. A. Thaler,
R. A. Schneider,
W. Weise
Abstract:
We study the extension of a phenomenologically successful quasiparticle model that describes lattice results of the equation of state of the deconfined phase of QCD for Tc <= T < 4 Tc, to finite quark chemical potential mu. The phase boundary line Tc(mu), the pressure difference (p(T,mu)-p(T,mu=0))/T^4 and the quark number density nq(T,mu)/T^3 are calculated and compared to recent lattice result…
▽ More
We study the extension of a phenomenologically successful quasiparticle model that describes lattice results of the equation of state of the deconfined phase of QCD for Tc <= T < 4 Tc, to finite quark chemical potential mu. The phase boundary line Tc(mu), the pressure difference (p(T,mu)-p(T,mu=0))/T^4 and the quark number density nq(T,mu)/T^3 are calculated and compared to recent lattice results. Good agreement is found up to quark chemical potentials of order mu = Tc.
△ Less
Submitted 22 October, 2003; v1 submitted 21 October, 2003;
originally announced October 2003.
-
Probing the QCD Equation of State
Authors:
R. A. Schneider,
T. Renk,
M. Thaler,
A. Polleri,
W. Weise
Abstract:
We propose a novel quasiparticle interpretation of the equation of state of deconfined QCD at finite temperature. Using appropriate thermal masses, we introduce a phenomenological parametrisation of the onset of confinement in the vicinity of the phase transition. Lattice results of bulk thermodynamic quantities are well reproduced, the extension to small quark chemical potential is also success…
▽ More
We propose a novel quasiparticle interpretation of the equation of state of deconfined QCD at finite temperature. Using appropriate thermal masses, we introduce a phenomenological parametrisation of the onset of confinement in the vicinity of the phase transition. Lattice results of bulk thermodynamic quantities are well reproduced, the extension to small quark chemical potential is also successful. We then apply the model to dilepton production and charm suppression in ultrarelativistic heavy-ion collisions.
△ Less
Submitted 4 November, 2002;
originally announced November 2002.
-
Consistent Treatment of Propagator Modifications in Elastic Nucleon-Nucleus Scattering within the Spectator Expansion
Authors:
C. R. Chinn,
Ch. Elster,
R. M. Thaler,
S. P. Weppner
Abstract:
The theory of the elastic scattering of a nucleon from a nucleus is presented in the form of a Spectator Expansion of the optical potential. Particular attention is paid to the treatment of the free projectile$\,-\,$nucleus propagator when the coupling of the struck target nucleon to the residual target must be taken into consideration. First order calculations within this framework are shown fo…
▽ More
The theory of the elastic scattering of a nucleon from a nucleus is presented in the form of a Spectator Expansion of the optical potential. Particular attention is paid to the treatment of the free projectile$\,-\,$nucleus propagator when the coupling of the struck target nucleon to the residual target must be taken into consideration. First order calculations within this framework are shown for neutron total cross-sections and for proton scattering for a number of target nuclides at a variety of energies. The calculated values of these observables are in very good agreement with measurement.
△ Less
Submitted 28 March, 1995;
originally announced March 1995.
-
Total Cross Sections for Neutron Scattering
Authors:
C. R. Chinn,
Ch. Elster,
R. M. Thaler,
S. P. Weppner
Abstract:
Measurements of neutron total cross-sections are both extensive and extremely accurate. Although they place a strong constraint on theoretically constructed models, there are relatively few comparisons of predictions with experiment. The total cross-sections for neutron scattering from $^{16}$O and $^{40}$Ca are calculated as a function of energy from $50-700$~MeV laboratory energy with a micros…
▽ More
Measurements of neutron total cross-sections are both extensive and extremely accurate. Although they place a strong constraint on theoretically constructed models, there are relatively few comparisons of predictions with experiment. The total cross-sections for neutron scattering from $^{16}$O and $^{40}$Ca are calculated as a function of energy from $50-700$~MeV laboratory energy with a microscopic first order optical potential derived within the framework of the Watson expansion. Although these results are already in qualitative agreement with the data, the inclusion of medium corrections to the propagator is essential to correctly predict the energy dependence given by the experiment.
△ Less
Submitted 18 October, 1994;
originally announced October 1994.
-
Application of Multiple Scattering Theory to Lower Energy Elastic Nucleon-Nucleus Reactions
Authors:
C. R. Chinn,
Ch. Elster,
R. M. Thaler,
S. P. Weppner
Abstract:
The optical model potentials for nucleon-nucleus elastic scattering at $65$~MeV are calculated for $^{12}$C, $^{16}$O, $^{28}$Si, $^{40}$Ca, $^{56}$Fe, $^{90}$Zr and $^{208}$Pb in first order multiple scattering theory, following the prescription of the spectator expansion, where the only inputs are the free NN potentials, the nuclear densities and the nuclear mean field as derived from microsco…
▽ More
The optical model potentials for nucleon-nucleus elastic scattering at $65$~MeV are calculated for $^{12}$C, $^{16}$O, $^{28}$Si, $^{40}$Ca, $^{56}$Fe, $^{90}$Zr and $^{208}$Pb in first order multiple scattering theory, following the prescription of the spectator expansion, where the only inputs are the free NN potentials, the nuclear densities and the nuclear mean field as derived from microscopic nuclear structure calculations. These potentials are used to predict differential cross sections, analyzing powers and spin rotation functions for neutron and proton scattering at 65 MeV projectile energy and compared with available experimental data.
△ Less
Submitted 18 October, 1994;
originally announced October 1994.
