-
Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English
Authors:
Runtao Zhou,
Guangya Wan,
Saadia Gabriel,
Sheng Li,
Alexander J Gates,
Maarten Sap,
Thomas Hartvigsen
Abstract:
Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning tasks, leading to their widespread deployment. However, recent studies have highlighted concerning biases in these models, particularly in their handling of dialectal variations like African American English (AAE). In this work, we systematically investigate dialectal disparities in LLM reasoning tasks. We develop…
▽ More
Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning tasks, leading to their widespread deployment. However, recent studies have highlighted concerning biases in these models, particularly in their handling of dialectal variations like African American English (AAE). In this work, we systematically investigate dialectal disparities in LLM reasoning tasks. We develop an experimental framework comparing LLM performance given Standard American English (SAE) and AAE prompts, combining LLM-based dialect conversion with established linguistic analyses. We find that LLMs consistently produce less accurate responses and simpler reasoning chains and explanations for AAE inputs compared to equivalent SAE questions, with disparities most pronounced in social science and humanities domains. These findings highlight systematic differences in how LLMs process and reason about different language varieties, raising important questions about the development and deployment of these systems in our multilingual and multidialectal world. Our code repository is publicly available at https://github.com/Runtaozhou/dialect_bias_eval.
△ Less
Submitted 6 March, 2025;
originally announced March 2025.
-
The increasing fragmentation of global science limits the diffusion of ideas
Authors:
Alexander J. Gates,
Jianjian Gao,
Indraneel Mane
Abstract:
Global science is often portrayed as a unified system of shared knowledge and open exchange. Yet this vision contrasts with emerging evidence that scientific recognition is uneven and increasingly fragmented along regional and cultural lines. Traditional models emphasize Western dominance in knowledge production but overlook regional dynamics, reinforcing a core-periphery narrative that sustains d…
▽ More
Global science is often portrayed as a unified system of shared knowledge and open exchange. Yet this vision contrasts with emerging evidence that scientific recognition is uneven and increasingly fragmented along regional and cultural lines. Traditional models emphasize Western dominance in knowledge production but overlook regional dynamics, reinforcing a core-periphery narrative that sustains disparities and marginalizes less prominent countries. In this study, we introduce a rank-based signed measure of national citation preferences, enabling the construction of a global recognition network that distinguishes over- and under-recognition between countries. Using a multinomial logistic link prediction model, we assess how economic, cultural, and scientific variables shape the presence and direction of national citation preferences. We uncover a global structure composed of multiple scientific communities, characterized by strong internal citation preferences and negative preferences between them-revealing growing fragmentation in the international scientific system. A separate weighted logistic regression framework suggests that this network significantly influences the international diffusion of scientific ideas, even after controlling for common covariates. Together, these findings highlight the structural barriers to equitable recognition and underscore the importance of scientific community membership in shaping influence, offering valuable insights for policymakers aiming to foster inclusive and impactful global science.
△ Less
Submitted 10 June, 2025; v1 submitted 8 April, 2024;
originally announced April 2024.
-
Mapping Philanthropic Support of Science
Authors:
Louis M. Shekhtman,
Alexander J. Gates,
Albert-László Barabási
Abstract:
While philanthropic support for science has increased in the past decade, there is limited quantitative knowledge about the patterns that characterize it and the mechanisms that drive its distribution. Here, we map philanthropic funding to universities and research institutions based on IRS tax forms from 685,397 non-profit organizations. We identify nearly one million grants supporting institutio…
▽ More
While philanthropic support for science has increased in the past decade, there is limited quantitative knowledge about the patterns that characterize it and the mechanisms that drive its distribution. Here, we map philanthropic funding to universities and research institutions based on IRS tax forms from 685,397 non-profit organizations. We identify nearly one million grants supporting institutions involved in science and higher education, finding that in volume and scope, philanthropic funding has grown to become comparable to federal research funding. Yet, distinct from government support, philanthropic funders tend to focus locally, indicating that criteria beyond research excellence play an important role in funding decisions. We also show evidence of persistence, i.e., once a grant-giving relationship begins, it tends to continue in time. Finally, we leverage the bipartite network of supporters and recipients to help us demonstrate the predictive power of the underlying network in foreseeing future funder-recipient relationships. The developed toolset could offer funding recommendations to organizations and help funders diversify their portfolio. We discuss the policy implications of our findings for philanthropic funders, individual researchers, and quantitative understanding of philanthropy.
△ Less
Submitted 7 December, 2022; v1 submitted 9 June, 2022;
originally announced June 2022.
-
Historical comparison of gender inequality in scientific careers across countries and disciplines
Authors:
Junming Huang,
Alexander J. Gates,
Roberta Sinatra,
Albert-Laszlo Barabasi
Abstract:
There is extensive, yet fragmented, evidence of gender differences in academia suggesting that women are under-represented in most scientific disciplines, publish fewer articles throughout a career, and their work acquires fewer citations. Here, we offer a comprehensive picture of longitudinal gender discrepancies in performance through a bibliometric analysis of academic careers by reconstructing…
▽ More
There is extensive, yet fragmented, evidence of gender differences in academia suggesting that women are under-represented in most scientific disciplines, publish fewer articles throughout a career, and their work acquires fewer citations. Here, we offer a comprehensive picture of longitudinal gender discrepancies in performance through a bibliometric analysis of academic careers by reconstructing the complete publication history of over 1.5 million gender-identified authors whose publishing career ended between 1955 and 2010, covering 83 countries and 13 disciplines. We find that, paradoxically, the increase of participation of women in science over the past 60 years was accompanied by an increase of gender differences in both productivity and impact. Most surprisingly though, we uncover two gender invariants, finding that men and women publish at a comparable annual rate and have equivalent career-wise impact for the same size body of work. Finally, we demonstrate that differences in dropout rates and career length explain a large portion of the reported career-wise differences in productivity and impact. This comprehensive picture of gender inequality in academia can help rephrase the conversation around the sustainability of women's careers in academia, with important consequences for institutions and policy makers.
△ Less
Submitted 9 July, 2019;
originally announced July 2019.
-
Apache Hive: From MapReduce to Enterprise-grade Big Data Warehousing
Authors:
Jesús Camacho-Rodríguez,
Ashutosh Chauhan,
Alan Gates,
Eugene Koifman,
Owen O'Malley,
Vineet Garg,
Zoltan Haindrich,
Sergey Shelukhin,
Prasanth Jayachandran,
Siddharth Seth,
Deepak Jaiswal,
Slim Bouguerra,
Nishant Bangarwa,
Sankar Hariappan,
Anishek Agarwal,
Jason Dere,
Daniel Dai,
Thejas Nair,
Nita Dembla,
Gopal Vijayaraghavan,
Günther Hagleitner
Abstract:
Apache Hive is an open-source relational database system for analytic big-data workloads. In this paper we describe the key innovations on the journey from batch tool to fully fledged enterprise data warehousing system. We present a hybrid architecture that combines traditional MPP techniques with more recent big data and cloud concepts to achieve the scale and performance required by today's anal…
▽ More
Apache Hive is an open-source relational database system for analytic big-data workloads. In this paper we describe the key innovations on the journey from batch tool to fully fledged enterprise data warehousing system. We present a hybrid architecture that combines traditional MPP techniques with more recent big data and cloud concepts to achieve the scale and performance required by today's analytic applications. We explore the system by detailing enhancements along four main axis: Transactions, optimizer, runtime, and federation. We then provide experimental results to demonstrate the performance of the system for typical workloads and conclude with a look at the community roadmap.
△ Less
Submitted 26 March, 2019;
originally announced March 2019.
-
CANA: A python package for quantifying control and canalization in Boolean Networks
Authors:
Rion Brattig Correia,
Alexander J. Gates,
Xuan Wang,
Luis M. Rocha
Abstract:
Logical models offer a simple but powerful means to understand the complex dynamics of biochemical regulation, without the need to estimate kinetic parameters. However, even simple automata components can lead to collective dynamics that are computationally intractable when aggregated into networks. In previous work we demonstrated that automata network models of biochemical regulation are highly…
▽ More
Logical models offer a simple but powerful means to understand the complex dynamics of biochemical regulation, without the need to estimate kinetic parameters. However, even simple automata components can lead to collective dynamics that are computationally intractable when aggregated into networks. In previous work we demonstrated that automata network models of biochemical regulation are highly canalizing, whereby many variable states and their groupings are redundant (Marques-Pita and Rocha, 2013). The precise charting and measurement of such canalization simplifies these models, making even very large networks amenable to analysis. Moreover, canalization plays an important role in the control, robustness, modularity and criticality of Boolean network dynamics, especially those used to model biochemical regulation (Gates and Rocha, 2016; Gates et al., 2016; Manicka, 2017). Here we describe a new publicly-available Python package that provides the necessary tools to extract, measure, and visualize canalizing redundancy present in Boolean network models. It extracts the pathways most effective in controlling dynamics in these models, including their effective graph and dynamics canalizing map, as well as other tools to uncover minimum sets of control variables.
△ Less
Submitted 9 May, 2018; v1 submitted 9 March, 2018;
originally announced March 2018.
-
Pragmatic-Pedagogic Value Alignment
Authors:
Jaime F. Fisac,
Monica A. Gates,
Jessica B. Hamrick,
Chang Liu,
Dylan Hadfield-Menell,
Malayandi Palaniappan,
Dhruv Malik,
S. Shankar Sastry,
Thomas L. Griffiths,
Anca D. Dragan
Abstract:
As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value-alignment problem. In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users' objectives as they go. We argue that a meaningful s…
▽ More
As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value-alignment problem. In robotics, value alignment is key to the design of collaborative robots that can integrate into human workflows, successfully inferring and adapting to their users' objectives as they go. We argue that a meaningful solution to value alignment must combine multi-agent decision theory with rich mathematical models of human cognition, enabling robots to tap into people's natural collaborative capabilities. We present a solution to the cooperative inverse reinforcement learning (CIRL) dynamic game based on well-established cognitive models of decision making and theory of mind. The solution captures a key reciprocity relation: the human will not plan her actions in isolation, but rather reason pedagogically about how the robot might learn from them; the robot, in turn, can anticipate this and interpret the human's actions pragmatically. To our knowledge, this work constitutes the first formal analysis of value alignment grounded in empirically validated cognitive models.
△ Less
Submitted 5 February, 2018; v1 submitted 19 July, 2017;
originally announced July 2017.
-
Element-centric clustering comparison unifies overlaps and hierarchy
Authors:
Alexander J. Gates,
Ian B. Wood,
William P. Hetrick,
Yong-Yeol Ahn
Abstract:
Clustering is one of the most universal approaches for understanding complex data. A pivotal aspect of clustering analysis is quantitatively comparing clusterings; clustering comparison is the basis for many tasks such as clustering evaluation, consensus clustering, and tracking the temporal evolution of clusters. In particular, the extrinsic evaluation of clustering methods requires comparing the…
▽ More
Clustering is one of the most universal approaches for understanding complex data. A pivotal aspect of clustering analysis is quantitatively comparing clusterings; clustering comparison is the basis for many tasks such as clustering evaluation, consensus clustering, and tracking the temporal evolution of clusters. In particular, the extrinsic evaluation of clustering methods requires comparing the uncovered clusterings to planted clusterings or known metadata. Yet, as we demonstrate, existing clustering comparison measures have critical biases which undermine their usefulness, and no measure accommodates both overlapping and hierarchical clusterings. Here we unify the comparison of disjoint, overlapping, and hierarchically structured clusterings by proposing a new element-centric framework: elements are compared based on the relationships induced by the cluster structure, as opposed to the traditional cluster-centric philosophy. We demonstrate that, in contrast to standard clustering similarity measures, our framework does not suffer from critical biases and naturally provides unique insights into how the clusterings differ. We illustrate the strengths of our framework by revealing new insights into the organization of clusters in two applications: the improved classification of schizophrenia based on the overlapping and hierarchical community structure of fMRI brain networks, and the disentanglement of various social homophily factors in Facebook social networks. The universality of clustering suggests far-reaching impact of our framework throughout all areas of science.
△ Less
Submitted 12 June, 2019; v1 submitted 19 June, 2017;
originally announced June 2017.
-
Modularity and the spread of perturbations in complex dynamical systems
Authors:
Artemy Kolchinsky,
Alexander J. Gates,
Luis M. Rocha
Abstract:
We propose a method to decompose dynamical systems based on the idea that modules constrain the spread of perturbations. We find partitions of system variables that maximize 'perturbation modularity', defined as the autocovariance of coarse-grained perturbed trajectories. The measure effectively separates the fast intramodular from the slow intermodular dynamics of perturbation spreading (in this…
▽ More
We propose a method to decompose dynamical systems based on the idea that modules constrain the spread of perturbations. We find partitions of system variables that maximize 'perturbation modularity', defined as the autocovariance of coarse-grained perturbed trajectories. The measure effectively separates the fast intramodular from the slow intermodular dynamics of perturbation spreading (in this respect, it is a generalization of the 'Markov stability' method of network community detection). Our approach captures variation of modular organization across different system states, time scales, and in response to different kinds of perturbations: aspects of modularity which are all relevant to real-world dynamical systems. It offers a principled alternative to detecting communities in networks of statistical dependencies between system variables (e.g., 'relevance networks' or 'functional networks'). Using coupled logistic maps, we demonstrate that the method uncovers hierarchical modular organization planted in a system's coupling matrix. Additionally, in homogeneously-coupled map lattices, it identifies the presence of self-organized modularity that depends on the initial state, dynamical parameters, and type of perturbations. Our approach offers a powerful tool for exploring the modular organization of complex dynamical systems.
△ Less
Submitted 23 December, 2015; v1 submitted 14 September, 2015;
originally announced September 2015.
