Do LLMs exhibit demographic parity in responses to queries about Human Rights?
Authors:
Rafiya Javed,
Jackie Kay,
David Yanni,
Abdullah Zaini,
Anushe Sheikh,
Maribeth Rauh,
Ramona Comanescu,
Iason Gabriel,
Laura Weidinger
Abstract:
This research describes a novel approach to evaluating hedging behaviour in large language models (LLMs), specifically in the context of human rights as defined in the Universal Declaration of Human Rights (UDHR). Hedging and non-affirmation are behaviours that express ambiguity or a lack of clear endorsement on specific statements. These behaviours are undesirable in certain contexts, such as que…
▽ More
This research describes a novel approach to evaluating hedging behaviour in large language models (LLMs), specifically in the context of human rights as defined in the Universal Declaration of Human Rights (UDHR). Hedging and non-affirmation are behaviours that express ambiguity or a lack of clear endorsement on specific statements. These behaviours are undesirable in certain contexts, such as queries about whether different groups are entitled to specific human rights; since all people are entitled to human rights. Here, we present the first systematic attempt to measure these behaviours in the context of human rights, with a particular focus on between-group comparisons. To this end, we design a novel prompt set on human rights in the context of different national or social identities. We develop metrics to capture hedging and non-affirmation behaviours and then measure whether LLMs exhibit demographic parity when responding to the queries. We present results on three leading LLMs and find that all models exhibit some demographic disparities in how they attribute human rights between different identity groups. Futhermore, there is high correlation between different models in terms of how disparity is distributed amongst identities, with identities that have high disparity in one model also facing high disparity in both the other models. While baseline rates of hedging and non-affirmation differ, these disparities are consistent across queries that vary in ambiguity and they are robust across variations of the precise query wording. Our findings highlight the need for work to explicitly align LLMs to human rights principles, and to ensure that LLMs endorse the human rights of all groups equally.
△ Less
Submitted 26 February, 2025;
originally announced February 2025.
Cyber-physical risks of hacked Internet-connected vehicles
Authors:
Skanda Vivek,
David Yanni,
Peter J. Yunker,
Jesse L. Silverberg
Abstract:
The integration of automotive technology with Internet-connectivity promises to both dramatically improve transportation, while simultaneously introducing the potential for new unknown risks. Internet-connected vehicles are like digital data because they can be targeted for malicious hacking. Unlike digital data, however, Internet-connected vehicles are cyber-physical systems that physically inter…
▽ More
The integration of automotive technology with Internet-connectivity promises to both dramatically improve transportation, while simultaneously introducing the potential for new unknown risks. Internet-connected vehicles are like digital data because they can be targeted for malicious hacking. Unlike digital data, however, Internet-connected vehicles are cyber-physical systems that physically interact with each other and their environment. As such, the extension of cybersecurity concerns into the cyber-physical domain introduces new possibilities for self-organized phenomena in traffic flow. Here, we study a scenario envisioned by cybersecurity experts leading to a large number of Internet-connected vehicles being suddenly and simultaneously disabled. We investigate post-hack traffic using agent-based simulations, and discover the critical relevance of percolation for probabilistically predicting the outcomes on a multi-lane road in the immediate aftermath of a vehicle-targeted cyber attack. We develop an analytic percolation-based model to rapidly assess road conditions given the density of disabled vehicles and apply it to study the street network of Manhattan (NY, USA) revealing the city's vulnerability to this particular cyber-physical attack.
△ Less
Submitted 28 February, 2019;
originally announced March 2019.
