-
Parameter estimation in Comparative Judgement
Authors:
Ian Hamilton,
Nick Tawn
Abstract:
Comparative Judgement is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most co…
▽ More
Comparative Judgement is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimates. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered. Further, we propose a superior approach based on bootstrapping. It is shown to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions and initial penalization method.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
The many routes to the ubiquitous Bradley-Terry model
Authors:
Ian Hamilton,
Nick Tawn,
David Firth
Abstract:
The rating of items based on pairwise comparisons has been a topic of statistical investigation for many decades. Numerous approaches have been proposed. One of the best known is the Bradley-Terry model. This paper seeks to assemble and explain a variety of motivations for its use. Some are based on principles or on maximising an objective function; others are derived from well-known statistical m…
▽ More
The rating of items based on pairwise comparisons has been a topic of statistical investigation for many decades. Numerous approaches have been proposed. One of the best known is the Bradley-Terry model. This paper seeks to assemble and explain a variety of motivations for its use. Some are based on principles or on maximising an objective function; others are derived from well-known statistical models, or stylised game scenarios. They include both examples well-known in the literature as well as what are believed to be novel presentations.
△ Less
Submitted 7 August, 2025; v1 submitted 21 December, 2023;
originally announced December 2023.
-
Investigating the 'old boy network' using latent space models
Authors:
Ian Hamilton
Abstract:
This paper investigates the nature of institutional ties between a group of English schools, including a large proportion of private schools that might be thought of as contributing to the 'old boy network'. The analysis is based on a network of bilaterally-determined school rugby union fixtures. The primary importance of geographical proximity in the determination of these fixtures supplies a spa…
▽ More
This paper investigates the nature of institutional ties between a group of English schools, including a large proportion of private schools that might be thought of as contributing to the 'old boy network'. The analysis is based on a network of bilaterally-determined school rugby union fixtures. The primary importance of geographical proximity in the determination of these fixtures supplies a spatial 'ground truth' against which the performance of models is assessed. A Bayesian fitting of the latent position cluster model is found to provide the best fit of the models examined. This is used to demonstrate a variety of methods that together provide a consistent and nuanced interpretation of the factors influencing community and edge formation in the network. The influence of homophily in fees and the proportion of boarders is identified as notable, with evidence that this is driven by a community of schools having the highest proportion of boarders and charging the highest fees, suggestive of the existence and nature of an 'old boy network' at an institutional level.
△ Less
Submitted 22 December, 2021;
originally announced December 2021.
-
Retrodictive Modelling of Modern Rugby Union: Extension of Bradley-Terry to Multiple Outcomes
Authors:
Ian Hamilton,
David Firth
Abstract:
Frequently in sporting competitions it is desirable to compare teams based on records of varying schedule strength. Methods have been developed for sports where the result outcomes are win, draw, or loss. In this paper those ideas are extended to account for any finite multiple outcome result set. A principle-based motivation is supplied and an implementation presented for modern rugby union, wher…
▽ More
Frequently in sporting competitions it is desirable to compare teams based on records of varying schedule strength. Methods have been developed for sports where the result outcomes are win, draw, or loss. In this paper those ideas are extended to account for any finite multiple outcome result set. A principle-based motivation is supplied and an implementation presented for modern rugby union, where bonus points are awarded for losing within a certain score margin and for scoring a certain number of tries. A number of variants are discussed including the constraining assumptions that are implied by each. The model is applied to assess the current rules of the Daily Mail Trophy, a national schools tournament in England and Wales.
△ Less
Submitted 21 December, 2021;
originally announced December 2021.
