Showing 1–4 of 4 results for author: Slud, E

Analyzing the Machine Learning Conference Review Process

Authors: David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, Tom Goldstein

Abstract: Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of pa… ▽ More Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers. △ Less

Submitted 25 November, 2020; v1 submitted 24 November, 2020; originally announced November 2020.

Comments: NeurIPS Workshop on Navigating the Broader Impacts of AI Research. Full version at arXiv:2010.05137

An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

Authors: David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, Tom Goldstein

Abstract: Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of pa… ▽ More Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers. △ Less

Submitted 26 October, 2020; v1 submitted 10 October, 2020; originally announced October 2020.

Comments: 19 pages, 6 Figures

Introduction

Authors: P. Lahiri, Eric Slud

Abstract: The Statistics Consortium at the University of Maryland, College Park, hosted a two-day workshop on Bayesian Methods that Frequentists Should Know during April 30--May 1, 2008. The event was co-sponsored by the Institute of Mathematical Statistics (IMS), Office of Research and Methodology, National Center for Health Statistics, Survey Research Methods Section (SRMS) of the American Statistical Ass… ▽ More The Statistics Consortium at the University of Maryland, College Park, hosted a two-day workshop on Bayesian Methods that Frequentists Should Know during April 30--May 1, 2008. The event was co-sponsored by the Institute of Mathematical Statistics (IMS), Office of Research and Methodology, National Center for Health Statistics, Survey Research Methods Section (SRMS) of the American Statistical Association, and Washington Statistical Society. The workshop was intended to bring out the positive features of Bayesian statistics in solving real-life problems, including complex problems in sample surveys and production of high-quality official statistics. △ Less

Submitted 29 September, 2011; originally announced September 2011.

Comments: Published in at http://dx.doi.org/10.1214/11-STS359 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-STS-STS359

Journal ref: Statistical Science 2011, Vol. 26, No. 2, 161-161

Discussion of "Impact of Frequentist and Bayesian Methods on Survey Sampling Practice: A Selective Appraisal" by J. N. K. Rao

Authors: Eric Slud

Abstract: Discussion of "Impact of Frequentist and Bayesian Methods on Survey Sampling Practice: A Selective Appraisal" by J. N. K. Rao [arXiv:1108.2356] Discussion of "Impact of Frequentist and Bayesian Methods on Survey Sampling Practice: A Selective Appraisal" by J. N. K. Rao [arXiv:1108.2356] △ Less

Submitted 19 August, 2011; originally announced August 2011.

Comments: Published in at http://dx.doi.org/10.1214/11-STS346C the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-STS-STS346C

Journal ref: Statistical Science 2011, Vol. 26, No. 2, 262-265