-
Statistical Significance and Effect Sizes of Differences among Research Universities at the Level of Nations and Worldwide based on the Leiden Rankings
Authors:
Loet Leydesdorff,
Lutz Bornmann,
John Mingers
Abstract:
The Leiden Rankings can be used for grouping research universities by considering universities which are not statistically significantly different as homogeneous sets. The groups and intergroup relations can be analyzed and visualized using tools from network analysis. Using the so-called "excellence indicator" PPtop-10%--the proportion of the top-10% most-highly-cited papers assigned to a univers…
▽ More
The Leiden Rankings can be used for grouping research universities by considering universities which are not statistically significantly different as homogeneous sets. The groups and intergroup relations can be analyzed and visualized using tools from network analysis. Using the so-called "excellence indicator" PPtop-10%--the proportion of the top-10% most-highly-cited papers assigned to a university--we pursue a classification using (i) overlapping stability intervals, (ii) statistical-significance tests, and (iii) effect sizes of differences among 902 universities in 54 countries; we focus on the UK, Germany, Brazil, and the USA as national examples. Although the groupings remain largely the same using different statistical significance levels or overlapping stability intervals, these classifications are uncorrelated with those based on effect sizes. Effect sizes for the differences between universities are small (w <.2). The more detailed analysis of universities at the country level suggests that distinctions beyond three or perhaps four groups of universities (high, middle, low) may not be meaningful. Given similar institutional incentives, isomorphism within each eco-system of universities should not be underestimated. Our results suggest that networks based on overlapping stability intervals can provide a first impression of the relevant groupings among universities. However, the clusters are not well-defined divisions between groups of universities.
△ Less
Submitted 13 October, 2018; v1 submitted 30 October, 2017;
originally announced October 2017.
-
A Framework for Validating IS Research Based on a Pluralist Account of Truth and Correctness
Authors:
John Mingers,
Craig Standing
Abstract:
Research in information systems includes a wide range of approaches which make a contribution in terms of knowledge, understanding, or practical developments. The measure of any research is, ultimately, its validity: are its finding true, or its recommendations correct? However, empirical studies show that discussion of validity in research is often weak. In this paper we examine the nature of tru…
▽ More
Research in information systems includes a wide range of approaches which make a contribution in terms of knowledge, understanding, or practical developments. The measure of any research is, ultimately, its validity: are its finding true, or its recommendations correct? However, empirical studies show that discussion of validity in research is often weak. In this paper we examine the nature of truth and correctness in order to construct a validation framework that can encompass all the varied forms of research. Within philosophy, there has been much debate about truth, is it correspondence, coherence, consensual or pragmatic, and in fact current views revolve around the idea of a pluralist view of truth: it is one and many. Related to truth is the concept of correctness, and in particular the necessity of both internal correctness and external correctness. The framework we develop based on these concepts of truth and correctness has been applied to a range of research forms including positivist, mathematical, interpretive, design science, critical and action oriented. The benefits are: i) that a greater and more explicit focus on validity criteria will produce better research; ii) having a single framework can unite what at times seem conflicting approaches to research; iii) having criteria made explicit should encourage debate and further development.
△ Less
Submitted 27 December, 2016;
originally announced January 2017.
-
Evaluating Journal Quality: A Review of Journal Citation Indicators and Ranking in Business and Management
Authors:
John Mingers,
Liying Yang
Abstract:
Evaluating the quality of academic journal is becoming increasing important within the context of research performance evaluation. Traditionally, journals have been ranked by peer review lists such as that of the Association of Business Schools (UK) or though their journal impact factor (JIF). However, several new indicators have been developed, such as the h-index, SJR, SNIP and the Eigenfactor w…
▽ More
Evaluating the quality of academic journal is becoming increasing important within the context of research performance evaluation. Traditionally, journals have been ranked by peer review lists such as that of the Association of Business Schools (UK) or though their journal impact factor (JIF). However, several new indicators have been developed, such as the h-index, SJR, SNIP and the Eigenfactor which take into account different factors and therefore have their own particular biases. In this paper we evaluate these metrics both theoretically and also through an empirical study of a large set of business and management journals. We show that even though the indicators appear highly correlated in fact they lead to large differences in journal rankings. We contextualize our results in terms of the UK's large scale research assessment exercise (the RAE/REF) and particularly the ABS journal ranking list. We conclude that no one indicator is superior but that the h-index (which includes the productivity of a journal) and SNIP (which aims to normalize for field effects) may be the most effective at the moment.
△ Less
Submitted 25 April, 2016; v1 submitted 22 April, 2016;
originally announced April 2016.
-
Throwing Out the Baby with the Bathwater: The Undesirable Effects of National Research Assessment Exercises on Research
Authors:
John Mingers,
Leroy White
Abstract:
The evaluation of the quality of research at a national level has become increasingly common. The UK has been at the forefront of this trend having undertaken many assessments since 1986, the latest being the Research Excellence Framework in 2014. The argument of this paper is that, whatever the intended results in terms of evaluating and improving research, there have been many, presumably uninte…
▽ More
The evaluation of the quality of research at a national level has become increasingly common. The UK has been at the forefront of this trend having undertaken many assessments since 1986, the latest being the Research Excellence Framework in 2014. The argument of this paper is that, whatever the intended results in terms of evaluating and improving research, there have been many, presumably unintended, results that are highly undesirable for research and the university community more generally. We situate our analysis using Bourdieu's theory of cultural reproduction and then focus on the peculiarities of the 2008 RAE and the 2014 REF the rules of which allowed for, and indeed encouraged, significant game-playing on the part of striving universities. We conclude with practical recommendations to maintain the general intention of research assessment without the undesirable side-effects.
△ Less
Submitted 2 February, 2015;
originally announced February 2015.
-
A Review of Theory and Practice in Scientometrics
Authors:
John Mingers,
Loet Leydesdorff
Abstract:
Scientometrics is the study of the quantitative aspects of the process of science as a communication system. It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent years it has come to play a major role in the measurement and evaluation of research performance. In this review we consider: the historical development of scientometrics, sources o…
▽ More
Scientometrics is the study of the quantitative aspects of the process of science as a communication system. It is centrally, but not only, concerned with the analysis of citations in the academic literature. In recent years it has come to play a major role in the measurement and evaluation of research performance. In this review we consider: the historical development of scientometrics, sources of citation data, citation metrics and the "laws" of scientometrics, normalisation, journal impact factors and other journal metrics, visualising and mapping science, evaluation and policy, and future developments.
△ Less
Submitted 7 April, 2015; v1 submitted 22 January, 2015;
originally announced January 2015.
-
Identifying Research Fields within Business and Management: A Journal Cross-Citation Analysis
Authors:
John Mingers,
Loet Leydesdorff
Abstract:
A discipline such as business and management (B&M) is very broad and has many fields within it, ranging from fairly scientific ones such as management science or economics to softer ones such as information systems. There are at least two reasons why it is important to identify these sub-fields accurately. Firstly, for the purpose of normalizing citation data as it is well known that citation rate…
▽ More
A discipline such as business and management (B&M) is very broad and has many fields within it, ranging from fairly scientific ones such as management science or economics to softer ones such as information systems. There are at least two reasons why it is important to identify these sub-fields accurately. Firstly, for the purpose of normalizing citation data as it is well known that citation rates vary significantly between different disciplines. Secondly, because journal rankings and lists tend to split their classifications into different subjects, for example the the Association of Business Schools (ABS) list, which is a standard in the UK, has 22 different fields. Unfortunately, at the moment these are created in an ad hoc manner with no underlying rigour. The purpose of this paper is to identify possible sub-fields in B&M rigorously based on actual citation patterns. We have examined 450 journals in B&M which are included in the ISI Web of Science (WoS) and analysed the cross-citation rates between them enabling us to generate sets of coherent and consistent sub-fields that minimise the extent to which journals appear in several categories. Implications and limitations of the analysis are discussed
△ Less
Submitted 30 December, 2012;
originally announced December 2012.
