Showing 1–27 of 27 results for author: Bornmann, L

Introducing multiverse analysis to bibliometrics: The case of team size effects on disruptive research

Authors: Christian Leibel, Lutz Bornmann

Abstract: Although bibliometrics has become an essential tool in the evaluation of research performance, bibliometric analyses are sensitive to a range of methodological choices. Subtle choices in data selection, indicator construction, and modeling decisions can substantially alter results. Ensuring robustness, meaning that findings hold up under different reasonable scenarios, is therefore critical for cr… ▽ More Although bibliometrics has become an essential tool in the evaluation of research performance, bibliometric analyses are sensitive to a range of methodological choices. Subtle choices in data selection, indicator construction, and modeling decisions can substantially alter results. Ensuring robustness, meaning that findings hold up under different reasonable scenarios, is therefore critical for credible research and research evaluation. To address this issue, this study introduces multiverse analysis to bibliometrics. Multiverse analysis is a statistical tool that enables analysts to transparently discuss modeling assumptions and thoroughly assess model robustness. Whereas standard robustness checks usually cover only a small subset of all plausible models, multiverse analysis includes all plausible models. We illustrate the benefits of multiverse analysis by testing the hypothesis posed by Wu et al. (2019) that small teams produce more disruptive research than large teams. While we found robust evidence of a negative effect of team size on disruption scores, the effect size is so small that its practical relevance seems questionable. Our findings underscore the importance of assessing the multiverse robustness of bibliometric results to clarify their practical implications. △ Less

Submitted 4 June, 2025; originally announced June 2025.

Comments: 50 pages, 3 figures, 11 tables

ACM Class: J.4

Metrics sonification: The introduction of new ways to present bibliometric data using publication data of Loet Leydesdorff as an example

Authors: Lutz Bornmann, Rouven Lazlo Haegner

Abstract: The visualization of publication and citation data is popular in bibliometrics. Although less common, the representation of empirical data as sound is an alternative form of presentation (in other fields than bibliometrics). In this representation, the data are mapped into sound and listened to by an audience. Approaches for the sonification of data have been developed in many fields since decades… ▽ More The visualization of publication and citation data is popular in bibliometrics. Although less common, the representation of empirical data as sound is an alternative form of presentation (in other fields than bibliometrics). In this representation, the data are mapped into sound and listened to by an audience. Approaches for the sonification of data have been developed in many fields since decades. Since sonification has several advantages for the presentation of data, this study is intended to introduce sonification to bibliometrics named as 'metrics sonification'. Metrics sonification is defined as the sonification of bibliometric information (measurements, data or results) for their empirical analysis and/or presentation. In this study, we used metadata of publications by Loet Leydesdorff (named as Loet in the following) to sonify their properties. Loet was a giant in the field of scientometrics, who passed away in 2023. The track based on Loet's publications can be listened to on SoundCloud using the following link: https://on.soundcloud.com/oxBTA32x4EgwvKVz5. The track has been composed in F minor; this key was chosen to express the sad occasion. The quantitative part of the track includes a parameter mapping (a sonification) of three properties of his publications: (1) publication output, (2) open access publication, and (3) citation impact of publications. The qualitative part (spoken audio) focuses on explanations of the parameter mapping and descriptions of the mapped papers (based on their titles and abstracts). The sonification of Loet's publications presented in this study is only one possible type of metrics sonification application. As the great number of projects from other disciplines have demonstrated, many other types of applications are possible in bibliometrics. △ Less

Submitted 9 June, 2024; originally announced June 2024.

Comments: The paper will be published in a special issue of the journal Scientometrics that is dedicated to Loet Leydesdorff

Empirical analysis of recent temporal dynamics of research fields: Annual publications in chemistry and related areas as an example

Authors: Lutz Bornmann, Robin Haunschild

Abstract: Changes in the number of publications in a certain field might reflect the dynamic of scientific progress in this field, since an increase in the number of publications can be interpreted as an increase in the field-specific knowledge. In this paper, we present a methodological approach to analyse the dynamics of science on lower aggregation levels, i.e., the level of research fields. Our trend an… ▽ More Changes in the number of publications in a certain field might reflect the dynamic of scientific progress in this field, since an increase in the number of publications can be interpreted as an increase in the field-specific knowledge. In this paper, we present a methodological approach to analyse the dynamics of science on lower aggregation levels, i.e., the level of research fields. Our trend analysis approach is able to uncover very recent trends, and the methods used to study the trends are simple to understand for the possible recipients of the results. In order to demonstrate the trend analysis approach, we focused in this study on the annual number of publications (and patents) in chemistry (and related areas) between 2014 and 2020 identifying those fields in chemistry with the highest dynamics (largest rates of change in publication counts). The study is based on the mono-disciplinary literature database CAplus. Our results reveal that the number of publications in the CAplus database is increasing since many years. Research regarding optical phenomena and electrochemical technologies was found to be among the emerging topics in recent years. △ Less

Submitted 30 August, 2022; v1 submitted 12 January, 2022; originally announced January 2022.

Journal ref: Journal of Informetrics, 2022

Excellence networks in science: A Web-based application based on Bayesian multilevel logistic regression (BMLR) for the identification of institutions collaborating successfully

Authors: Lutz Bornmann, Moritz Stefaner, Felix de Moya Anegon, Ruediger Mutz

Abstract: In this study we present an application which can be accessed via www.excellence-networks.net and which represents networks of scientific institutions worldwide. The application is based on papers (articles, reviews and conference papers) published between 2007 and 2011. It uses (network) data, on which the SCImago Institutions Ranking is based (Scopus data from Elsevier). Using this data, institu… ▽ More In this study we present an application which can be accessed via www.excellence-networks.net and which represents networks of scientific institutions worldwide. The application is based on papers (articles, reviews and conference papers) published between 2007 and 2011. It uses (network) data, on which the SCImago Institutions Ranking is based (Scopus data from Elsevier). Using this data, institutional networks have been estimated with statistical models (Bayesian multilevel logistic regression, BMLR) for a number of Scopus subject areas. Within single subject areas, we have investigated and visualized how successfully overall an institution (reference institution) has collaborated (compared to all the other institutions in a subject area), and with which other institutions (network institutions) a reference institution has collaborated particularly successfully. The "best paper rate" (statistically estimated) was used as an indicator for evaluating the collaboration success of an institution. This gives the proportion of highly cited papers from an institution, and is considered generally as an indicator for measuring impact in bibliometrics. △ Less

Submitted 12 January, 2016; v1 submitted 17 August, 2015; originally announced August 2015.

Comments: accepted for publication in the Journal of Informetrics

Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?

Authors: Lutz Bornmann, Werner Marx

Abstract: Evaluative bibliometrics compares the citation impact of researchers, research groups and institutions with each other across time scales and disciplines. Both factors - discipline and period - have an influence on the citation count which is independent of the quality of the publication. Normalizing the citation impact of papers for these two factors started in the mid-1980s. Since then, a range… ▽ More Evaluative bibliometrics compares the citation impact of researchers, research groups and institutions with each other across time scales and disciplines. Both factors - discipline and period - have an influence on the citation count which is independent of the quality of the publication. Normalizing the citation impact of papers for these two factors started in the mid-1980s. Since then, a range of different methods have been presented for producing normalized citation impact scores. The current study uses a data set of over 50,000 records to test which of the methods so far presented correlate better with the assessment of papers by peers. The peer assessments come from F1000Prime - a post-publication peer review system of the biomedical literature. Of the normalized indicators, the current study involves not only cited-side indicators, such as the mean normalized citation score, but also citing-side indicators. As the results show, the correlations of the indicators with the peer assessments all turn out to be very similar. Since F1000 focuses on biomedicine, it is important that the results of this study are validated by other studies based on datasets from other disciplines or (ideally) based on multi-disciplinary datasets. △ Less

Submitted 9 December, 2014; v1 submitted 20 October, 2014; originally announced October 2014.

Comments: Accepted for publication in the Journal of Informetrics

Which of the world's institutions employ the most highly cited researchers? An analysis of the data from highlycited.com

Authors: Lutz Bornmann, Johann Bauer

Abstract: A few weeks ago, Thomson Reuters published a list of the highly cited researchers worldwide (highlycited.com). Since the data is freely available for downloading and includes the names of the researchers' institutions, we produced a ranking of the institutions on the basis of the number of highly cited researchers per institution. This ranking is intended to be a helpful amendment of other availab… ▽ More A few weeks ago, Thomson Reuters published a list of the highly cited researchers worldwide (highlycited.com). Since the data is freely available for downloading and includes the names of the researchers' institutions, we produced a ranking of the institutions on the basis of the number of highly cited researchers per institution. This ranking is intended to be a helpful amendment of other available institutional rankings. △ Less

Submitted 28 July, 2014; v1 submitted 8 July, 2014; originally announced July 2014.

Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime

Authors: Lutz Bornmann

Abstract: Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be o… ▽ More Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be of interest for science or rather for other segments of society. The results show that papers with the tag "good for teaching" do achieve higher altmetric counts than papers without this tag - if the quality of the papers is controlled. At the same time, a higher citation count is shown especially by papers with a tag that is specifically scientifically oriented ("new finding"). The findings indicate that papers tailored for a readership outside the area of research should lead to societal impact. If altmetric data is to be used for the measurement of societal impact, the question arises of its normalization. In bibliometrics, citations are normalized for the papers' subject area and publication year. This study has taken a second analytic step involving a possible normalization of altmetric data. As the results show there are particular scientific topics which are of especial interest for a wide audience. Since these more or less interesting topics are not completely reflected in Thomson Reuters' journal sets, a normalization of altmetric data should not be based on the level of subject categories, but on the level of topics. △ Less

Submitted 16 February, 2015; v1 submitted 30 June, 2014; originally announced June 2014.

Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Authors: Lutz Bornmann

Abstract: Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an… ▽ More Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed. △ Less

Submitted 10 September, 2014; v1 submitted 27 June, 2014; originally announced June 2014.

Comments: Accepted for publication in the Journal of Informetrics

Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references

Authors: Lutz Bornmann, Ruediger Mutz

Abstract: Many studies in information science have looked at the growth of science. In this study, we re-examine the question of the growth of science. To do this we (i) use current data up to publication year 2012 and (ii) analyse it across all disciplines and also separately for the natural sciences and for the medical and health sciences. Furthermore, the data are analysed with an advanced statistical te… ▽ More Many studies in information science have looked at the growth of science. In this study, we re-examine the question of the growth of science. To do this we (i) use current data up to publication year 2012 and (ii) analyse it across all disciplines and also separately for the natural sciences and for the medical and health sciences. Furthermore, the data are analysed with an advanced statistical technique - segmented regression analysis - which can identify specific segments with similar growth rates in the history of science. The study is based on two different sets of bibliometric data: (1) The number of publications held as source items in the Web of Science (WoS, Thomson Reuters) per publication year and (2) the number of cited references in the publications of the source items per cited reference year. We have looked at the rate at which science has grown since the mid-1600s. In our analysis of cited references we identified three growth phases in the development of science, which each led to growth rates tripling in comparison with the previous phase: from less than 1% up to the middle of the 18th century, to 2 to 3% up to the period between the two world wars and 8 to 9% to 2012. △ Less

Submitted 8 May, 2014; v1 submitted 19 February, 2014; originally announced February 2014.

Comments: Accepted for publication in the Journal of the Association for Information Science and Technology

Sampling Issues in Bibliometric Analysis

Authors: Richard Williams, Lutz Bornmann

Abstract: Bibliometricians face several issues when drawing and analyzing samples of citation records for their research. Drawing samples that are too small may make it difficult or impossible for studies to achieve their goals, while drawing samples that are too large may drain resources that could be better used for other purposes. This paper considers three common situations and offers advice for dealing… ▽ More Bibliometricians face several issues when drawing and analyzing samples of citation records for their research. Drawing samples that are too small may make it difficult or impossible for studies to achieve their goals, while drawing samples that are too large may drain resources that could be better used for other purposes. This paper considers three common situations and offers advice for dealing with each. First, an entire population of records is available for an institution. We argue that, even though all records have been collected, the use of inferential statistics, significance testing, and confidence intervals is both common and desirable. Second, because of limited resources or other factors, a sample of records needs to be drawn. We demonstrate how power analyses can be used to determine in advance how large the sample needs to be to achieve the study's goals. Third, the sample size may already be determined, either because the data have already been collected or because resources are limited. We show how power analyses can again be used to determine how large effects need to be in order to find effects that are statistically significant. Such information can then help bibliometricians to develop reasonable expectations as to what their analysis can accomplish. While we focus on issues of interest to bibliometricians, our recommendations and procedures can easily be adapted for other fields of study. △ Less

Submitted 17 November, 2015; v1 submitted 10 January, 2014; originally announced January 2014.

How have the Eastern European countries of the former Warsaw Pact developed since 1990? A bibliometric study

Authors: Marcin Kozak, Lutz Bornmann, Loet Leydesdorff

Abstract: Did the demise of the Soviet Union in 1991 influence the scientific performance of the researchers in Eastern European countries? Did this historical event affect international collaboration by researchers from the Eastern European countries with those of Western countries? Did it also change international collaboration among researchers from the Eastern European countries? Trying to answer these… ▽ More Did the demise of the Soviet Union in 1991 influence the scientific performance of the researchers in Eastern European countries? Did this historical event affect international collaboration by researchers from the Eastern European countries with those of Western countries? Did it also change international collaboration among researchers from the Eastern European countries? Trying to answer these questions, this study aims to shed light on international collaboration by researchers from the Eastern European countries (Russia, Ukraine, Belarus, Moldova, Bulgaria, the Czech Republic, Hungary, Poland, Romania and Slovakia). The number of publications and normalized citation impact values are compared for these countries based on InCites (Thomson Reuters), from 1981 up to 2011. The international collaboration by researchers affiliated to institutions in Eastern European countries at the time points of 1990, 2000 and 2011 was studied with the help of Pajek and VOSviewer software, based on data from the Science Citation Index (Thomson Reuters). Our results show that the breakdown of the communist regime did not lead, on average, to a huge improvement in the publication performance of the Eastern European countries and that the increase in international co-authorship relations by the researchers affiliated to institutions in these countries was smaller than expected. Most of the Eastern European countries are still subject to changes and are still awaiting their boost in scientific development. △ Less

Submitted 11 December, 2013; originally announced December 2013.

Tracing the origin of a scientific legend by Reference Publication Year Spectroscopy (RPYS): the legend of the Darwin finches

Authors: Werner Marx, Lutz Bornmann

Abstract: In a previews paper we introduced the quantitative method named Reference Publication Year Spectroscopy (RPYS). With this method one can determine the historical roots of research fields and quantify their impact on current research. RPYS is based on the analysis of the frequency with which references are cited in the publications of a specific research field in terms of the publication years of t… ▽ More In a previews paper we introduced the quantitative method named Reference Publication Year Spectroscopy (RPYS). With this method one can determine the historical roots of research fields and quantify their impact on current research. RPYS is based on the analysis of the frequency with which references are cited in the publications of a specific research field in terms of the publication years of these cited references. In this study, we illustrate that RPYS can also be used to reveal the origin of scientific legends. We selected Darwin finches as an example for illustration. Charles Darwin, the originator of evolutionary theory, was given credit for finches he did not see and for observations and insights about the finches he never made. We have shown that a book published in 1947 is the most-highly cited early reference cited within the relevant literature. This book had already been revealed as the origin of the term Darwin finches by Sulloway through careful historical analysis. △ Less

Submitted 22 November, 2013; originally announced November 2013.

Comments: Accepted for publication in Scientometrics

How to improve the prediction based on citation impact percentiles for years shortly after the publication date?

Authors: Lutz Bornmann, Loet Leydesdorff, Jian Wang

Abstract: The findings of Bornmann, Leydesdorff, and Wang (in press) revealed that the consideration of journal impact improves the prediction of long-term citation impact. This paper further explores the possibility of improving citation impact measurements on the base of a short citation window by the consideration of journal impact and other variables, such as the number of authors, the number of cited r… ▽ More The findings of Bornmann, Leydesdorff, and Wang (in press) revealed that the consideration of journal impact improves the prediction of long-term citation impact. This paper further explores the possibility of improving citation impact measurements on the base of a short citation window by the consideration of journal impact and other variables, such as the number of authors, the number of cited references, and the number of pages. The dataset contains 475,391 journal papers published in 1980 and indexed in Web of Science (WoS, Thomson Reuters), and all annual citation counts (from 1980 to 2010) for these papers. As an indicator of citation impact, we used percentiles of citations calculated using the approach of Hazen (1914). Our results show that citation impact measurement can really be improved: If factors generally influencing citation impact are considered in the statistical analysis, the explained variance in the long-term citation impact can be much increased. However, this increase is only visible when using the years shortly after publication but not when using later years. △ Less

Submitted 19 November, 2013; originally announced November 2013.

Comments: Accepted for publication in the Journal of Informetrics. arXiv admin note: text overlap with arXiv:1306.4454

The Wisdom of Citing Scientists

Authors: Lutz Bornmann, Werner Marx

Abstract: This Brief Communication discusses the benefits of citation analysis in research evaluation based on Galton's "Wisdom of Crowds" (1907). Citations are based on the assessment of many which is why they can be ascribed a certain amount of accuracy. However, we show that citations are incomplete assessments and that one cannot assume that a high number of citations correlate with a high level of usef… ▽ More This Brief Communication discusses the benefits of citation analysis in research evaluation based on Galton's "Wisdom of Crowds" (1907). Citations are based on the assessment of many which is why they can be ascribed a certain amount of accuracy. However, we show that citations are incomplete assessments and that one cannot assume that a high number of citations correlate with a high level of usefulness. Only when one knows that a rarely cited paper has been widely read is it possible to say (strictly speaking) that it was obviously of little use for further research. Using a comparison with 'like' data, we try to determine that cited reference analysis allows a more meaningful analysis of bibliometric data than times-cited analysis. △ Less

Submitted 7 August, 2013; originally announced August 2013.

Comments: Accepted for publication in the Journal of the American Society for Information Science and Technology

The normalization of citation counts based on classification systems

Authors: Lutz Bornmann, Werner Marx, Andreas Barth

Abstract: If we want to assess whether the paper in question has had a particularly high or low citation impact compared to other papers, the standard practice in bibliometrics is to normalize citations in respect of the subject category and publication year. A number of proposals for an improved procedure in the normalization of citation impact have been put forward in recent years. Against the background… ▽ More If we want to assess whether the paper in question has had a particularly high or low citation impact compared to other papers, the standard practice in bibliometrics is to normalize citations in respect of the subject category and publication year. A number of proposals for an improved procedure in the normalization of citation impact have been put forward in recent years. Against the background of these proposals this study describes an ideal solution for the normalization of citation impact: in a first step, the reference set for the publication in question is collated by means of a classification scheme, where every publication is associated with a single principal research field or subfield entry (e. g. via Chemical Abstracts sections) and a publication year. In a second step, percentiles of citation counts are calculated for this set and used to assign the normalized citation impact score to the publications (and also to the publication in question). △ Less

Submitted 29 July, 2013; originally announced July 2013.

Comments: Accepted for publication in the journal publications

Is there currently a scientific revolution in scientometrics?

Authors: Lutz Bornmann

Abstract: The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society. The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society. △ Less

Submitted 24 July, 2013; originally announced July 2013.

Comments: Accepted for publication in the Journal of the American Society for Information Science and Technology

Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P100)

Authors: Lutz Bornmann, Loet Leydesdorff, Jian Wang

Abstract: Percentile-based approaches have been proposed as a non-parametric alternative to parametric central-tendency statistics to normalize observed citation counts. Percentiles are based on an ordered set of citation counts in a reference set, whereby the fraction of papers at or below the citation counts of a focal paper is used as an indicator for its relative citation impact in the set. In this stud… ▽ More Percentile-based approaches have been proposed as a non-parametric alternative to parametric central-tendency statistics to normalize observed citation counts. Percentiles are based on an ordered set of citation counts in a reference set, whereby the fraction of papers at or below the citation counts of a focal paper is used as an indicator for its relative citation impact in the set. In this study, we pursue two related objectives: (1) although different percentile-based approaches have been developed, an approach is hitherto missing that satisfies a number of criteria such as scaling of the percentile ranks from zero (all other papers perform better) to 100 (all other papers perform worse), and solving the problem with tied citation ranks unambiguously. We introduce a new citation-rank approach having these properties, namely P100. (2) We compare the reliability of P100 empirically with other percentile-based approaches, such as the approaches developed by the SCImago group, the Centre for Science and Technology Studies (CWTS), and Thomson Reuters (InCites), using all papers published in 1980 in Thomson Reuters Web of Science (WoS). How accurately can the different approaches predict the long-term citation impact in 2010 (in year 31) using citation impact measured in previous time windows (years 1 to 30)? The comparison of the approaches shows that the method used by InCites overestimates citation impact (because of using the highest percentile rank when papers are assigned to more than a single subject category) whereas the SCImago indicator shows higher power in predicting the long-term citation impact on the basis of citation rates in early years. Since the results show a disadvantage in this predictive ability for P100 against the other approaches, there is still room for further improvements. △ Less

Submitted 17 September, 2013; v1 submitted 19 June, 2013; originally announced June 2013.

Comments: Accepted for publication in the Journal of Informetrics

MSC Class: 62Pxx

How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations

Authors: Lutz Bornmann, Werner Marx

Abstract: Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher's productivi… ▽ More Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher's productivity and the impact of his or her publications in a single number (h is the number of publications with at least h citations); however, it is not possible to cover the multidimensional complexity of research performance and to undertake inter-personal comparisons with this number. This study therefore includes recommendations for a set of indicators to be used for evaluating researchers. Our proposals relate to the selection of data on which an evaluation is based, the analysis of the data and the presentation of the results. △ Less

Submitted 14 October, 2013; v1 submitted 15 February, 2013; originally announced February 2013.

Comments: Accepted for publication in Scientometrics

How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects

Authors: Lutz Bornmann, Richard Williams

Abstract: Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are f… ▽ More Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are for institutional evaluative bibliometrics. As an illustration, we will calculate a regression model using publications (and citation data) produced by four universities in German-speaking countries from 1980 to 2010. We will show how these predictions and effects can be estimated and plotted, and how this makes it far easier to get a practical feel for the substantive meaning of results in evaluative bibliometric studies. We will focus particularly on Average Adjusted Predictions (AAPs), Average Marginal Effects (AMEs), Adjusted Predictions at Representative Values (APRVs) and Marginal Effects at Representative Values (MERVs). △ Less

Submitted 9 January, 2013; originally announced January 2013.

Ranking and mapping of universities and research-focused institutions worldwide based on highly-cited papers: A visualization of results from multi-level models

Authors: Lutz Bornmann, Moritz Stefaner, Felix de Moya Anegon, Ruediger Mutz

Abstract: The web application presented in this paper allows for an analysis to reveal centres of excellence in different fields worldwide using publication and citation data. Only specific aspects of institutional performance are taken into account and other aspects such as teaching performance or societal impact of research are not considered. Based on data gathered from Scopus, field-specific excellence… ▽ More The web application presented in this paper allows for an analysis to reveal centres of excellence in different fields worldwide using publication and citation data. Only specific aspects of institutional performance are taken into account and other aspects such as teaching performance or societal impact of research are not considered. Based on data gathered from Scopus, field-specific excellence can be identified in institutions where highly-cited papers have been frequently published. The web application combines both a list of institutions ordered by different indicator values and a map with circles visualizing indicator values for geocoded institutions. Compared to the mapping and ranking approaches introduced hitherto, our underlying statistics (multi-level models) are analytically oriented by allowing (1) the estimation of values for the number of excellent papers for an institution which are statistically more appropriate than the observed values; (2) the calculation of confidence intervals as measures of accuracy for the institutional citation impact; (3) the comparison of a single institution with an "average" institution in a subject area, and (4) the direct comparison of at least two institutions. △ Less

Submitted 24 July, 2013; v1 submitted 3 December, 2012; originally announced December 2012.

Comments: Accepted for publication in the journal Online Information Review

The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000

Authors: Lutz Bornmann, Loet Leydesdorff

Abstract: The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the… ▽ More The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the 'true' correlation with the ratings reaches at least a medium effect size. △ Less

Submitted 6 November, 2012; originally announced November 2012.

The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

Authors: Lutz Bornmann, Loet Leydesdorff, Ruediger Mutz

Abstract: Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describe… ▽ More Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment. △ Less

Submitted 2 November, 2012; originally announced November 2012.

Statistical Tests and Research Assessments: A comment on Schneider (2012)

Authors: Lutz Bornmann, Loet Leydesdorff

Abstract: In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect… ▽ More In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect size measures to research evaluations, but disagree that these procedures would replace significance testing. Accordingly, effect size measures were added to the Excel sheets that we bring online for testing performance differences between institutions in the Leiden Ranking and the SCImago Institutions Ranking. △ Less

Submitted 12 October, 2012; originally announced October 2012.

How to analyse percentile impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes and top-cited papers

Authors: Lutz Bornmann

Abstract: According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be a… ▽ More According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be analysed meaningfully for an evaluation study. Publication sets from four universities are compared with each other to provide sample data. These suggestions take into account on the one hand the distribution of percentiles over the publications in the sets (here: universities) and on the other hand concentrate on the range of publications with the highest citation impact - that is, the range which is usually of most interest in the evaluation of scientific performance. △ Less

Submitted 8 June, 2012; originally announced June 2012.

Citation impact of papers published from six prolific countries: A national comparison based on InCites data

Authors: Lutz Bornmann, Loet Leydesdorff

Abstract: Using the InCites tool of Thomson Reuters, this study compares normalized citation impact values calculated for China, Japan, France, Germany, United States, and the UK throughout the time period from 1981 to 2010. The citation impact values are normalized to four subject areas: natural sciences; engineering and technology; medical and health sciences; and agricultural sciences. The results show a… ▽ More Using the InCites tool of Thomson Reuters, this study compares normalized citation impact values calculated for China, Japan, France, Germany, United States, and the UK throughout the time period from 1981 to 2010. The citation impact values are normalized to four subject areas: natural sciences; engineering and technology; medical and health sciences; and agricultural sciences. The results show an increasing trend in citation impact values for France, the UK and especially for Germany across the last thirty years in all subject areas. The citation impact of papers from China is still at a relatively low level (mostly below the world average), but the country follows an increasing trend line. The USA exhibits a relatively stable pattern of high citation impact values across the years. With small impact differences between the publication years, the US trend is increasing in engineering and technology but decreasing in medical and health sciences as well as in agricultural sciences. Similar to the USA, Japan follows increasing as well as decreasing trends in different subject areas, but the variability across the years is small. In most of the years, papers from Japan perform below or approximately at the world average in each subject area. △ Less

Submitted 3 May, 2012; originally announced May 2012.

Comments: 14 pages

Which are the best cities for psychology research worldwide? A map visualizing city ratios of observed and expected numbers of highly-cited papers

Authors: Lutz Bornmann, Loet Leydesdorff, Günter Krampen

Abstract: We present scientometric results about world-wide centers of excellence in psychology. Based on Web of Science data, domain-specific excellence can be identified for cities where highly cited papers are published. Data refer to all psychology articles published in 2007 which are documented in the Social Science Citation Index and to their citation frequencies from 2007 to May 2011. Visualized are… ▽ More We present scientometric results about world-wide centers of excellence in psychology. Based on Web of Science data, domain-specific excellence can be identified for cities where highly cited papers are published. Data refer to all psychology articles published in 2007 which are documented in the Social Science Citation Index and to their citation frequencies from 2007 to May 2011. Visualized are 214 cities with an article output of at least 50 in 2007. Statistical z tests are used for the evaluation of the degree to which an observed number of top-cited papers (top-10%) for a city differs from the number expected on the basis of randomness in the selection of papers. Map visualizing city ratios on significant differences between observed and expected numbers of highly-cited papers point at excellence centers in cities at the East and West Coast of the United States as well as in Great Britain, Germany, the Netherlands, Ireland, Belgium, Sweden, Finland, Australia, and Taiwan. Furthermore, positive but non-significant differences in favor of high citation rates are documented for some cities in the United States, Great Britain, the Netherlands, the Scandinavian and the German-speaking countries, Belgium, France, Spain, Israel, South Korea, and China. Scientometric results show convincingly that highly-cited psychological research articles come from the Anglo-American countries and some of the non-English European countries in which the number of English-language publications has increased during the last decades. △ Less

Submitted 26 July, 2011; originally announced July 2011.

Comments: 16 pages, two figures

MSC Class: 62-XX

New approaches for increasing the reliability of the h index research performance measurement

Authors: Lutz Bornmann, Ruediger Mutz, Hans-Dieter Daniel

Abstract: In the year 2005 Jorge Hirsch introduced the h index for quantifying the research output of scientists. Today, the h index is a widely accepted indicator of research performance. The h index has been criticized for its insufficient reliability - the ability to discriminate reliably between meaningful amounts of research performance. Taking as an example an extensive data set with bibliometric da… ▽ More In the year 2005 Jorge Hirsch introduced the h index for quantifying the research output of scientists. Today, the h index is a widely accepted indicator of research performance. The h index has been criticized for its insufficient reliability - the ability to discriminate reliably between meaningful amounts of research performance. Taking as an example an extensive data set with bibliometric data on scientists working in the field of molecular biology, we compute h2 lower, h2 upper, and sRM values and present them as complementary approaches that improve the reliability of the h index research performance measurement. △ Less

Submitted 27 August, 2009; originally announced August 2009.

Comments: 20 pages, 2 figures

MSC Class: 62-XX