-
A discussion of measuring the top-1 percent most-highly cited publications: Quality and impact of Chinese papers
Authors:
Caroline S. Wagner,
Lin Zhang,
Loet Leydesdorff
Abstract:
The top 1 percent most highly cited articles are watched closely as the vanguards of the sciences. Using Web of Science data, one can find that China had overtaken the USA in the relative participation in the top 1 percent in 2019, after outcompeting the EU on this indicator in 2015. However, this finding contrasts with repeated reports of Western agencies that the quality of Chinese output in sci…
▽ More
The top 1 percent most highly cited articles are watched closely as the vanguards of the sciences. Using Web of Science data, one can find that China had overtaken the USA in the relative participation in the top 1 percent in 2019, after outcompeting the EU on this indicator in 2015. However, this finding contrasts with repeated reports of Western agencies that the quality of Chinese output in science is lagging other advanced nations, even as it has caught up in numbers of articles. The difference between the results presented here and the previous results depends mainly upon field normalizations, which classify source journals by discipline. Average citation rates of these subsets are commonly used as a baseline so that one can compare among disciplines. However, the expected value of the top 1 percent of a sample of N papers is N 100, ceteris paribus. Using the average citation rates as expected values, errors are introduced by using the mean of highly skewed distributions and a specious precision in the delineations of the subsets. Classifications can be used for the decomposition, but not for the normalization. When the data is thus decomposed, the USA ranks ahead of China in biomedical fields such as virology. Although the number of papers is smaller, China outperforms the US in the field of Business and Finance in the Social Sciences Citation Index when p is less than .05. Using percentile ranks, subsets other than indexing based classifications can be tested for the statistical significance of differences among them.
△ Less
Submitted 1 February, 2022;
originally announced February 2022.
-
How have the Eastern European countries of the former Warsaw Pact developed since 1990? A bibliometric study
Authors:
Marcin Kozak,
Lutz Bornmann,
Loet Leydesdorff
Abstract:
Did the demise of the Soviet Union in 1991 influence the scientific performance of the researchers in Eastern European countries? Did this historical event affect international collaboration by researchers from the Eastern European countries with those of Western countries? Did it also change international collaboration among researchers from the Eastern European countries? Trying to answer these…
▽ More
Did the demise of the Soviet Union in 1991 influence the scientific performance of the researchers in Eastern European countries? Did this historical event affect international collaboration by researchers from the Eastern European countries with those of Western countries? Did it also change international collaboration among researchers from the Eastern European countries? Trying to answer these questions, this study aims to shed light on international collaboration by researchers from the Eastern European countries (Russia, Ukraine, Belarus, Moldova, Bulgaria, the Czech Republic, Hungary, Poland, Romania and Slovakia). The number of publications and normalized citation impact values are compared for these countries based on InCites (Thomson Reuters), from 1981 up to 2011. The international collaboration by researchers affiliated to institutions in Eastern European countries at the time points of 1990, 2000 and 2011 was studied with the help of Pajek and VOSviewer software, based on data from the Science Citation Index (Thomson Reuters). Our results show that the breakdown of the communist regime did not lead, on average, to a huge improvement in the publication performance of the Eastern European countries and that the increase in international co-authorship relations by the researchers affiliated to institutions in these countries was smaller than expected. Most of the Eastern European countries are still subject to changes and are still awaiting their boost in scientific development.
△ Less
Submitted 11 December, 2013;
originally announced December 2013.
-
How to improve the prediction based on citation impact percentiles for years shortly after the publication date?
Authors:
Lutz Bornmann,
Loet Leydesdorff,
Jian Wang
Abstract:
The findings of Bornmann, Leydesdorff, and Wang (in press) revealed that the consideration of journal impact improves the prediction of long-term citation impact. This paper further explores the possibility of improving citation impact measurements on the base of a short citation window by the consideration of journal impact and other variables, such as the number of authors, the number of cited r…
▽ More
The findings of Bornmann, Leydesdorff, and Wang (in press) revealed that the consideration of journal impact improves the prediction of long-term citation impact. This paper further explores the possibility of improving citation impact measurements on the base of a short citation window by the consideration of journal impact and other variables, such as the number of authors, the number of cited references, and the number of pages. The dataset contains 475,391 journal papers published in 1980 and indexed in Web of Science (WoS, Thomson Reuters), and all annual citation counts (from 1980 to 2010) for these papers. As an indicator of citation impact, we used percentiles of citations calculated using the approach of Hazen (1914). Our results show that citation impact measurement can really be improved: If factors generally influencing citation impact are considered in the statistical analysis, the explained variance in the long-term citation impact can be much increased. However, this increase is only visible when using the years shortly after publication but not when using later years.
△ Less
Submitted 19 November, 2013;
originally announced November 2013.
-
Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P100)
Authors:
Lutz Bornmann,
Loet Leydesdorff,
Jian Wang
Abstract:
Percentile-based approaches have been proposed as a non-parametric alternative to parametric central-tendency statistics to normalize observed citation counts. Percentiles are based on an ordered set of citation counts in a reference set, whereby the fraction of papers at or below the citation counts of a focal paper is used as an indicator for its relative citation impact in the set. In this stud…
▽ More
Percentile-based approaches have been proposed as a non-parametric alternative to parametric central-tendency statistics to normalize observed citation counts. Percentiles are based on an ordered set of citation counts in a reference set, whereby the fraction of papers at or below the citation counts of a focal paper is used as an indicator for its relative citation impact in the set. In this study, we pursue two related objectives: (1) although different percentile-based approaches have been developed, an approach is hitherto missing that satisfies a number of criteria such as scaling of the percentile ranks from zero (all other papers perform better) to 100 (all other papers perform worse), and solving the problem with tied citation ranks unambiguously. We introduce a new citation-rank approach having these properties, namely P100. (2) We compare the reliability of P100 empirically with other percentile-based approaches, such as the approaches developed by the SCImago group, the Centre for Science and Technology Studies (CWTS), and Thomson Reuters (InCites), using all papers published in 1980 in Thomson Reuters Web of Science (WoS). How accurately can the different approaches predict the long-term citation impact in 2010 (in year 31) using citation impact measured in previous time windows (years 1 to 30)? The comparison of the approaches shows that the method used by InCites overestimates citation impact (because of using the highest percentile rank when papers are assigned to more than a single subject category) whereas the SCImago indicator shows higher power in predicting the long-term citation impact on the basis of citation rates in early years. Since the results show a disadvantage in this predictive ability for P100 against the other approaches, there is still room for further improvements.
△ Less
Submitted 17 September, 2013; v1 submitted 19 June, 2013;
originally announced June 2013.
-
The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000
Authors:
Lutz Bornmann,
Loet Leydesdorff
Abstract:
The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the…
▽ More
The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the 'true' correlation with the ratings reaches at least a medium effect size.
△ Less
Submitted 6 November, 2012;
originally announced November 2012.
-
The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits
Authors:
Lutz Bornmann,
Loet Leydesdorff,
Ruediger Mutz
Abstract:
Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describe…
▽ More
Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment.
△ Less
Submitted 2 November, 2012;
originally announced November 2012.
-
Statistical Tests and Research Assessments: A comment on Schneider (2012)
Authors:
Lutz Bornmann,
Loet Leydesdorff
Abstract:
In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect…
▽ More
In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect size measures to research evaluations, but disagree that these procedures would replace significance testing. Accordingly, effect size measures were added to the Excel sheets that we bring online for testing performance differences between institutions in the Leiden Ranking and the SCImago Institutions Ranking.
△ Less
Submitted 12 October, 2012;
originally announced October 2012.
-
Does the specification of uncertainty hurt the progress of scientometrics?
Authors:
Loet Leydesdorff
Abstract:
In "Caveats for using statistical significance tests in research assessments,"--Journal of Informetrics 7(1)(2013) 50-62, available at arXiv:1112.2516 -- Schneider (2013) focuses on Opthof & Leydesdorff (2010) as an example of the misuse of statistics in the social sciences. However, our conclusions are theoretical since they are not dependent on the use of one statistics or another. We agree with…
▽ More
In "Caveats for using statistical significance tests in research assessments,"--Journal of Informetrics 7(1)(2013) 50-62, available at arXiv:1112.2516 -- Schneider (2013) focuses on Opthof & Leydesdorff (2010) as an example of the misuse of statistics in the social sciences. However, our conclusions are theoretical since they are not dependent on the use of one statistics or another. We agree with Schneider insofar as he proposes to develop further statistical instruments (such as effect sizes). Schneider (2013), however, argues on meta-theoretical grounds against the specification of uncertainty because, in his opinion, the presence of statistics would legitimate decision-making. We disagree: uncertainty can also be used for opening a debate. Scientometric results in which error bars are suppressed for meta-theoretical reasons should not be trusted.
△ Less
Submitted 16 November, 2012; v1 submitted 24 September, 2012;
originally announced September 2012.
-
Citation impact of papers published from six prolific countries: A national comparison based on InCites data
Authors:
Lutz Bornmann,
Loet Leydesdorff
Abstract:
Using the InCites tool of Thomson Reuters, this study compares normalized citation impact values calculated for China, Japan, France, Germany, United States, and the UK throughout the time period from 1981 to 2010. The citation impact values are normalized to four subject areas: natural sciences; engineering and technology; medical and health sciences; and agricultural sciences. The results show a…
▽ More
Using the InCites tool of Thomson Reuters, this study compares normalized citation impact values calculated for China, Japan, France, Germany, United States, and the UK throughout the time period from 1981 to 2010. The citation impact values are normalized to four subject areas: natural sciences; engineering and technology; medical and health sciences; and agricultural sciences. The results show an increasing trend in citation impact values for France, the UK and especially for Germany across the last thirty years in all subject areas. The citation impact of papers from China is still at a relatively low level (mostly below the world average), but the country follows an increasing trend line. The USA exhibits a relatively stable pattern of high citation impact values across the years. With small impact differences between the publication years, the US trend is increasing in engineering and technology but decreasing in medical and health sciences as well as in agricultural sciences. Similar to the USA, Japan follows increasing as well as decreasing trends in different subject areas, but the variability across the years is small. In most of the years, papers from Japan perform below or approximately at the world average in each subject area.
△ Less
Submitted 3 May, 2012;
originally announced May 2012.
-
Accounting for the Uncertainty in the Evaluation of Percentile Ranks
Authors:
Loet Leydesdorff
Abstract:
In a recent paper entitled "Inconsistencies of Recently Proposed Citation Impact Indicators and how to Avoid Them," Schreiber (2012, at arXiv:1202.3861) proposed (i) a method to assess tied ranks consistently and (ii) fractional attribution to percentile ranks in the case of relatively small samples (e.g., for n < 100). Schreiber's solution to the problem of how to handle tied ranks is convincing,…
▽ More
In a recent paper entitled "Inconsistencies of Recently Proposed Citation Impact Indicators and how to Avoid Them," Schreiber (2012, at arXiv:1202.3861) proposed (i) a method to assess tied ranks consistently and (ii) fractional attribution to percentile ranks in the case of relatively small samples (e.g., for n < 100). Schreiber's solution to the problem of how to handle tied ranks is convincing, in my opinion (cf. Pudovkin & Garfield, 2009). The fractional attribution, however, is computationally intensive and cannot be done manually for even moderately large batches of documents. Schreiber attributed scores fractionally to the six percentile rank classes used in the Science and Engineering Indicators of the U.S. National Science Board, and thus missed, in my opinion, the point that fractional attribution at the level of hundred percentiles-or equivalently quantiles as the continuous random variable-is only a linear, and therefore much less complex problem. Given the quantile-values, the non-linear attribution to the six classes or any other evaluation scheme is then a question of aggregation. A new routine based on these principles (including Schreiber's solution for tied ranks) is made available as software for the assessment of documents retrieved from the Web of Science (at http://www.leydesdorff.net/software/i3).
△ Less
Submitted 9 April, 2012;
originally announced April 2012.
-
Which are the best cities for psychology research worldwide? A map visualizing city ratios of observed and expected numbers of highly-cited papers
Authors:
Lutz Bornmann,
Loet Leydesdorff,
Günter Krampen
Abstract:
We present scientometric results about world-wide centers of excellence in psychology. Based on Web of Science data, domain-specific excellence can be identified for cities where highly cited papers are published. Data refer to all psychology articles published in 2007 which are documented in the Social Science Citation Index and to their citation frequencies from 2007 to May 2011. Visualized are…
▽ More
We present scientometric results about world-wide centers of excellence in psychology. Based on Web of Science data, domain-specific excellence can be identified for cities where highly cited papers are published. Data refer to all psychology articles published in 2007 which are documented in the Social Science Citation Index and to their citation frequencies from 2007 to May 2011. Visualized are 214 cities with an article output of at least 50 in 2007. Statistical z tests are used for the evaluation of the degree to which an observed number of top-cited papers (top-10%) for a city differs from the number expected on the basis of randomness in the selection of papers. Map visualizing city ratios on significant differences between observed and expected numbers of highly-cited papers point at excellence centers in cities at the East and West Coast of the United States as well as in Great Britain, Germany, the Netherlands, Ireland, Belgium, Sweden, Finland, Australia, and Taiwan. Furthermore, positive but non-significant differences in favor of high citation rates are documented for some cities in the United States, Great Britain, the Netherlands, the Scandinavian and the German-speaking countries, Belgium, France, Spain, Israel, South Korea, and China. Scientometric results show convincingly that highly-cited psychological research articles come from the Anglo-American countries and some of the non-English European countries in which the number of English-language publications has increased during the last decades.
△ Less
Submitted 26 July, 2011;
originally announced July 2011.
-
The semantic mapping of words and co-words in contexts
Authors:
Loet Leydesdorff,
Kasper Welbers
Abstract:
Meaning can be generated when information is related at a systemic level. Such a system can be an observer, but also a discourse, for example, operationalized as a set of documents. The measurement of semantics as similarity in patterns (correlations) and latent variables (factor analysis) has been enhanced by computer techniques and the use of statistics; for example, in "Latent Semantic Analysis…
▽ More
Meaning can be generated when information is related at a systemic level. Such a system can be an observer, but also a discourse, for example, operationalized as a set of documents. The measurement of semantics as similarity in patterns (correlations) and latent variables (factor analysis) has been enhanced by computer techniques and the use of statistics; for example, in "Latent Semantic Analysis". This communication provides an introduction, an example, pointers to relevant software, and summarizes the choices that can be made by the analyst. Visualization ("semantic mapping") is thus made more accessible.
△ Less
Submitted 29 January, 2011; v1 submitted 23 November, 2010;
originally announced November 2010.
-
The relation between Pearson's correlation coefficient r and Salton's cosine measure
Authors:
Leo Egghe,
Loet Leydesdorff
Abstract:
The relation between Pearson's correlation coefficient and Salton's cosine measure is revealed based on the different possible values of the division of the L1-norm and the L2-norm of a vector. These different values yield a sheaf of increasingly straight lines which form together a cloud of points, being the investigated relation. The theoretical results are tested against the author co-citatio…
▽ More
The relation between Pearson's correlation coefficient and Salton's cosine measure is revealed based on the different possible values of the division of the L1-norm and the L2-norm of a vector. These different values yield a sheaf of increasingly straight lines which form together a cloud of points, being the investigated relation. The theoretical results are tested against the author co-citation relations among 24 informetricians for whom two matrices can be constructed, based on co-citations: the asymmetric occurrence matrix and the symmetric co-citation matrix. Both examples completely confirm the theoretical results. The results enable us to specify an algorithm which provides a threshold value for the cosine above which none of the corresponding Pearson correlations would be negative. Using this threshold value can be expected to optimize the visualization of the vector space.
△ Less
Submitted 6 November, 2009;
originally announced November 2009.
