gms | German Medical Science

In comparison with other medical disciplines and based on citation metrics, German epidemiology is remarkably strong: this is the conclusion reached by H.-E. Wichmann, following careful review of the “Stanford list” in its version of 2020 [1], [2]. According to his count and definition, there are 54 epidemiologists with German affiliation in this compilation of approximately 150,000 authors worldwide. Epidemiologists account for only about 1.5% of all German health scientists on the Stanford list. However, among the most highly cited, almost every fifth scientist is an epidemiologist.

Publishing such an article may cause discomfort, especially for a journal editor who wrote her epidemiological master thesis on the topic of publication bias [3]: All that needs to be said about publishing and scientometrics seemed already written more than 25 years ago. “Underreporting research is scientific misconduct” [4] on the one hand, and “We need less research, better research, and research for the right reasons” [5] on the other hand: While publication activity is indispensable and an indicator of active, relevant research, publication pressure, the sheer quantity, is a clear disincentive. As the authors of a review paper on the assessment of scientists put it, citing Goodhart’s Law: [an assessment measure potentially] “ceases to be a valid measurement when it becomes an optimization target.” [6]

For the year of Altman’s ”Scandal” essay (1994), Pubmed records 438,248 citations. In 2005, the figure was already 700,342. This was the year when John Ioannidis, now a co-author of the Stanford list, denounced the lack of reproducibility in medical research. Not quite as explicitly as Altman before him, he identified misguided research motivation as part of the causal chain (“Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure”) [7]. As highly regarded and cited as this work was: By 2019, citation numbers in Pubmed have doubled again to almost 1.5 million hits per year (Nov 9, 2021, search term: year[dp]). We are afraid that the years 2020 and 2021 will climb to sad peaks, much of it waste [8], [9].

To reward researchers all over the world for contributing to this inflation while not considering reproducibility and quality seems absurd. Periodic rally to the problem as done by the Lancet series on research waste or excellent single activities of individuals does not seem sufficient to change the game [10], [11], [12]. Thus why would we want to go on playing the numbers game? Whom and what do we serve by reporting on citation counts?

We will, of course, publish the paper. We are pleased the MIBE was chosen for this. Working with the Stanford list may promote important critical discussion of quantitative measurement of research output. It also allows for observations on how research is published and cited in different scientific fields. Both H.-E. Wichmann and the authors of the Stanford list explicitly point out that all parameters discussed here are unsuitable for evaluating younger researchers in particular.

Two exemplary observations from this international dataset are shared, complementing the analysis by H.-E. Wichmann:

1.

Epidemiologists publish primarily in clinical journals. The Stanford list uses a journal-based classification to categorize researchers, resulting in 22 fields and 179 subfields. “Epidemiology” is a subfield of “Public Health & Health Services”. For example, a person whose most frequent subfield is “Epidemiology” would be considered an epidemiologist. Still, even those whose most frequent subfield is “Epidemiology“ publish, on average, only about one-third of their articles in epidemiological journals. As a side note, the same applies to “Public Health“ and to “Health Services”, where the proportion is even lower. By contrast, nursing scientists and clinical researchers publish predominantly within their own fields. Against this international background, it is not surprising that all German epidemiologists identified by H.-E. Wichmann were assigned to the main field “Clinical Medicine”, rather than “Public Health & Health Services”.

2.

On the international level, epidemiologists are among the most frequently cited scientists. In analogy to the observations for German researchers and subject to differing definitions, this becomes more evident in the top-cited group. Interestingly, among the life sciences, the field “Public Health & Health Services“ shows by far the lowest citation numbers for specific percentiles (Table 1 [Tab. 1]). However, on the level of subfields, epidemiology falls out of line. For example, nearly 14,000 citations mark the 99% rank. At the other end of the extreme, in nursing sciences, 2,300 citations are sufficient to make it into the top 1%. This gap widens further when more recent data are used as available since October 2021 (https://elsevier.digitalcommonsdata.com/datasets/btchxktzyw/3).

Among all 179 subfields, “Epidemiology“ ranks fourth behind “Nuclear physics”, “Developmental biology”, and “Astronomy”, ahead of but closely followed by first “Immunology“ and then the major clinical subjects of cardiovascular diseases/blood, oncology, and metabolism. Of similar magnitude at ninth place is Bioinformatics. In comparison with the other data sciences: “Medical Informatics“ is in 117^th place. Biometrics would probably best be assigned to the subcategory “Statistics and probability“ (38^th place).

What does this tell us? H.-E. Wichmann cites the high quality and relevance of epidemiological studies, often conducted by consortia. We may also conclude we are successful primarily through content-motivated research questions and publications, especially within the highly cited complexes of cardiovascular medicine, oncology, and diabetes/metabolic diseases. Articles that distinguish us as a methodological subject and would thus appear in subject-specific journals are less common and less visible. When we bundle epidemiology, biometry, and medical informatics under the common roof of “The data sciences”, we may wish to consider this: Epidemiology has always been more than just a methodological subject and a set of data collection and analysis tools. It is the combination of content and methods that sets us apart and makes us visible.

All citation and publication measures currently in common use entail the problem of wrong incentives [6]. Many are subject to influencing factors that have little to do with research quality: Age (many years of research activity), membership in large consortia (many authors per article), taking broad views on co-authorships and self-citation, and a focus on en-vogue research while avoiding novel, risky projects. Depending on the type of measure used, data dredging and salami-slicing (number of publications) or suppression of not-so-exciting results with the consequence of publication bias (citations per paper, Hirsch index) may be added to this list. The journal impact factor is not discussed here as it is an obsolete measure. In epidemiology, with its often large projects of long duration, one might, in addition, recommend focusing activities to the “exploitation phase“ while stepping back from those parts of the research process which are not immediately publishable. At least one needs to stay put long enough for harvest.

The composite score combines several measures and thus potentially compensates for the various disadvantages of the different approaches. Most importantly, it corrects for co-authorship, unlike, for example, the Hirsch index, for which it is irrelevant whether one is first or senior author or ranked 16^th out of 53. What does this mean for epidemiologists? As H.-E. Wichmann points out, the rating of epidemiologists relative to other medical subjects drops when the composite score is used instead of the citation frequency – a sign that citation frequency in our discipline is much about participation in large projects and consortia. Does the score compensate for other disincentives? Not sure. Rather, even more may arise: In the composite score, long-time heads of large departments with a “boss is always senior author“ motto are likely to win (or the prince royale if favorably placed on author lists). In this respect, it is not only due to typical characteristics of consortial research in epidemiology but a good signal if German epidemiologists, on average, do not benefit from the composite score.

The Stanford list does not give any information on whether someone considers himself an epidemiologist or whether he is perhaps only involved in epidemiological research as a clinical expert (lack of specificity). More importantly, epidemiologists who publish predominantly in a clinical subfield do not count (lack of sensitivity). For Germany, we are obliged to H.-E. Wichmann as an epidemiologist for his definition: An epidemiologist is affiliated with an epidemiological research institution, is active in the epidemiological scientific community within one of the major societies, or takes an active role in a relevant epidemiology project. This definition is not without potential for misclassification either. Epidemiologist is not a protected term; there is no sharp demarcation to related fields, simultaneous self-attribution to several scientific specialties is conceivable, and self-labeling as epidemiologist is potentially subject to temporal trends (increase in a pandemic?). Virologists, microbiologists, and hygienists also publish in journals classified as epidemiological, without necessarily seeing themselves as epidemiologists or being classified as such by H.-E. Wichmann.

As a spokesperson for the Epidemiology Certificate Commission in Germany, the author of this editorial feels obliged to point out a self-evident solution for defining an epidemiologist. The certificate is awarded jointly by the four societies DGEpi, GMDS, DGSMP, and IBS-DR. It attests advanced and comprehensive knowledge and research experience in epidemiology [13]. Of 13,000 epidemiologists in Germany (number estimated by H.-E. Wichmann), 97 scientists hold this certificate, i.e. about 0.7%. 27 of these 97 scientists with German affiliation are on the Stanford list and represent more than 50% of the German epidemiologists identified by H.-E. Wichmann. Four additional certificate holders are on the list with non-German affiliations (3 USA, 1 Netherlands).

One last note on proportions: Nine of the 54 top epidemiologists on the Wichmann list are women. This is not even one fifth, slightly below the proportion of women with an epidemiology certificate (19 of 97) and considerably lower than what seems common in German epidemiological institutes. Based on empirical evidence, scientometrics may put women at a disadvantage [14], [15]. Reasons include, among other things, frequently interrupted research vitae, confounding by age, and possibly, being less inclined to self-citation.

The Stanford list and its composite score contribute to a practice of research measurement we should view critically. High citation frequency mirrors high relevance of research – no offense intended to the top-listed individuals. Being on the list, however, also comes with mere seniority and with presiding over a large epidemiological institute for a long time. Those committed primarily to advancing methods, those who do not publish for the major clinical disciplines, and those still on the circuit for a professorship are less likely to make it on the list. On the pro-side of the list are the transparency and public accessibility of the data material, which allows for simultaneous consideration of different parameters.

To clarify who is an epidemiologist, we suggest the reader considers obtaining a certificate. In order to improve the research situation, those who already hold a chair may feel encouraged to do what the mid-level faculty can only do to a limited extent: advocate alternative ways of evaluating research and staff. Examples are plentiful [12]. We cordially invite further discussion of the topic. Additional contributions to the MIBE are welcome!

Antje Timmer is a Certification Commission on Epidemiology spokesperson and represents the GMDS in this role.