This is a guest post by Andreas Pacher who initiated the Observatory of International Relations (OOIR), a website which tracks Political Science and IR journals to continually list their latest papers. Follow OOIR on Twitter: @ObserveIR.

You may have noticed that the Impact Factors of IR journals are sometimes followed by a statement that it ranks “nth out of 85 journals in the category ‘International Relations’”. For instance, International Organization ranks “1st out of 85 journals in the category ‘International Relations’”, while Alternatives ranks “76 out of 85 journals in the category ‘International Relations’” (in 2017 rankings).

The ranking and categorization are based on Web of Science’s Journal Citation Reports. Now, have you ever wondered about the composition of this category which allegedly comprises 85 IR journals? At a closer look, one finds a club of scholarly journals from various issue areas – from a multitude of academic disciplines whose cultures follow different publication paces and citation patterns, making them qualitatively distinct entities. In other words, the ranking is heterogeneous. Does it not raise the question whether it compares the incomparable?

The list contains outlets such as The Bulletin of the Atomic Scientists, Marine Policy and Space Policy, journals which I seldom encounter in the reference lists of IR papers. Let’s look at one of them as an example.

Marine Policy consistently ranks relatively high with an IF of over 2.0. Its most cited paper in the past 5 years was one about global catch statistics of sharks. The journal’s citation patterns in 2017 reveals that its articles were mainly cited by Ocean & Coastal Management, by ICES Journal of Marine Science, by PLoS ONE and by Fisheries Research. These are journals listed in the categories of ‘Oceanography’ or ‘Marine & Freshwater Biography’, among others, but not in ‘International Relations’. However, in these subject areas, journals obtain IF which are many times over the top IR journals – Oceanography’s highest-ranking journal Annual Review of Marine Science has an IF of higher than 15.0, which is more than three times more than the most-cited IR journal in 2017, International Organization (IF: 4.517). Oceanography simply follows a different scholarly pace than IR.

This perfectly illustrates the heterogeneity: The way in which the ranking measures the 85 journals does not convey a shared meaning across journals; an IF of 2.0 for Marine Policy does not mean the same as an IF of 2.0 for, say, International Studies Perspectives.

If we inspect Web of Science’s category ‘International Relations’ further, we also witness IR’s awkward relations to history, international law, and IPE (International Political Economy). The list contains historical journals such as Current History and War in History or Journal of Cold War Studies – but no Diplomatic History (why not?). It also contains law journals such as Common Market Law Review, American Journal of International Law, and Cornell International Law Journal – but no Global Constitutionalism (why not?). And it contains IPE journals with such verbose names as Emergent Markets Finance and Trade or Asian Journal of WTO & International Health Law and Policy (the journal with the longest name in the list). While I do not wish to dismiss them as irrelevant, are there any IR researchers who regularly check the latest findings from those journals? According to Web of Science’s Journal Citation Reports, again, the journals which most often cite the latter two are not IR outlets.

If the defining foundation of the discipline of IR is its focus on a macro-scale realm regardless of any societal-functional aspects, then one may certainly laud the ranking’s multidisciplinarity. Nevertheless, it seems to me that the discipline’s practical boundaries could be better reflected through a more homogeneous ranking, either based on inter-citation patterns among the compared journals, or on surveys probing those journals which are deemed impactful by the IR researchers themselves. (But then again, I have not ruminated through this completely – perhaps this idea would simply prolong the status quo, which is not really worth striving for.)

Moreover, it may be worth asking the question of ‘missing journals’. I can immediately think of four journals which primarily deal with diplomacy, and which are nevertheless absent from the Web of Science list – namely The Hague Journal of Diplomacy, Diplomacy & Statecraft, Diplomatic History, and Place Branding & Public Diplomacy. Another one which comes to my mind is International Negotiation. IR scholars have published worthwhile papers in each of these outlets. And if a cross-disciplinary inclusion of journals is inevitable, why would sociology or human geography journals be missing? I have the impression that these are much closer tied to IR than Oceanography.

Finally, Web of Science’s club of 85 alleged IR journals also contains a few outlets which mainly publish articles in ‘LOTE’ (languages other than English): Examples are Korea Observer, the Turkish Uluslararasi Iliskiler, and the Norwegian Internasjonal Politikk. While the inclusion of ‘peripheral’ scholarship seems virtuous, this also prompts the question of why just these were added, and not others (e.g. Chinese or Arabic ones, not to speak of other languages).

Certainly, IR’s multidisciplinarity requires multidisciplinary journals – and novel research always takes nourishment from a variety of disciplines. But does this justify the mixture of heterogeneous units from unlike subject areas into a single ranking, in particular if this widespread ranking serves as the main reference point for measuring and comparing IR publications?

I only had a fragmentary look, but perhaps this may already serve as an invitation to raise awareness about the metadata behind the most frequently used ranking of IR journals, and to rethink its current state’s benefits and disadvantages to the discipline.