In line with broader structural reconfigurations requiring performance evaluation, in the last decades the social sciences have been transformed by the new culture of evaluation. Universities, research centers, higher education systems are increasingly adopting programs of evaluation defined through metrics and criteria of measurement of performances. But far from being a self-evident exercise, metrics themselves are performative. So, the more evaluation through quantification is put to work, the more it changes the way knowledge production is performed.

In the attempt to find the more and more standardized measures, the quantification of scientific quality introduced new constraints, new opportunities but also new risks for the work of social scientists. While in so doing scientific disciplines are facing the disciplining power of new evaluation tools, the global rise of a new “governance by numbers” (Shore & Wright, 2015) is triggering new controversies about what count as valuable. While there’s already a relevant literature on the history of quantification in the social sciences (Desrosières, 2016), less is known about the mechanisms and dynamics triggered by the rise of quantification in the evaluation of social scientific knowledge. Thus, research on the accountability of scientific work is necessary to put under scrutiny the changing meaning of scientific activity and what is considered valuable in the scientific craft.

The Quantified Scholar gives us the opportunity to discuss several challenges that social scientists are facing in the ever-changing culture of evaluation (Pardo-Guerra, 2022). For several reasons, the book by Pardo-Guerra is at the same time an exercise of reflexivity on the real meanings and motivational sources that shape our profession as much as a realist account of the new evaluative tools that shape its everyday life. Captured in a constant struggle between the ideal definition as a mission and the real function as a work/service, the social sciences are nowadays redefining the meaning of “relevant knowledge production”, the professional goals and the relationship they entertain with society more broadly.

In this dialectic, both external pressures and internal dynamics reconfigures the relation between discipline and objects while redefining what is considered relevant and valuable.

The more we are exposed to rating and ranking systems, the more a sociological analysis of quantification is required to investigate which values and goals are embedded in the ‘numbers that count’ for evaluation.

As Pardo-Guerra (2022) reminds us: “Like farmers, blacksmiths, grocers, tailors, and marketing specialists, scholars are workers” (p. 53). Like these crafts, knowledge work is equally subjected to managerial scrutiny, to logics of accountability, resource allocation, budgetary constraints and routinary processes of evaluation of scores and results. Knowledge work takes places in specific institutional settings that organize incentives and sanctions, standards and procedures, as much as it triggers specific forms of competition and collaboration among peers.

Notwithstanding the very famous Weberian conception of Beruf to identify intellectual labor (Weber, 1917) — the relevance and necessity of a deep, inner vocation to practice a profession — other external variables contribute to shape the way knowledge work is materially performed in the everyday life of the scientific communities and institutions we inhabit and in the value we assign to the products of intellectual work. The operations and logics we put in place to evaluate scholarly work are thus important instruments that not only assess and classify things, but that also contribute, like in a Mertonian self-fulfilling prophecy, to reshape the strategies, practices and maybe the culture of workers of science.

So how does this very specific work change under the new quantitative culture of evaluation? The Quantified Scholar brings us to detect the effects of quantified evaluation systems on various disciplines within the realm of the social sciences (economics, political science, sociology, anthropology). For this careful inquiry Juan Pablo Pardo-Guerra mobilizes a wide array of methods from computational analysis to in-depth interviews and conduct an in-depth, comparative sociological inquiry of the British case, an environment he has directly experienced as a scholar.

However, the book raises important questions that go well beyond the British case. For this reason, the Focus section of Sociologica hosts here the critical reactions of two scholars that, while holding positions in European universities where they practice professional social sciences, are specialist of different disciplines, hold a different international background, and work in countries that experienced a different institutional history of evaluation of academic labor.

In his essay, Étienne Ollion (2023), French scholar at the intersection between political sociology and computational social sciences, mentions the differences that academic evaluation had in France. Although measurement of academic performance had a more limited impact in the French space of social sciences, Ollion raises interesting hypotheses: “school oriented” approaches — which made of France the space of “Grand Theorists” — as much as interdisciplinarity are nowadays decreasing. At the same time, Ollion questions what is specifically relevant (and problematic) of REF (Research Excellence Framework). Several implications emerge, whether we consider the pressure to publish in specific formats or in international spaces or whether we aim to be relevant for an external audience (e.g., the media, the policymakers). In this way, he invites to investigate more specific dimensions that make “evaluation” impactful (and problematic) for the present and future of scholarly work.

The second comment is by Andrea Saltelli (2023), a mathematician with an international background developed in Europe and serving as an expert for European institutions, which work has not only focused on theoretical and applied modeling but has also investigated the “politics of modeling” (Saltelli & Di Fiore, 2023). The points he raises and the references he makes in his essay push indeed Pardo-Guerra to look at the “political” implications of quantification. If quantification has been described also in its ideological purpose — a “management by numbers” — it has triggered new forms of resistance and countermovements that, rather than refusing it, have produced fairer and alternative forms of quantification.

The conclusions of The Quantified Scholar bring us in the end to have a more detailed and maybe nuanced view of the implications of quantified research evaluation programs as they have been put in place in the British case. Its general concerns are useful and informative as much as they can in some cases sound alarming. By standardizing its practices, British social scientists have conducted less risky and innovative research, thus suggesting new risks of homogenization and loss of disciplinary biodiversity.

In any case, further research is needed on how evaluation and quantification is changing the meaning and value of scientific work. A “science of science” that scrutinizes its material conditions and organizational practices and that takes seriously the idea of being performed in a specific “workplace”. In some ways, scientific workplaces are indeed affected by similar processes of managerialization and quantification that characterize many other working environments. But also, they are populated by collectives of workers equipped with very precious resources that can co-determine the rules of the game as much as producing novel, transformative processes of “reactivity”. For sure, the following comments will trigger further reactions.