Two exceptional scholars, Andrea Saltelli and Étienne Ollion, offered their generous and thoughtful comments about The Quantified Scholar (Pardo-Guerra, 2022). I thank them for their wonderful engagement and their poignant discussions. In what follows, I take stock of their observations, expanding on the themes explored in my book and elsewhere.

An accomplished political sociologist who is transforming the methodological vanguard of the field, Étienne Ollion (2023) offers important questions about the book. For Ollion, the question of causality is central. Can we attribute the changes to social science knowledge in the United Kingdom to the research evaluations, or might other dynamics be at play? Could it be, for example, that the patterns observed in the book stem from the need to attend to the demands of the market for students rather than those of the research exercises? Is it, perhaps, a mere consequence of the greater availability of “technologies of ordinalization” that allow scholars to measure themselves and their peers (think: Google Scholar)?

I admit grappling with similar questions as I conducted the research that informed The Quantified Scholar. To what extent were the patterns I encountered merely the consequence of convergence due to competitive pressures on, for example, teaching or of the evaluation itself? Three arguments reassured me of the findings.

The first involves an unsuspecting variable. As I explore in the book, a feature of research evaluations as practiced in the United Kingdom is that they focus on disciplinary units rather than administrative units. The exercises assess “sociology” or “management and business” as fields, not as departments. This leads to sometimes awkward situations. Given that scholars are submitted to the evaluation by their managers, they can be placed in situations where they do not “fit”. An anthropologist at an institution with many sociologists, for example, may be submitted to the “sociology” evaluation panel or not submitted at all. This means their work will not be read by disciplinary peers, leading to a potential penalty in their careers. These “categorical dissonances” are purely administrative artefacts. They derive from how the evaluation is implemented. And yet we find that this situation of categorical dissonance predicts the mobility of scholars to a considerable degree. It tells us, in particular, that being at institutions where one’s work isn’t evaluated by colleagues increases the likelihood of exit. By itself, this variable only tells a correlational story. But tied to a difference in differences analysis, it allows examining whether the evaluations are effectively tied to greater rates of mobility and, by proxy, epistemic and organizational change. I was partly reassured to see that these analyses showed a small but significant effect.

This is still a somewhat limited result. Greater confidence in the explanation came from the specific methodological strategy adopted for this study — what I call the extended computational case method (Pardo-Guerra & Pahwah, 2023). Rather than limiting the analysis to quantitative models supported by qualitative evidence, the strategy I pursued in this project involved looping and iterating across evidentiary domains. In other words, findings from the quantitative models informed questions and conversations with informants, who also provided ideas for additional quantitative studies and qualitative paths. Iterated over different forms of data, this allowed exploring some of the processes and mechanisms that informed the evaluations. The key finding of the book (that it isn’t primarily the evaluations that change knowledge but how we echo our politics of merit onto them) was a product of this iterative strategy. That coherent findings emerged from this abductive framework provided additional confidence in the connections between evaluations and epistemic shifts — not the least because this iterative process showed evaluations to play a less direct role than what I deemed initially.

Other factors can still not be dismissed. Could this not have been all a product of pressures on teaching? It may be true that some of the organizational re-alignments have to do with filling teaching positions on particular topics that might be of interest to prospective students. But it is unclear if students select degrees based on what classes are offered (the evidence suggests they do not; institutional status is a stronger signal) or if universities react to the market rather than mimic their peers (e.g. White, 1983). That there is no similar vocabulary as “REFable” for teaching, both institutionally and vocationally, or that institutional retentions of staff on the basis of exceptional teaching are relatively rare, suggests that research was the horse driving the cart.

Ollion’s other question — “What, in the REF, matters to the observed phenomena?” — is relevant here. As it came across in conversations and reflections, the power of the REF was multiple. It was, indeed, an administrative instrument that disciplined our fields through practices akin to peer review. It created new languages of “impact” and “exchange” that shifted how we think of our contributions. But it also served as a resonance box for longstanding practices of merit and prestige. This, I think, is what matters most of REF, not so much how it worked or counted things but how it incentivized behaviors that, openly or not, catered to specific hierarchies of merit and prestige. The REF, like a market, was a mirror that reflected, distorted, and amplified some of the features of our craft. Some mirrors are better than others.

Andrea Saltelli (2023) offers a distinctly critical take. To summarize his argument, The Quantified Scholar lacks politics. Where are the discussions of the role of New Public Management in transforming British public research institutions? Where does it engage with the relentless marketization of knowledge? “The word, ‘neoliberal’”, we are reminded, “appears the first time in the book on page 47” (p. 158).

Without any doubt, public research institutions in the United Kingdom have been transformed by the arsenal of neoliberal ideas that imbued British policymaking since the early 1980s. A distinct turn toward markets and accountability changed how universities crafted their organizational strategies, hired personnel, and directed their research. Many exceptional studies, some of which Saltelli cites, have documented the consequences of these transformations. The work on this is vast.

Yet central to this vast body of research about the effects of neoliberal politics in higher education and public knowledge systems, there is a stubborn complication. As a systematic pressure largely understood as being external to academia, neoliberalism certainly had effects on how universities are run today. We know so much in my institution, the University of California, where dwindling state support has been replaced by private gifts, auxiliary enterprises, and financialized student debt. But lacking robust counterfactuals, disentangling endogenous processes tied to the institutional logics of higher education from the exogenous pressures created by policy and regulation is a vexing problem.

Take a simple example: the gendered demographics of academic staff. In the early 1980s, women represented less than 13.5% of academic employees in Britain. The number today is closer to 48%, with distinct patterns of stratification across ranks (as in 1980, women are less likely to occupy full professorships, reflecting the historical barriers that remain in education). The numbers are worse for Black and Minority Ethnic scholars, yet they also show an increase in representation over time. This change occurred precisely during the period when neoliberalism in general, and New Public Management in particular, affected British higher education. Are we to say that these transformations happened despite neoliberalism? Or because of neoliberal interventions? Were they purely exogenous, or completely endogenous? Surely, both directions played a role. And surely, no-one truly wants to return to the cloistered academia of the past. Greater representation certainly resulted from the constant grassroot efforts of some groups of academics to change their sites of employment as well as from the relentless labor of progressive organizations that shifted labor policies across the country more generally. But greater representation was also made possible by breaking with some of the traditional patriarchal logics of British higher education that placed hard limits on women’s access to the professoriate and that clashed with the meritocratic discourses of neoliberal subjects. As some of my informants noted, the new “market-like” mechanisms of things like REF created impersonal incentives on hiring and made certain careers more possible than what they were before.

Lacking a counterfactual where neoliberalism and its proxies did not shape higher education and research-oriented institutions, parsing causes and identifying culprits becomes a tremendously complicated task. More importantly, perhaps, focusing on “large” politics outside of universities obscures the role of the everyday politics of merit and worth that we collectively reproduce inside our institutions and that have the most immediate effects on our experience. It is, in other words, fundamentally disempowering. We can blame neoliberalism. We can lament its realization in policies, managerial expansion, and byzantine administrative structures. We can (and must!) contest it in the street and in the ballot box. But as a systemic process, our individual protests against these political fractals are likely ineffective tools for transforming our workplace. At least in the short run. This does not entail rejecting “any notion of neoliberal forcing”, but being attentive to scales of agency and effective contestation.

These politics of merit are well documented, from the early work of Max Weber and Robert Merton to the more recent outstanding contributions of Wendy Espeland, Mary Blair-Loy, Erin Cech, Erin Leahey, Victoria Reyes, and many others. What these scholars have shown is that processes of differentiation and exclusion within academia, while tied to broader discourses of accountability and merit, are performed through our actions as rankers, evaluators, and reproducers of disciplinary norms. Targeted neoliberal interventions certainly exist, but I am doubtful that such things as the “nudge unit” played as large a role in shaping knowledge as the more pedestrian politics of prestige we collectively perform around our journals and publications. Few in the state apparatus would know (or indeed care to know) about the citations our papers receive or where in the prestige hierarchy of our journals they are. The same cannot be said of ourselves and our peers. That these politics of prestige and academic celebrity precede New Public Management by at least a half century (isn’t this what Weber partly gesticulated at in his essay on science?) makes them a more plausible part of the explanation. Thus, my focus in The Quantified Scholar.

Saltelli (2023) and I nevertheless agree on the importance of doing something about the status quo. Whereas he calls for “a movement of contestation […] opposing the most obtuse practices of academic ranking” (p. 159), I propose developing networks and communities around reflexive solidarity.

Reflexive solidarity, I feel, is doubly powerful. It stems from a recognition of hierarchies and their role in modulating our profession and workplaces. I see this kind of solidarity as implicating more than just “academic staff” to consider fully and humbly the other sources of labor that make our profession possible. We should be, of course, critical of the hierarchies of prestige that we and our managers so often use to evaluate our worth — in journals, citations, pedigree, and metrics generally. But we should also be reflexive about how our labors are but one element in the constitution of the modern university. Any contestation is valueless if it does not include precarious academics and non-academic staff as co-participants, as peers. This is why I tend to be leery of calls for academics to lead movements, not because we are lacking in our competence or personal ethics, but because we are often complicit in making those less powerful than us invisible in our workplace.

Reflexive solidarity also entails recognizing that a shared future that is free from the problems of our individualist present comes at a cost. This is often ignored in debates but is an important part of critique and contestation. The kinds of institutions that academics would like to build, for example, might simply be different than those that would best serve staff, or students, or our local communities. Ours is not the only possible world, and this recognition sometimes gets lost in calls for reform. We might not get what we want, and that is fine.

Radically, reflexive solidarity implies taking active stands, sometimes with distinct reputational costs. Actively opposing interventions such as rankings involves much more than just criticizing these in public or organizing vast movements against them. It requires, in particular, contesting their use in our own workplaces, questioning their value, and being contrasted with useful, practical alternatives that make sense to local stakeholders, work that is invisible by design but nonetheless critical. Becoming this kind of “killjoy” (Ahmed, 2023) doesn’t come free. Congruence is, indeed, costly. We bemoan the takeover of academic journals by corporate interests, yet we continue to feed their gears with our intellectual labor. No-one pays us to review; if reported at all in our vitae, this information is completely insignificant for many of our institutions. But we do it anyway. We might tell ourselves that this is reciprocity for how our own work was reviewed, but this seems more of a rationalization of habits than critical intervention. Contesting this neoliberal practice (It’s for-profit extraction!) would require something bolder: starving for-profit journals of publications and reviews, steering away from their venues in citations and references, inundating open science repositories with the knowledge we produce, pushing for reform in our institutions to evaluate knowledge, not the status of publications. But this is a costly task. Not all scholars have the resources and protection this requires. Yet solidarity would mean that those who do, those who possess the scarce privilege of being able to act on this problem, take the appropriate steps. Solidarity is this surrendering to a common good, not for oneself or instrumentally for others. This is, in my view, precisely what we need.