1 Introduction

Despite the novelty of the technology, Large Language Models (LLMs) have seen nascent adoption by social scientists for research purposes in part due to the widespread public availability of such tools made possible by commercial offers. Within the social sciences we observe two approaches to LLM-produced textual content that seem to dominate. First, there are many studies that have creatively applied social science research methods to study algorithmic systems themselves (for an overview see Moats & Seaver, 2019; and articles in this Symposium). In particular, with the advent of LLMs we see a range of applications of social science methods from psychology experiments (Almeida et al., 2024) to variations on ethnography (Demuro & Gurney, 2024) as alternative ways to understand the capacities and limits of these technologies. Second, there are a number of studies that have used LLMs to augment existing social science methods either in data analysis or data generation (Møller et al., 2024). While all of these deserve discussion, in this essay we focus on the use of LLMs to produce — or experiment with producing — synthetic data as research material for social science questions. In particular, we are concerned with studies that seek to use such LLM-generated data to model or make inferences about how people might act, react, or respond in a variety of situations.

In a widely cited article, Grossmann et al. (2023) argue that the capacity of LLMs for “simulating human-like responses and behaviors offers opportunities to test theories and hypotheses about human behavior at great scale and speed” (p. 1108). The authors go as far as to imagine that “LLMs may supplant human participants for data collection” (p. 1109). This enthusiasm seems to be shared by an increasing number of scholars (e.g., Jansen et al., 2023; Aher et al., 2023; Argyle et al., 2023; Hämäläinen et al., 2023; Törnberg et al., 2023). At the same time, many scholars question some of the assumptions underlying these studies and suggest caution in deploying LLMs as social science data-generation mechanisms (e.g., von der Heyde et al., 2023; Bisbee et al., 2023; Agnew et al., 2024).

If LLMs could generate data as if they were “participants”, answering questions, making decisions or arguing for those, this would certainly represent a seismic shift for many social sciences that have often been struggling with hard-to-find participants, expensive data collection and questionable convenience samples. Yet while Grossmann et al. (2023) might claim that LLMs are now able to produce language in a “contextually aware and semantically accurate fashion” (p. 1108) current evidence within NLP research suggests that this is not quite the case just yet (Cui et al., 2023). While LLMs might represent a tempting opportunity for social science research given the ease and apparent facility in language production in response to carefully worded prompts, the use of such data for generating insights about people requires serious critical consideration.

In what follows, we review the extant literature and the emergent debate on the topic and discuss what we see as the main concerns with the use of LLMs as a source of social science research data.

First, we will discuss what type of synthetic data is actually produced by LLMs and in what way it differs from other approaches to producing synthetic data. Second, we will explore the implications of the cautions Grossmann et al. (2023) note — the fidelity of training data and considerations of its representativeness, the problem of LLM biases, the challenges of transformer-type models and their propensity for hallucinations, the emergent art of prompt engineering and the idea of benchmark selection. Third, we will consider what kind of knowledge can be gained from the analysis of LLM-produced data if we were to imagine that the problems of benchmarking, transparency in model training, and data fidelity can be addressed and what sort of legitimacy this knowledge might claim. Ultimately, we reflect on the proposition recently put forward by Bail (2024), that researchers could work towards the creation of open-source LLMs specifically trained and deployed for social research, by highlighting what we see as the major challenges that would need to be addressed.

2 Particularities of LLM-generated Data

LLM-produced data can be seen as a type of generated synthetic data. Synthetic data are datasets generated using purpose-built mathematical models or algorithms (Jordon et al., 2022) instead of being extracted from existing digital systems or produced through particular forms of data collection. The idea of synthetic data has a significant history in statistics and particularly within governmental and public institutions that need to make population data available for analysis while ensuring confidentiality (Abowd & Vilhuber, 2008) where population datasets have typically been treated through data augmentation and statistical disclosure control techniques (Raghunathan, 2021). Synthetic simulation data also has a long history in areas such as computer vision, which typically uses model specifications to generate datasets (Nikolenko, 2021). The advent of greater computational capacities and more complex machine learning models made synthetic data augmentation and generation faster, cheaper, less complex, and increasingly popular (Jordon et al., 2022).

Typically, synthetic data are seen to address any or all of three primary data challenges: data scarcity, data privacy, and data bias (Van der Schaar & Qian, 2023). While there is no doubt that generating synthetic data addresses the issue of scarcity by producing more data given a set of specifications, whether these data are of sufficient quality to be useful and whether they are able to address the challenges of privacy and bias depends on context (Abowd & Vilhuber, 2008; Belgodere et al., 2023; Figueira & Vaz, 2022; Jacobsen, 2023). With the advent of generative AI models in the development of synthetic data pipelines, we see the development of new frameworks proposed to evaluate the quality of the generated datasets. For example, Eigenschink et al. (2023) propose five assessment criteria for synthetically produced datasets: representativeness, novelty, realism, diversity, and coherence. They argue that high-quality synthetic data produced via generative AI should be able to capture population-level properties of the original data, create novel data points that are realistic (given what we know of the original data), and show internal diversity while maintaining coherency with the original data. They also note that the importance of the individual criteria varies significantly across domains and that the ways in which such criteria should be tested differ. Without delving too deeply into the specific framework, it is clear that the focus is on the ability of the synthetic data to reproduce key characteristics of the original data.

The development and rapid adoption of LLMs have led to the use of these systems for a broad range of data generation tasks. Whitney & Norman (2024) distinguish between generated, augmented, and procedurally created synthetic datasets differentiating between them “based on how derivative of a real-world training dataset they are” (p. 4). Procedurally created synthetic data rely on purpose-built models that create data along a set of explicitly pre-specified parameters, while generated data, such as what LLMs produce, arise from a model abstracting from a training dataset in response to a particular input. Here, the training dataset is crucial for ensuring that the resulting dataset is usefully similar to the data we might expect to collect from people. LLM-driven systems, however, are not built with faithful data generation as a goal. Rather than aiming to produce data that resemble a given original dataset, LLMs have been trained to predict the occurrence of the next most likely letter, word, or group of words given the linguistic patterns of a text. Trained on ever larger amounts of data, LLMs are able to mimic human-produced content (Jakesch et al., 2023) showing emergent abilities to perform tasks that were not explicitly present in the training data¹ (Radford et al., 2019; Wei et al., 2022). There is no effort here to adhere to particular pre-existing patterns in the data.

Given the ease with which current LLM implementations can generate human-like text on a near-infinite number of topics, it was not a huge leap to imagine the use of these technologies for generating data for research purposes. Seen as a convenient type of data, such LLM-generated datasets are typically obtained given sufficient effort in prompt engineering and, occasionally, the use of different available LLM implementations for comparison (Dillion et al., 2023, Horton, 2023). It is possible to classify LLMs as a type of synthetic data generator (Jordon et al., 2022). Keeping in mind that the goal of synthetic data generators is to produce data that resemble particular aspects of real data while addressing issues such as privacy, lack of diversity or data scarcity, one might ask if these goals are achieved by LLMs.

On the surface, LLMs can certainly generate data that mimics human-produced data, thus potentially resembling aspects of real data. Yet there is one important caveat to consider. Synthetic data is typically evaluated for utility and fidelity — how useful they are for a particular task and how well they resemble a real dataset given parameters important to the task. Here different types of fidelity may be considered but the important question is what is necessary for the task, but they require some capacity to compare the data we intend to mimic or augment and the synthetic output. At times, fidelity issues can be quite insidious, as Johnson & Hajisharif (2024) demonstrate a lack of intersectional fidelity in their tests with GAN-produced synthetic census data. How are we to evaluate the fidelity of an LLM-produced dataset, especially given the lack of access to the models generating the data or their training data? How might we evaluate differences in these data and their implications? Despite continuous advances, LLMs continue to display a lack of internal consistency in output. For example, Atil et al. (2024) have recently reported how, when systematically studied, LLMs show a lack of stability even when the input is the same. This was observed despite all the hyper-parameters being set to maximize the deterministic nature of the model. Importantly, stability not only varied between different models (both commercial and open source) but also within the same model as a function of the task.

Questions of utility and fidelity, of course, will vary depending on the particular applications of LLM-produced data, which we will address in the rest of the essay.

3 Applications of LLM-produced Data in Social Science Research

While clear-cut divisions are inherently problematic, here we identify what we see as major streams in LLM adoption for data production in the context of different types of social science research. Organizing the existing methodological experimentation around three directions allows us to see the existing similarities and differences between these approaches as well as the shared underlying assumptions about “what” LLMs can do and why. It is worth noting that while LLMs have been used to generate a variety of data we will generally treat this as textual data for two main reasons. First, the underlying training data of LLMs are essentially textual in nature. Second, even when LLMs are used to produce numeric values (e.g., producing a number that expresses agreement or disagreement on a scale) that is achieved through a textual prompt that asks the model to express the output as a number that would, otherwise, be textual.

When researchers use LLMs to produce synthetic research data, they are leveraging the underlying large amount of training data that characterizes these models as a proxy for individual data or data about specific groups and populations. This underlying, often unspoken, premise holds regardless of the specific research design and the specific LLM used. Within this perspective, there is no difference between using GPT-4, LLAMA-3 or Claude 3.5. What appears to be revolutionary is that LLM technology has reached a scale that allows emergent and unprecedented capabilities (Grossmann et al., 2023), irrespective of specific implementations. Nevertheless, the vast majority of research in this overview relies on OpenAI’s GPT-3.5 and GPT-4 models. This is not due to any explicit analysis of the specific capabilities of GPT versus alternative models but rather due to the easy access provided by OpenAI’s well-developed set of APIs.

4 The Challenge of Representativeness, Privacy, Bias and Hallucinations

As scholarly excitement grows around the capacity to produce increasingly varied types of LLM-generated data, we return to the typical challenges such data are expected to address: data scarcity, privacy concerns and regulations, and lack of diversity and data bias. The papers we reviewed differed substantially in how they discussed these concerns.

The vast majority of the papers we reviewed were clearly motivated by the problem of scarcity. Results are often praised in light of the “low cost and high speed” of LLM data generation (Hämäläinen et al., 2023; Törnberg et al., 2023; Argyle et al., 2023). Agnew et al. (2024) also identify scarcity as the most common reason. As is often the case with social research, scarcity in this context is due to cost. For the most part, people are not exactly scarce — not in the way that medical images of patients affected by a rare disease can be — but they can be expensive or complicated to engage. As a result, a number of authors are enthusiastic about the possibility to scale research in social and behavioral science, where it has notoriously relied on small and unrepresentative samples (Bail, 2024; Grossmann et al., 2023; Horton, 2023). There is no doubt that LLM-produced data can surely come in any volume necessary and at a very low cost, yet it is not clear whether such scaling is, in fact, defensible or useful.

When considering the challenge of privacy and regulatory limitations, none of the papers attend to the issue, although human participants do require a growing level of privacy protection and this directly translates both in ethical limits as well as into augmented costs for data collection, processing, and storage. This is not unexpected. Existing research shows that the actual risk that LLM-based synthetic data poses via the generation of non-maliciously prompted data seems quite low (Yan et al., 2024) and it is fair to assume that LLM-based synthetic data would not fall under the protection of regulations such as GDPR and would not require complex reviews from research ethics committees.

It is the issue of data bias, diversity, and representativeness that is discussed extensively across the reviewed research. After all, for LLM-generated textual data to be viable for social science research, the capacity to produce data that is representative of populations of interest is key. What seems to be the bottom line for much of the existing research is well exemplified by Argyle et al. (2023) when they argue that “algorithmic bias” in LLMs should be treated not as a macro-level property to be corrected, but as a feature that allows the model to produce outputs that reflect expected biases in the population and different subgroups. This argument builds on the idea that, since LLMs are trained on massive amounts of online data, the data will be able to capture fine details of the social system and of the several populations in it. This assumption is often paired with the assumed ability of LLMs to be conditioned, through prompting or fine-tuning, to assume specific points of view. In this way, LLMs are able to “extract” responses that faithfully represent actual subgroups or demographics from their massive amount of training data.

Yet there are many papers that document how LLMs tend to fail to generate output that is representative of various population subgroups (Bisbee et al., 2023; Simmons & Savinov, 2024; von der Heyde et al., 2023; Cao et al. 2023). These apparently contradictory results are not surprising at this early stage. Given that the sheer amount of data needed to achieve the performance reached by recent models is hardly obtainable through curated datasets, it is difficult to know exactly what the specific model ingested as data as well as what information about specific subgroups and with what level of reliability could be extracted from it. Such capacity to create what would essentially amount to data segmentation by sub-groups appears to be one of the key arguments in favor of LLM-generated data (Argyle et al., 2023; Aher et al., 2023).

“Bias as a feature to be exploited” is a cornerstone of the idea of algorithmic fidelity. Scholars argue that biased training data and its incorporation into the model is what gives the model the ability to faithfully reproduce social groups (Argyle et al., 2023). At the same time, since their large-scale commercial deployment, biases in LLMs’ outputs have been at the center of public attention (Gordon, 2023) as well as academic research (Fang et al., 2024). So much so that we have witnessed several attempts by commercial companies such as Google and OpenAI to mitigate model bias in their final output, often with mixed results (Goodman & Sandoval, 2024). Even when accepting the idea of algorithmic fidelity as unproblematic, researchers’ interests and platforms’ commercial plans do not seem aligned and research into the level of bias that is actually present in the final outputs of commercial models shows contradictory results (Tjuatja et al., 2023).

There are two fundamental questions — still largely unanswered — that suggest a careful approach to the idea of algorithmic fidelity and its consequent concept of algorithmic or “silicone” sampling (Argyle et al., 2023). First, what is the actual amount of bias that LLMs can reproduce? Second, what is the relation between the training data and emergent behaviors displayed by the models?

5 The Art and Challenge of Prompt Engineering and Evaluation

Social science research relies on robust methodological descriptions for evaluating research output and, in some fields, for ensuring replicability of results. With LLM-generated textual data, the methodological descriptions typically focus on prompt engineering as many papers argue that “proper conditioning” (Argyle, 2023) is key to ensuring fidelity of this kind of research. In many cases (with the notable exception of Park et al. 2022, 2023) prompt engineering is described as a tuning process necessary to achieve the best outputs/responses from the LLM, rather than a process with possibly profound consequences for the resulting data. Horton (2023) provides a good example of how prompts are often simply “listed”.

Listing the prompts used in the research process seems to speak more to the problem of transparency of research procedures and replicability than to the problem of data production. If prompt engineering is a matter of replicability of the results, this means that the selected prompt becomes the way to unlock the model’s ability to generate the desired data or the desired population. With the same prompting, the same or similar data would be produced again in the future. Yet, when investigating this specific assumption, Bisbee et al. (2023) found that generated data varies significantly both for small changes in the wording of the prompts as well as for the same prompt but asked at different moments in time. Similarly Atil et al. (2024) have reported overall instability of the output even when the conditions for deterministic behaviors are met.

If repeated prompts do not assure replicability of the research, then we have to consider how that should be documented. Lack of replicability can have many causes, from the hallucinatory nature of LLMs to the commercial nature of available platforms — platforms that are constantly updated and upgraded to offer an improved commercial product that does not need to be backward consistent. This has implications for how such data may need to be interpreted and what kinds of insights might be warranted. After all, differences due to emergent behavior have different implications to differences due to changes implemented at the platform level for commercial reasons.

Prompts lead us to consider evaluation and benchmarking. The papers we reviewed above offer different approaches to evaluating the resulting datasets for usefulness, fidelity, “faithfulness”, or “believability”. Where evaluations would typically cleave close to the purpose of data, they also need to be systematic and replicable (thus becoming consistent and robust research instruments). Current implementations run the gamut (thus Törnberg’s essay [2024] in this special issue) but they seem to often get reduced to measures of believability — does this output look like human output — which does not address the issue of usefulness given the assumptions of surveys or interviews and given the fact that people are just notoriously bad at distinguishing human and AI output even for older version of LLMs (Köbis & Mossink, 2021). Nor do they consider how normative ideas of what counts as “human” may be embedded in and reproduced through use of such evaluative measures (Rhee, 2018). Where believability is useful for creating non-player characters in computer games, because their goal is only to be “believable” in interactions with players, it isn’t a great measure of utility for making inferences about human responses. Just because the produced content is “believable” does not mean it has epistemic legitimacy.

This acknowledgement of different standards of “believability” draws attention to the importance the context of use and epistemological standpoint. For some social scientists, the idea that LLMs could produce “believable” material seems quite alien to the experience of, for example, conducting fieldwork to acquire relatively small amounts of qualitative data about lived experiences. For others, the material produced by an LLM may be sufficiently “believable” to be useful in a simulation. Underlying these differences are epistemological assumptions about how knowledge can and should be produced in order to say something useful about the world. Using “believability” as a way of assessing the material also tends to obscure darker questions: assessing “believability” requires a baseline in which humanness is quantified and measured, and can be used to compare outputs from LLMs. This quantification process reproduces highly problematic norms about who counts as human and why (D’Ignazio & Klein, 2023; & Gebru, 2018; Rhee, 2018).

The measure of faithfulness then may seem a better form of evaluation for some branches of the social sciences at least. Here we see replications of prior psychological or economics experiments with the idea that if LLM output aligns with what we know about how people respond to the known situation, then output produced in response to novel situations will be similarly aligned. Once again we run into several problems. The fields of psychology and economics have been going through a crisis of replicability — where scholars seem to be unable to replicate old and established experimental evidence with new studies with people. What does it say then if LLMs replicate the canon, even as it is challenged by studies with people? Many psychology and economics studies, as well as large-scale surveys in political science and demography, have been criticized for studying an unattainable ideal of the average person, statistically derived but non-existent in practice. Based on the results obtained by Bisbee et al. (2023) and by von der Heyde et al., (2023), LLMs might be in a similar situation, they can produce a set of averages — unattainable in practice.

As Horton (2023) notes, sure all models are wrong, LLMs included, but that does not mean we can’t use them for thinking about what questions to ask and how to ask them. Yet it is worth reflecting on what questions might emerge given the particular notions of the average person embedded in LLMs and what kinds of questions might be left out. After all, if LLMs were to somehow produce data that might lead to fundamentally new questions, would it not by definition fail the test of faithfulness?

6 The Question of Legitimacy and Situatedness of Knowledge

While much current work acknowledges the limitations of the LLM-generated data and explores using currently available technology, there is also substantial agreement that we can expect the quality of LLM-generated textual data to improve as more complex models come online, model architectures evolve and data curation methods become ever more sophisticated (Hämäläinen et al., 2023; Törnberg et al., 2023; Park et al., 2023). This, however, does not address the epistemological issues that we have considered in this paper. Stepping back from specific technical challenges, there remain broader epistemological questions which are provoked by the increasing interest in LLM-generated data across the social sciences. Here we focus on two areas of concern: legitimacy and situated knowledge.

First, several of the papers we have analyzed build their justification on a specific version of the data scarcity argument. Data is scarce for many reasons due to the costs and challenges associated with recruiting people for research studies. LLMs promise an infinite number of quasi-human participants, lowering that cost by making quasi-human data abundant. While the cost of data is indeed a serious barrier for many researchers, it is not clear to what extent this can justify the use of LLMs without a thoughtful assessment of its epistemic legitimacy. Data scarcity is also just as likely to indicate that the research participants are unwilling to take part or have never been considered as “valid” participants before. In such cases, there may be no existing data with which to compare, or the data may be considered highly problematic. In such cases, LLM-generated data may exacerbate existing data inequalities. Following Agnew et al. (2024) critique, using such data to solve a data scarcity problem risks misrepresentation or ignores the “real” question of why there is no data. The research community was tempted, in the not-too-distant past, to assume that large quantities of digital data could be used as good proxies for complex social phenomena, only to find out that this was not the case (Jungherr et al., 2012).

Second, qualitative scholars have long struggled to claim the validity of their insights, especially in fields dominated by alternative epistemological positions where quantitatively produced knowledge with marginally defensible claims to generalizability was seen as the only legitimate sort. There is much excitement about the capacity to scale prior experiments and studies on limited samples through the use of LLM-generated data (Agnew et al., 2024; Bail, 2024). This stems from the underlying assumption that such data can be seen as more representative of population groups that the researchers wish to study, also known as fixing the diversity problem. Insights derived from such simulations then would only need limited substantiation in “the real world” as it were. There is nothing inherently wrong with simulations, but the challenge here is in understanding how much, and in what ways will LLM-generated outcomes differ from human answers, and in identifying how and in what ways these models may be wrong, especially if we have limited prior data with which to compare against. The danger, as we see it, is in the well-documented tendency of LLMs to produce output that reduces already expected diversity when compared to prior studies (von der Heyde et al., 2023), essentially replicating the status quo, because this is strongly embedded in the training data.

LLM transformer architectures innovate beyond the problem of the more traditional machine learning models, which in their reliance on past data for making predictions are by definition “always fighting the last war again” (Groves, 2015). Yet they too are constrained by whatever reality the training data represent — the models, after all, can only inhabit a reality described in that training data and no other. Despite the vast amount of data used for training OpenAI’s GPT models, the gargantuan effort to clean training data and make them less toxic (Perrigo, 2023) speak to the desperate lack of quality or representativeness of these data. Yes, these data are the largest and “the best we got” but there is a good reason why unvarnished and uncorrected models are not made available — the mirror they hold up to humanity is profoundly terrible (Finkelstein, 2008). The models that are made available for consumption are adjusted, cleaned up, made palatable and “value-aligned” resulting in output that might create an imaginary generic average person, but who that person is, is difficult to assess. While simulating human social systems with LLMs provides intriguing insights into the models themselves, what such output might reveal about us more generally, is a question that requires cautious consideration.

How might we “situate” the knowledge produced using LLM-generated data then? The papers we cite default to lists of prompts and details of the technical setup, but arguably also require deeper considerations of where the models are wrong and what is missing, tempering the excitement with the possibility of sweeping statements about “human” behavior. What counts as “data” influences how “data” are understood, collected and processed, and are intimately connected with establishing and validating the boundaries of “proper” knowledge production (Haraway, 1988; Kitchin & Lauriault, 2018). LLM-generated data require deep considerations of fidelity — both intersectional and otherwise (Johnson & Hajisharif, 2024) — as well as of positionality, paying attention to what kinds of knowledge are made possible with the use of these data and which are foreclosed.

7 Data Must Be Cooked with Care

The idea of a computational assistant that could precisely and flexibly analyze vast amounts of complex data with quasi-qualitative skills is undoubtedly tempting for researchers, who have often struggled to adapt their methods to the growing amount and complexity of the data at their disposal. LLMs show such remarkable analytical skills as a result of the unprecedentedly large amount of data used in their training phase, but it is a leap to simply assume that these training data represent a viable proxy for the social reality behind the model. While still in its infancy, compared to other applications of LLMs, the idea of LLMs as data generators is intriguing, given the range of scholarly struggles with the complexities of data access. Some types of data may be more abundant, but they may not be easier to obtain or use.

Data is a complex and contested term, yet it has come to define the digital world we inhabit. Early debates around big data contested notions of raw data (Helmond, 2014), pointing out that data are never raw, out there, merely waiting to be collected (Bowker, 2008). Rather, data are always made, created, cooked as it were and if we are to acknowledge this, data ought to be cooked with care (Bowker, 2013). More importantly, when it comes to scientific practice, different epistemologies and methodologies “cook” data differently — seeing some methods of data generation as more legitimate than others. In many ways, synthetic data generation offers a way to create tidy, well-appointed datasets that are ultimately made specifically for this or that purpose, without the problems of cleaning messy data. Such control is one of the attractive qualities of synthetic data for many (Savage, 2023). What sort of cooking happens when generating data using LLMs? This is a much more difficult question to answer given myriad assumptions about training data, prompt processing, and emergent behaviors that must be made. Arguably, LLMs offer ease of generating data, but they provide far less control over the recipe, compared to any other approach to generating similar data from human participants.

In this essay we have looked at different examples of using LLMs for data production within the social sciences. We have discussed how LLM-generated data are similar to what we generally define as synthetic data but also where it differs. LLM-generated data aim to solve the problems of scarcity and privacy. LLMs’ ability to produce large amounts of seemingly realistic data, as well as their ability to role-play various demographics with a very tenuous identifiability with the underlying training data perfectly address these needs. When it comes to bias, studies proposing the use of LLM-generated data take a different approach compared to other types of synthetic data. Rather than seeing bias as a problem that should be measured, quantified, and potentially addressed, LLM-based approaches attempt to embrace it, leveraging it, either implicitly or explicitly, as algorithmic fidelity. This difference has interesting consequences and originates from the difference in goals. While most of the recent interest towards synthetic data is driven by the need to feed more and more high-quality data into AI models to improve their performance, LLM-generated data is the output of an AI model that could potentially be used directly for research or prototyping activities. This has potential but we argue that current approaches overlook a number of important issues.

8 The Future of Data?

As we have highlighted in the sections above, the idea of using LLM-generated data for research in the social sciences is relatively new and its robustness is still disputed. The inherent complexity of LLMs, as well as their fast-paced evolution suggest caution when researchers make assumptions about the models’ algorithmic fidelity or about their actual ability to be conditioned to represent various segments of the population. These are, after all, products not developed for research. Regardless of their commercial nature, which introduces additional complexity due to misaligned goals between tech companies and academic researchers, LLMs have not been developed as proxies of society. They have not been fed growing amounts of data aiming at improving their ability to represent a societal digital twin. Au contraire, many of the current trends (Saracco, 2023) that we see in the actual development of LLMs seem to suggest that the future will not be more data and more algorithmic fidelity but smaller models, trained with smaller amounts of data and fewer parameters that will still be able to score similar results in reasoning tasks.

While today LLMs can surely prove useful to produce data for prototyping research or testing initial hypotheses, their reliability should constantly be questioned and confirmed. Of course, the results we have discussed here leave open the possibility that future LLMs could be specifically designed, trained and developed for research applications. This possibility has recently been proposed by Bail (2024). Imagining large-scale LLMs developed and dedicated to research is not simple and requires a substantial change in the way social scientists approach their research tools, but it could also open up unprecedented opportunities. As many authors have noticed, what is potentially revolutionary are LLM-based models trained on large amounts of data, rather than the specific commercial implementation. Commercial solutions, while currently more advanced than open-source alternatives, come with many of the problems and limitations that we have discussed above (from unknown safeguards to fine-tuning and opaque training data). Open-source LLMs are better for ethical reasons (Spirling, 2023) and they may, at least in theory, offer better transparency, and improved control and could be based on ad-hoc training data. Yet if research-oriented open-source LLMs might be the future of LLMs for social research, it is probably a good idea to reiterate some of the key challenges they will have to face: representativeness, segmentability and data curation.

Representativeness: As many authors who have proposed the use of LLMs for data generation argue, bias that derives from biased data should not be considered a problem but as a feature of the system. Given biased training data, from a research point of view, one might want that bias to be transferred to the model’s outputs. The challenge is that this is not what has been observed. While LLMs seem to be able to faithfully reproduce biases and leaning at the level of large groups they systematically fail at representing smaller groups and minorities. Algorithmic fidelity, in other words, is not stable when the system is prompted to represent certain parts of the overall population. What is more, while some types of differences may be reproduced, there is always the danger of what Johnson & Hajisharif (2024) term “intersectional hallucination” where the inherent LLM bias and built-in attempts to mitigate it might result in strange demographic configurations.

Segmentability: Directly building on the assumption of bias as a way to faithfully represent the underlying data, there is the idea that LLMs can be segmented and conditioned to represent specific sub-populations. The extent to which this is true is still unclear. While prompting and fine tuning have shown some ability to condition the results, limits have also been observed as well as a considerable amount of inconsistency even with stable prompting.

Data curation: While LLMs require, by definition, a large amount of training data, complete lack of control over what constitutes training data is problematic both for ethical and legal reasons (Rahman & Santacana, 2023). With the progress shown by smaller models (Saracco, 2023), research LLMs should carefully consider to what extent curated training data is a possibility and what would be the consequences. Over the years researchers working with hard-to-get data have developed considerable experience with projects of data donation (Araujo et al., 2022). This experience could be leveraged to coordinate massive collaborative efforts that would select training data not because it is available or accessible but because it has been deemed relevant. The limits and consequences of such an approach are, clearly, unknown but the ethical and legal risks of the alternatives might end up being too large for non-profit research institutions.

As things are right now these problems have been scarcely investigated and solid ways to measure them and their impact on the LLMs’ ability to work as data-generation tools for social scientists have not been proposed. This should probably be a key part of any research agenda that leads to the actual development and deployment of LLMs as data generators for social sciences.