Articles that discuss the use of LLMs in science.
| Title | Type of Resource | Description of Resource | Link to Resource | Open Science | Use of LLM | Research Discipline(s) |
|---|---|---|---|---|---|---|
| The illusion of artificial inclusion | Discussion Article | Human participants play a central role in the development of modern artificial intelligence (AI) technology, in psychological science, and in user research. Recent advances in generative AI have attracted growing interest to the possibility of replacing human participants in these domains with AI surrogates. We survey several such "substitution proposals" to better understand the arguments for and against substituting human participants with modern generative AI. Our scoping review indicates that the recent wave of these proposals is motivated by goals such as reducing the costs of research and development work and increasing the diversity of collected data. However, these proposals ignore and ultimately conflict with foundational values of work with human participants: representation, inclusion, and understanding. This paper critically examines the principles and goals underlying human participation to help chart out paths for future work that truly centers and empowers participants. | artificial inclusion | Preprint | Data Collection, Data Generation | Computer Science |
| Generative AI Can Supercharge Your Academic Research | Discussion Article, Use Case Example | Conducting relevant scholarly research can be a struggle. Educators must employ innovative research methods, carefully analyze complex data, and then master the art of writing clearly, all while keeping the interest of a broad audience in mind. Generative AI is revolutionizing this sometimes tedious aspect of academia by providing sophisticated tools to help educators navigate and elevate their research. But there are concerns, too. AI’s capabilities are rapidly expanding into areas that were once considered exclusive to humans, like creativity and ingenuity. This could lead to improved productivity, but it also raises questions about originality, data manipulation, and credibility in research. With a simple prompt, AI can easily generate falsified datasets, mimic others’ research, and avoid plagiarism detection. [4 how to tutorials follow] | Using LLM in Research Process | Open Source | Other | Other |
| Automated Social Science: A Structural Causal Model-Based Approach | Discussion Article | We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are proposed and tested, finding evidence for some and not others. In the auction experiment, we show that the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from an LLM are inaccurate. However, the LLM’s predictions are dramatically improved if the model can condition on the fitted structural causal model. When given a proposed structural causal model for one of the scenarios, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those effects. This suggests that social simulations give the model insight not available purely through direct elicitation. In short, the LLM knows more than it can (immediately) tell. | Automated Social Science | Open Source | Other | Other |
| ChatGPT Undermines Human Reflexivity, Scientific Responsibility and Responsible Management Research | Discussion Article | We herewith certify that this essay represents original and independent scholarship. That is, generative AI was not used in the idea-generating phase of this essay, nor was it used to assist the writing or editing of this essay (with the exception of serving the purpose of a ‘bad’ example). We observe with great concern that many journal publishers – unlike Science – become complicit in undermining the meaning of the term ‘original scholarship’ by allowing the use of generative AI in the research process, while actual enforceability of relevant policies is low. Eventually, we need to be mindful of the deskilling of the (academic) mental sphere, while corporate influences on what constitutes knowledge is set to grow. | Underminds | Open Source | Other | Business |
| The future of research in an artificial intelligence-driven world. [multiple essays] | Discussion Article | Current and future developments in artificial intelligence (AI) systems have the capacity to revolutionize the research process for better or worse. On the one hand, AI systems can serve as collaborators as they help streamline and conduct our research. On the other hand, such systems can also become our adversaries when they impoverish our ability to learn as theorists, or when they lead us astray through inaccurate, biased, or fake information. No matter which angle is considered, and whether we like it or not, AI systems are here to stay. In this curated discussion, we raise questions about human centrality and agency in the research process, and about the multiple philosophical and practical challenges we are facing now and ones we will face in the future. [multiple essays] | Future of Research | Preprint | Other | Business, Other |
| Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review | Discussion Article | The emergence of systems based on large language models (LLMs) such as OpenAI’s ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks. | LLMs in Peer Review | Open Source | Science Communication, Other | Other |
| ChatGPT in Thematic Analysis: Can AI become a research assistant in qualitative research? | Discussion Article | The release of ChatGPT in November 2022 heralded a new era in various professional fields, yet its application in qualitative data analysis (QDA) remains underdeveloped. This article presents an experiment involving applying ChatGPT (Model GPT-4) to thematic analysis. By employing an adapted version of King et al.’s (2018) Template Analysis framework, this article aims to assess how ChatGPT can help with QDA in a full analytical process of a sample dataset provided by Lumivero. My experiment includes applying ChatGPT in four stages: data familiarization; preliminary coding and initial template formation; clustering and template modification and finalization; and theme development. Findings reveal GPT-4’s capacity in efficiency and speed in grasping the data and generating codes, subcodes, clusters, and themes, alongside its learning and adapting capabilities. However, the current version of the model has limitations in terms of effectively handling detailed analysis of large databases and producing consistent results, as well as the need to move across workspaces and the lack of relevant training data for QDA purposes. | ChatGPT in Thematic Analysis | Preprint | Data Analysis | Sociology |
| Could AI change the scientific publishing market once and for all? | Discussion Article | Artificial-intelligence tools in research like ChatGPT are playing an increasingly transformative role in revolutionizing scientific publishing and re-shaping its economic background. They can help academics to tackle such issues as limited space in academic journals, accessibility of knowledge, delayed dissemination, or the exponential growth of academic output. Moreover, AI tools could potentially change scientific communication and academic publishing market as we know them. They can help to promote Open Access (OA) in the form of preprints, dethrone the entrenched journals and publishers, as well as introduce novel approaches to the assessment of research output. It is also imperative that they should do just that, once and for all. | Human contribution in AI-powered research | Preprint | Science Communication | Other |
| Generative AI for Economic Research: Use Cases and Implications for Economists | Discussion Article | Generative AI, in particular large language models (LLMs) such as ChatGPT, has the potential to revolutionize research. I describe dozens of use cases along six domains in which LLMs are starting to become useful as both research assistants and tutors: ideation and feedback, writing, background research, data analysis, coding, and mathematical derivations. I provide general instructions and demonstrate specic examples of how to take advantage of each of these, classifying the LLM capabilities from experimental to highly useful. I argue that economists can reap signicant productivity gains by taking advantage of generative AI to automate micro tasks. Moreover, these gains will grow as the performance of AI systems across all of these domains will continue to improve. I also speculate on the longer-term implications of AI-powered cognitive automation for economic research. The online resources associated with this paper oer instructions for how to get started and will provide regular updates on the latest capabilities of generative AI that are useful for economists. | LLM in Econ | Open Source | Other | Economics |
| Can AI language models replace human participants? | Discussion Article | Recent work suggests that language models such as GPT can make human-like judgments across a number of domains. We explore whether and when language models might replace human participants in psychological science. We review nascent research, provide a theoretical model, and outline caveats of using AI as a participant. | Replace Human | Data Collection | Psychology |